test_distutils: test_get_exe_bytes() failure on AppVeyor

[vstinner]

BPO	32302
Nosy	@pfmoore, @vstinner, @tjguk, @zware, @zooba
PRs	bpo-32302: Fix distutils bdist_wininst for CRT v142 #4851 [3.6] bpo-32302: Fix distutils bdist_wininst for CRT v142 (GH-4851) #4861

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-12-14.11:06:42.655>
created_at = <Date 2017-12-13.15:35:38.374>
labels = ['3.7', 'tests', 'OS-windows']
title = 'test_distutils: test_get_exe_bytes() failure on AppVeyor'
updated_at = <Date 2017-12-14.11:06:42.654>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-12-14.11:06:42.654>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-14.11:06:42.655>
closer = 'vstinner'
components = ['Tests', 'Windows']
creation = <Date 2017-12-13.15:35:38.374>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 32302
keywords = ['patch']
message_count = 11.0
messages = ['308203', '308207', '308210', '308211', '308215', '308218', '308233', '308254', '308284', '308286', '308289']
nosy_count = 5.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'zach.ware', 'steve.dower']
pr_nums = ['4851', '4861']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue32302'
versions = ['Python 3.7']

[vstinner]

Example: https://ci.appveyor.com/project/python/cpython/build/3.7.0a0.9414

======================================================================
ERROR: test_get_exe_bytes (distutils.tests.test_bdist_wininst.BuildWinInstTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\projects\cpython\lib\distutils\tests\test_bdist_wininst.py", line 24, in test_get_exe_bytes
    exe_file = cmd.get_exe_bytes()
  File "C:\projects\cpython\lib\distutils\command\bdist_wininst.py", line 361, in get_exe_bytes
    f = open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\projects\\cpython\\lib\\distutils\\command\\wininst-14.12.exe'

---

[zooba]

I thought we'd special cased the v14 toolset already.

This should be an update to the Python code in distutils that selects the filename based on compiler version.

[vstinner]

Extract of pythoninfo of the failed build:

os.environ[VS110COMNTOOLS]: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\
os.environ[VS120COMNTOOLS]: C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\
os.environ[VS140COMNTOOLS]: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\

The build starts with:

Using "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MSBuild.exe" (found in the PATH)

[vstinner]

Oh, and the PATH:

os.environ[PATH]: C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\7-Zip;C:\Tools\GitVersion;C:\Tools\NuGet;C:\Program Files\Microsoft\Web Platform Installer\;C:\Tools\PsTools;C:\Program Files\Git LFS;C:\Program Files\Mercurial\;C:\Program Files (x86)\Subversion\bin;C:\Tools\WebDriver;C:\Tools\Coverity\bin;C:\Tools\MSpec;C:\Tools\NUnit\bin;C:\Tools\NUnit3;C:\Tools\xUnit;C:\Program Files\nodejs;C:\Program Files (x86)\iojs;C:\Program Files\iojs;C:\Users\appveyor\AppData\Roaming\npm;C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\;C:\Program Files\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\ManagementStudio\;C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn\;C:\Program Files\Microsoft SQL Server\130\DTS\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;C:\Ruby193\bin;C:\go\bin;C:\Program Files\Java\jdk1.8.0\bin;C:\Program Files (x86)\Apache\Maven\bin;C:\Python27;C:\Python27\Scripts;C:\Program Files\erl8.3\bin;C:\Program Files (x86)\CMake\bin;C:\Tools\curl\bin;C:\Program Files\LLVM\bin;C:\Users\appveyor\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\130;C:\Program Files\Amazon\AWSCLI\;C:\Program Files\dotnet\;C:\Tools\vcpkg;C:\Program Files (x86)\dotnet\;C:\Users\appveyor\AppData\Local\Microsoft\WindowsApps;C:\Users\appveyor\AppData\Local\Yarn\bin;C:\Users\appveyor\AppData\Roaming\npm;C:\Program Files\Docker;C:\Program Files\Git\cmd;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files\Git\usr\bin;C:\Program Files (x86)\Yarn\bin;C:\ProgramData\chocolatey\bin;C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\;C:\Program Files\Microsoft SQL Server\140\Tools\Binn\;C:\Program Files\Microsoft SQL Server\140\DTS\Binn\;C:\Program Files (x86)\nodejs\;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\TestPlatform;C:\Users\appveyor\AppData\Local\Microsoft\WindowsApps;C:\Users\appveyor\AppData\Local\Yarn\bin;C:\Users\appveyor\AppData\Roaming\npm;C:\Program Files\AppVeyor\BuildAgent\

[zooba]

This is the bit that needs fixing https://github.com/python/cpython/blob/master/Lib/distutils/command/bdist_wininst.py#L340

            bv = '.'.join(CRT_ASSEMBLY_VERSION.split('.', 2)[:2])
            if bv == '14.11':
                # v141 and v140 are binary compatible,
                # so keep using the 14.0 stub.
                bv = '14.0'

AppVeyor has clearly gotten hold of 14.12, which should also be binary compatible (version numbers here are not strictly SemVer, but the 14 indicates binary compatibility).

[vstinner]

@Steve: do you want to work on a fix?

[zooba]

Happy for someone else to do it. I won't have time this week - too much going on at work.

[vstinner]

Happy for someone else to do it. I won't have time this week - too much going on at work.

Sure. I wrote a PR to fix the exact bug.

[vstinner]

New changeset 9e7c136 by Victor Stinner in branch 'master':
bpo-32302: Fix distutils bdist_wininst for CRT v142 (bpo-4851)
9e7c136

[vstinner]

New changeset 645fa19 by Victor Stinner (Miss Islington (bot)) in branch '3.6':
bpo-32302: Fix distutils bdist_wininst for CRT v142 (GH-4851) (bpo-4861)
645fa19

[vstinner]

It seems like AppVeyor is fixed, thanks Steve for the help!

Later, it would be nice to write a more generic fix than hardcoding two minor versions.