Use VS 2017 compiler for build

[zooba]

BPO	31340
Nosy	@pfmoore, @tjguk, @jkloth, @ned-deily, @methane, @zware, @zooba
PRs	bpo-31340: Change to building with MSVC v141 (included with Visual Studio 2017) #3311 [3.6] bpo-31340: Change to building with MSVC v141 (included with Visual Studio 2017) (GH-3311) #3386

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2017-09-06.22:57:09.278>
created_at = <Date 2017-09-04.20:17:14.961>
labels = ['type-feature', '3.7', 'OS-windows', 'build']
title = 'Use VS 2017 compiler for build'
updated_at = <Date 2018-01-06.18:49:22.620>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2018-01-06.18:49:22.620>
actor = 'cgohlke'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2017-09-06.22:57:09.278>
closer = 'steve.dower'
components = ['Build', 'Windows']
creation = <Date 2017-09-04.20:17:14.961>
creator = 'steve.dower'
dependencies = []
files = []
hgrepos = []
issue_num = 31340
keywords = []
message_count = 14.0
messages = ['301253', '301486', '301537', '301538', '301562', '303862', '303863', '303864', '303867', '303868', '303884', '309399', '309547', '309568']
nosy_count = 8.0
nosy_names = ['paul.moore', 'tim.golden', 'jkloth', 'ned.deily', 'methane', 'cgohlke', 'zach.ware', 'steve.dower']
pr_nums = ['3311', '3386']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31340'
versions = ['Python 3.6', 'Python 3.7']

[zooba]

The newer MSVC (v141) is available and reliable, and theoretically binary compatible with v140. This means we can update both Python 3.6 and 3.7 to build with it.

Testing for this should include:

• pythoncore with v141 and remainder with v140
• Python with v141 and wheels with v140

[zooba]

New changeset 5fcd5e6 by Steve Dower in branch 'master':
bpo-31340: Change to building with MSVC v141 (included with Visual Studio 2017) (bpo-3311)
5fcd5e6

[zooba]

New changeset fd645ec by Steve Dower in branch '3.6':
[3.6] bpo-31340: Change to building with MSVC v141 (included with Visual Studio 2017) (GH-3311) (bpo-3386)
fd645ec

[zooba]

Note that this should not change anything for 3.6, but enables using VS 2017 to build. If you only have VS 2015 then it will use that, and I've enabled AppVeyor for both to make sure we keep it that way.

3.7 will also build with either, but there isn't as much importance in verifying VS 2015 compatibility.

All of that said, there should be no difference in the built binaries.

[methane]

Nice! VS2017 is much lighter than VS2015 for install size!
https://blogs.msdn.microsoft.com/vcblog/2016/08/22/the-lightweight-visual-studio-15-installer/

[cgohlke]

I have Visual Studio 2015 and Visual Studio 2017 Community editions with latest patches installed.

Building a minimal C extension on Python 3.6.3 with distutils now uses MSVC 14.11.25503. That is unexpected.

When building complex extensions that link to static libraries that were built with MSVC 14.0, linking fails, e.g. when building Pillow:

'jpeg.lib' was created with an older compiler than other objects; rebuild old objects and libraries

It looks like practically v141 is not binary compatible with v140.

[zooba]

Build artifacts (static libraries, in this case) are not compatible, but the built binaries are.

This may be a reasonable argument for not updating distutils's support in minor versions (and I'm totally happy to just stop updating distutils period - see the recent distutils-sig discussion for more context). If it had been raised before 3.6.3 was released then I'd have switched it back (or at least swapped the order, so if v140 was found then it would use that first and fall back on v141). Now it's done, I'm not sure whether it hurts more to change it again in 3.6.4 or not...

The immediate workaround is to set DISTUTILS_USE_SDK and run in a VS 2015 command prompt, which will bypass all the search logic. Better to rebuild the static libraries with v141 as well, but I know that's not trivial or always feasible. Alternatively, build with Python 3.6.2 or earlier - until we get the ABI stability properly figured out, that's still the best way to ensure compatibility with older releases.

I'll give the change back some more thought.

[cgohlke]

I build most of my binaries after calling the correct vcvarsall.bat so I did not notice until today, when I was trying to build outside that environment.

Also, I misread your comment (https://bugs.python.org/issue31340#msg301538) earlier and did not worry at that time.

I guess this might become a bigger issue once Appveyor switches to Python 3.6.3. Projects using Anaconda's prebuilt static libraries for v140 will fail to link.

How about giving priority to VS 2015, i.e. "If you have VS 2015 then it will use that"?

[zooba]

How about giving priority to VS 2015

That would be the fix, but it's not worth releasing an immediate 3.6.4 IMHO and by the time 3.6.4 comes about it could be more pain to change back than to leave it. That's the balance I'm thinking about for a while.

Including Ned so he's aware. If we do a snap 3.6.4 for some other reason then it's worth swapping the order, but after a few months out there I think the damage, small as it is, will have been dealt with.