_msvccompiler.py doesn't properly support manifests

[gladman]

BPO	24798
Nosy	@pfmoore, @tjguk, @merwok, @bitdancer, @zware, @zooba, @dstufft
Files	24798_1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2015-08-07.21:43:20.245>
created_at = <Date 2015-08-05.16:25:34.546>
labels = ['type-bug', 'library', 'OS-windows']
title = "_msvccompiler.py doesn't properly support manifests"
updated_at = <Date 2015-08-14.23:37:57.812>
user = 'https://bugs.python.org/gladman'

bugs.python.org fields:

activity = <Date 2015-08-14.23:37:57.812>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2015-08-07.21:43:20.245>
closer = 'steve.dower'
components = ['Distutils', 'Windows']
creation = <Date 2015-08-05.16:25:34.546>
creator = 'gladman'
dependencies = []
files = ['40130']
hgrepos = []
issue_num = 24798
keywords = ['patch']
message_count = 9.0
messages = ['248050', '248052', '248057', '248230', '248231', '248412', '248413', '248621', '248622']
nosy_count = 10.0
nosy_names = ['paul.moore', 'tim.golden', 'eric.araujo', 'r.david.murray', 'cgohlke', 'python-dev', 'zach.ware', 'steve.dower', 'dstufft', 'gladman']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24798'
versions = ['Python 3.5', 'Python 3.6']

[gladman]

I have been using _msvcompiler.py from Python 3.5 to build some executables but I have been unable to get it to generate and embed a manifest.

When I looked into this I found that the subroutine that sets up the parameters for generating a manifest ('manifest_get_embed_info' at around line 471) has the line:

        ld_args.append('/MANIFESTFILE:' + temp_manifest)

to set the manifest's name but doesn't actually ask for a manifest to be generated. Here is what is said about /MANIFESTFILE on MSDN:

"/MANIFESTFILE will have no effect if you do not also link with /MANIFEST."

After adding:

        ld_args.append('/MANIFEST')

before the above line, I then succeed in obtaining the manifest.

[zooba]

It looks like manifests aren't generated with CRT version info anymore either, so I'll take this opportunity to clean up the build process some more too.

[zooba]

I've simplified the manifest handling:

• embed by default, since we no longer need to filter CRT SxS info
• omit UAC info from DLLs
• use full PATH when running tools
• some other tidying

Not necessarily going to wait for lots of reviews, as I know very few people are interested in distutils, but feel free to chime in if you like. I'll give it a day or so.

[python-dev]

New changeset 8e966eba2b5e by Steve Dower in branch '3.5':
Issue bpo-24798: _msvccompiler.py doesn't properly support manifests
https://hg.python.org/cpython/rev/8e966eba2b5e

New changeset f61a083b843f by Steve Dower in branch 'default':
Issue bpo-24798: _msvccompiler.py doesn't properly support manifests
https://hg.python.org/cpython/rev/f61a083b843f

[zooba]

Made a couple more changes (including a fix for vcruntime140.dll embedding) after testing numpy's build, but that worked fine with the final fixes.

[cgohlke]

This change broke all my builds that link statically against 3rd party libraries built with the /MD flag. /MD was used at least since Python 2.3 and is the default for static libraries in Visual Studio 2015. Some of the broken builds: lxml, pillow, matplotlib, pygame, pycuda, pymssql, netcdf4, GDAL, psycopg2, pycurl, gmpy, and pyopenssl. All of these packages built OK with Python 3.5.0b4.

The build errors are of this kind:
error LNK2001: unresolved external symbol __imp_memchr
error LNK2001: unresolved external symbol __imp_strstr

The linker throws the following warning:
LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs; use /NODEFAULTLIB:library

[bitdancer]

Please open a new issue, referencing this one. Priority should be set to release blocker. (I forget if regular users can do that; if you can't I will.)

[cgohlke]

It seems the switch to '/MT' was consciously intended as Python 3.5 itself is now compiled with '/MT'.
For now I have patched _msvccompiler.py to use '/MD' and continue to link libraries built with '/MD'.

[zooba]

Those libraries will depend on vcruntime140.dll which is not installed with CPython right now. In order for packages built with future compilers to work we need to statically link that dependency but not ucrt, which there are linker options for.

I'm thinking hard to come up with better approaches, but most of them require predicting the future. Create a new issue if you like or I'll make one when I have something to propose.