Change of behavior for importlib between 3.4 and 3.5 with DLL loading

[ebfortin]

BPO	24748
Nosy	@brettcannon, @pfmoore, @ncoghlan, @larryhastings, @tjguk, @encukou, @ericsnowcurrently, @zware, @zooba
Files	pywintypes.py: pywintypes.py from pywin32 issue24748.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2015-09-06.04:04:37.806>
created_at = <Date 2015-07-29.13:20:27.402>
labels = ['extension-modules', 'deferred-blocker', 'type-bug', 'OS-windows']
title = 'Change of behavior for importlib between 3.4 and 3.5 with DLL loading'
updated_at = <Date 2015-09-07.06:46:39.025>
user = 'https://bugs.python.org/ebfortin'

bugs.python.org fields:

activity = <Date 2015-09-07.06:46:39.025>
actor = 'ncoghlan'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2015-09-06.04:04:37.806>
closer = 'larry'
components = ['Extension Modules', 'Windows']
creation = <Date 2015-07-29.13:20:27.402>
creator = 'ebfortin'
dependencies = []
files = ['40055', '40352']
hgrepos = []
issue_num = 24748
keywords = ['patch']
message_count = 25.0
messages = ['247573', '247578', '247579', '247581', '247582', '247590', '247593', '247600', '247602', '247606', '247608', '247609', '247613', '247670', '249277', '249312', '249636', '249729', '249748', '249808', '249895', '249903', '249957', '250068', '250082']
nosy_count = 12.0
nosy_names = ['brett.cannon', 'paul.moore', 'ncoghlan', 'larry', 'tim.golden', 'petr.viktorin', 'python-dev', 'eric.snow', 'zach.ware', 'steve.dower', 'James Salter', 'ebfortin']
pr_nums = []
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24748'
versions = ['Python 3.5', 'Python 3.6']

[ebfortin]

The pywin32 package use imp.load_dynamic() to load a DLL with Windows specific type. On Python 3.4+ imp.load_dynamic() point to the following code which use the newer importlib module:

        import importlib.machinery
        loader = importlib.machinery.ExtensionFileLoader(name, path)
        return loader.load_module()

In pywin32 a mechanism is used to be able to have part of a module globals defined in python (pywintypes.py), and the reminder in a DLL (pywintypesXX.dll). The code in pywin32 between 3.4 and 3.5 is the same.

In Python 3.4, calling imp.load_dynamic(), which point to the code above, inside the python part of the module definition loads the types defined in the DLL.

In Python 3.5, calling imp.load_dynamic(), which also points to the code above, inside the python part of the module definition reloads the same python module, not the DLL. Even though a path to a DLL is given AND when doing introspection of the module it clearly points to the specified DLL.

This is a change of behavior that breaks pywin32, but possibly other modules that rely on this behavior.

[ericsnowcurrently]

This is most likely related to PEP-489, which changed extension module loading in what was meant to be a backward-compatible way.

[ebfortin]

It is also possible that the root cause is related to Microsoft Windows Update 2999226 and/or 3065987. The behavior was the same between 3.4 and 3.5 on a machine without these updates and is believed to have changed after the install of these updates.

[zooba]

Did the behavior change for 3.4 after the updates? It seems unlikely.

Zach's right: we need a test to formally state the expectations here.

[ebfortin]

No the behavior only changed for 3.5. 3.4 works just fine.

[ebfortin]

I suggest the test should use pywin32. The test script could be only:

import pywintypes
dir(pywintypes)

Testing for an attribute that is defined in the DLL would make it pass or fail.

[zooba]

I'd rather have our own test pyd that shows a minimal repro of the behavior. Requiring pywin32 for running one test is excessive, won't reveal the issue (if it exists) on other platforms, and is not necessarily going to help someone debug the issue.

I agree this is probably due to PEP-489, which means that Eric is almost certainly the best person to look into it. It may also be due to the list of suffixes changing for 3.5 (".cp35-win32.pyd" is now a valid suffix as well as just ".pyd", for example), but looking at pywintypes.py that seems unlikely.

[encukou]

Eric or me.

I'm not sure I understand the description clearly. Etienne, would it be possible to write a smaller reproducer, that wouldn't be tied to Windows?
Or is this Windows-only behavior?

[ebfortin]

At this point I can't say if it's Windows only or if it affect all platform. I can't even guarantee that it will appear on all Windows platform. On my platform (see following), it doesn't work:

Windows 7 64 bits with updates 2999226 and 3065987 installed.

I can't confirm the update are related either. I just did a quick analysis and between the time 3.5 was working and wasn't anymore these updates were installed.

I will create a quick script to reproduce the behavior.

[ebfortin]

The only dll / pyd files I have are all in pywin32. I don't have a build environment for extensions. Can anyone provide me with a very simple extension DLL with at least one exported attribute?

[zooba]

3.5 won't work without 2999226 at all, so ignore that. It's the C runtime.

[ebfortin]

Is it possible that the C runtime introduced with 2999226, which I believe is the "universal runtime" Microsoft is trying to introduce, modified something that makes importlib break on 3.5???