Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[cgohlke]

BPO	25027
Nosy	@pfmoore, @larryhastings, @tjguk, @skrah, @zware, @eryksun, @zooba
Files	test_dll_load_failed.py 25027_1.patch 25027_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2015-09-12.15:31:38.716>
created_at = <Date 2015-09-08.08:05:36.919>
labels = ['extension-modules', 'OS-windows', 'release-blocker']
title = 'Python 3.5.0rc3 on Windows can not load more than 127 C extension modules'
updated_at = <Date 2015-09-12.15:31:38.715>
user = 'https://bugs.python.org/cgohlke'

bugs.python.org fields:

activity = <Date 2015-09-12.15:31:38.715>
actor = 'larry'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2015-09-12.15:31:38.716>
closer = 'larry'
components = ['Extension Modules', 'Windows']
creation = <Date 2015-09-08.08:05:36.919>
creator = 'cgohlke'
dependencies = []
files = ['40402', '40410', '40418']
hgrepos = []
issue_num = 25027
keywords = ['patch']
message_count = 19.0
messages = ['250169', '250184', '250189', '250195', '250216', '250222', '250224', '250226', '250227', '250240', '250256', '250262', '250272', '250273', '250279', '250287', '250296', '250313', '250522']
nosy_count = 9.0
nosy_names = ['paul.moore', 'larry', 'tim.golden', 'cgohlke', 'skrah', 'python-dev', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue25027'
versions = ['Python 3.5']

[cgohlke]

This issue was first mentioned at <http://bugs.python.org/issue24872#msg249591\>.

The attached script (test_dll_load_failed.py) builds up to 256 C extension modules and imports them. On Python 3.4.3 the script passes while on Python 3.5.0rc3 the script fails with:

Traceback (most recent call last):
File "test_dll_load_failed.py", line 42, in <module>
import_module(name)
File "X:\Python35\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
File "<frozen importlib._bootstrap>", line 577, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 903, in create_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

Tested on Windows 7 and 10, 64 bit, with Python 3.5.0rc3, 32 and 64 bit.

Due to this issue the "scientific stack" is practically unusable. For example, Pandas unit tests error or abort. In a Jupyter notebook, the following simple imports fail (using current builds from <http://www.lfd.uci.edu/~gohlke/pythonlibs/\>):

In [1]:

import matplotlib.pyplot
import pandas
import statsmodels.api
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-69fcce4de550> in <module>()
      1 import matplotlib.pyplot
      2 import pandas
----> 3 import statsmodels.api

<snip>

X:\Python35\lib\site-packages\scipy\signal\__init__.py in <module>()
    276 # The spline module (a C extension) provides:
    277 #     cspline2d, qspline2d, sepfir2d, symiirord1, symiirord2
--> 278 from .spline import *
    279 
    280 from .bsplines import *

ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

The cause of this issue is that as of Python 3.5.0rc1 C extension modules are linked statically to the multi-threaded runtime library (/MT) instead of the multi-threaded DLL runtime library (/MD). A process can not load more than 127 statically-linked CRT DLLs using LoadLibrary due to a limit of fiber-local storage (FLS) as mentioned in the following links:

<http://stackoverflow.com/questions/1437422\>
<https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/3546c3c4-1b36-4552-85c5-1b3ba860ee84\>

To put the 127 limit in perspective: the pywin32 package contains 51 C extension modules, pygame 36, scipy 65, and scikit-image 41.

In addition to C extension modules, the 127 limit also applies to statically-linked CRT DLLs that are dynamically loaded via Ctypes or LoadLibrary.

[skrah]

If the scientific stack is unusable, I think this should be a release
blocker.

[larryhastings]

This is your wheelhouse, Steve.

[zooba]

Let me experiment today with a few of the proposals I posted in the other thread and get back to you.

I suspect someone will need to ship vcruntime.dll, and I'd rather it was the extension.

[skrah]

Is Python-core built with /MD? I cannot see the flags in the buildbot
logs.

[zooba]

Kind-of... We use the same flags I described in my blog1 so that we don't have any version-specific dependencies.

You should (might) see /MT in the build logs, but then we replace most of the static CRT with the dynamic (but versionless) one. The versioned parts (including the FlsAlloc call - module initialization is compiler version specific) are statically linked.

I'm going to try and update distutils to build with /MD again (sorry Christoph!) and include vcruntime###.dll in the output. That way, people who bdist_wheel will include all of their own dependencies and don't have to worry about whether users are on Python 3.5.0 or 3.9.9.

[skrah]

It seems to be /MTd. Sorry for the noise (and yay! for horizontal scrolling :).

[skrah]

The reason I asked: We had issues where extension modules linked
against a different CRT had isolated locale settings from the
interpreter, i.e., changes made by the module would not be seen
by the interpreter.

I don't know if this is still an issue with the new runtimes.

[zooba]

It shouldn't be - locale state is in the shared part of the CRT. That is one of the reasons I was keen to move to this model (everyone seems to think that FILE* is the only problem with mixing CRT versions :) )

[zooba]

Attached a fix for distutils that will include the required vcruntime DLL with the extension.

This is purely Python source code and only needs to be patched on the build machine.

I have tested with a numpy build from source (setup.py bdist_wheel) and it works correctly on a clean Win7 machine with only a Python 3.5.0rc3 install.

When multiple vcruntime###.dll files are available, the first one that is loaded wins. This means that if you can guarantee a particular import occurs first, you can omit the DLL elsewhere in your package. There is no way to determine this automatically,

Because of the way I wrote the patch, if you build with DISTUTILS_USE_SDK set, you will get the old static linking unless you also define PY_VCRUNTIME_REDIST to the path of the file to include. If the redist file cannot be found (you probably don't have the compiler either, but assuming you do), you will get the old static linking. I think this is the right balance of "works-by-default" and "I know what I'm doing let me control it" (though now I put them next to each other, I could rename the variable to DISTUTILS_VCRUNTIME_REDIST).

Will work up a test for it, but wanted to get feedback on the approach first.

[cgohlke]

I understand that distributing dependent DLLs next to extension modules is considered the best approach <https://mail.python.org/pipermail/distutils-sig/2014-October/024990.html\> (which nevertheless fails in common cases), however vcruntime140.dll is a special case since it will be shared by almost all extension modules and can be considered a system library.

My Python 3.4 installation contains 913 .pyd files in 277 directories under Lib\site-packages. With the proposed change there will be ~277 redundant vcruntime140.dll files under Python 3.5. Size is not an issue since vcruntime140.dll is small (~87 KB for 64 bit).

Many extension modules are installed directly into Lib\site-packages (no package directory). Uninstalling any one of those extension modules using pip or "wininstaller" will delete vcruntime140.dll from Lib\site-packages, potentially breaking the other extension modules in Lib\site-packages.

IANAL, but under GPL "you may not distribute these [runtime] libraries in compiled DLL form with the program" <http://www.gnu.org/licenses/gpl-faq.html#WindowsRuntimeAndGPL\>.