importlib.machinery.ExtensionFileLoader cannot load several modules from the same shared object

[eudoxos]

BPO	16421
Nosy	@brettcannon, @amauryfa, @ncoghlan, @pitrou, @vstinner, @bitdancer, @asvetlov, @ericsnowcurrently
Files	many-modules-in-one-so_1.diff many-modules-in-one-so_2.diff: Patch version 2, with tests many-modules-in-one-so_3.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/asvetlov'
closed_at = <Date 2012-12-15.16:28:40.662>
created_at = <Date 2012-11-06.14:42:53.120>
labels = ['extension-modules', 'type-bug']
title = 'importlib.machinery.ExtensionFileLoader cannot load several modules from the same shared object'
updated_at = <Date 2012-12-15.16:28:40.660>
user = 'https://bugs.python.org/eudoxos'

bugs.python.org fields:

activity = <Date 2012-12-15.16:28:40.660>
actor = 'asvetlov'
assignee = 'asvetlov'
closed = True
closed_date = <Date 2012-12-15.16:28:40.662>
closer = 'asvetlov'
components = ['Extension Modules']
creation = <Date 2012-11-06.14:42:53.120>
creator = 'eudoxos'
dependencies = []
files = ['27909', '27911', '27918']
hgrepos = []
issue_num = 16421
keywords = ['patch']
message_count = 25.0
messages = ['174967', '174970', '174971', '174972', '174975', '174976', '174977', '174981', '174988', '174992', '175004', '175059', '175087', '175090', '175103', '175124', '176654', '176662', '177461', '177462', '177466', '177535', '177536', '177537', '177538']
nosy_count = 12.0
nosy_names = ['brett.cannon', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'Arfrever', 'r.david.murray', 'asvetlov', 'docs@python', 'python-dev', 'eric.snow', 'eudoxos']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16421'
versions = ['Python 3.4']

[eudoxos]

This issue was split off bpo-16194 (I am not copying the whole discussion here). It was suggested a new issue is opened for Python3, where a proper fix can be done.

Python internally caches dynamically loaded modules, but the cache is based on the filename only. If one file contains several modules, only the first of them is imported and initialized properly. This interface was previously exposed via imp.load_dynamic, the same codepath is used by importlib.machinery.ExtensionFileLoader now.

A solution is to cache by the (filename, modulename) tuple, which avoids any ambiguity.

I am attaching a simple patch for that, agains current hg tip (80272:ec00f8570c55), and tested with the scripot attached to bpo-16194.

I have not modified the test suite, I am not sure whether testing compiled modules is actually supported (did not see any compiled files in there), I will be glad for help.

[asvetlov]

Storing several modules in single so/pyd file is crazy idea from my perspective.

The test is definitely required.

BTW, Why version is set to 3.5?
Should component be set to "Interpreter code"?

[amauryfa]

It's not that crazy, if you consider that all builtin modules are stored in python33.dll.

[eudoxos]

Storing several modules in one .so file offloads some issues which have to be dealt with manually (like one module needing symbols from other module, or even cross-dependencies) to the linker. In any case, unless forbidden and signalled, it should be supported.

I set version to 3.4 and component to "Interpreter core", as you suggested, it is probably more appropriate.

How do I write test which requires a custom module to be compiled?

[asvetlov]

Amaury, I'm ok with pushing several modules into python33.dll or embedding it directly into executable.
For standard so/dll files you have to use different file names for modules to make regular import statement work. It can be done via symlink/hardlink, but looks a bit hairy.

Václav, python already have _testbuffer and _testcapi modules used for testing only. I think you can add yet another one.

[bitdancer]

There is an example in the test suite somewhere. Probably in the distutils tests. Search for xxmodule...but you'll need to create your own source. I'd see if you can write it out from the test rather than checking in another data file, but a data file is an option if needed.

[eudoxos]

@andrew: I was using symlinks to achieve this, until I found recently that some exotic systems (read: windows) have still troubles there, like not letting non-admins create symlinks.