PEP 489 -- Multi-phase extension module initialization

[encukou]

BPO	24268
Nosy	@brettcannon, @ronaldoussoren, @ncoghlan, @scoder, @larryhastings, @encukou, @ericsnowcurrently, @zware, @zooba, @Carreau
Files	pep0489.patches pep0489.patch pep0489-multiphase-extension-module-import-v2.patch: Ready to commit (refleak in test_importlib) refleak-fix.patch fix-dynload-init-name.patch fix-pep489-submodule.patch fix-short-names.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2015-05-26.12:00:28.742>
created_at = <Date 2015-05-23.07:53:42.895>
labels = ['type-feature']
title = 'PEP 489 -- Multi-phase extension module initialization'
updated_at = <Date 2015-06-03.22:31:45.117>
user = 'https://github.com/encukou'

bugs.python.org fields:

activity = <Date 2015-06-03.22:31:45.117>
actor = 'petr.viktorin'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2015-05-26.12:00:28.742>
closer = 'ncoghlan'
components = []
creation = <Date 2015-05-23.07:53:42.895>
creator = 'petr.viktorin'
dependencies = []
files = ['39470', '39471', '39474', '39475', '39477', '39493', '39546']
hgrepos = []
issue_num = 24268
keywords = ['patch']
message_count = 40.0
messages = ['243893', '243894', '243899', '243910', '243911', '243913', '243916', '243917', '243918', '243919', '243920', '243921', '243927', '243928', '243938', '243941', '243942', '243943', '243947', '243948', '243949', '243950', '243951', '243953', '243954', '243955', '243956', '243957', '243958', '244008', '244029', '244034', '244038', '244100', '244378', '244382', '244383', '244404', '244405', '244790']
nosy_count = 12.0
nosy_names = ['brett.cannon', 'ronaldoussoren', 'ncoghlan', 'scoder', 'larry', 'petr.viktorin', 'BreamoreBoy', 'python-dev', 'eric.snow', 'zach.ware', 'steve.dower', 'mbussonn']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24268'
versions = ['Python 3.5']

[encukou]

Here is the implementation for the recently accepted PEP-489.

Tested on Linux.

[encukou]

And here are all changes in a single patch.

[ncoghlan]

I'll get this merged tonight so we make the beta1 deadline, but I expect the initial docs to be fairly rudimentary and requiring further updates.

[ncoghlan]

Attached patch is the one I'm looking to commit, but -R 3:3 shows significant reference leaks in test_importlib, and possible problems in other tests as well.

I'm about to revert it to see if there were any pre-existing refleak issues before applying this.

[ncoghlan]

Initial implementation checked in at https://hg.python.org/cpython/rev/e729b946cc03

Larry, FYI regarding the refleak in test_importlib I just committed: as described in the commit message, I'm pretty sure it's a real refleak in the current PEP-489 implementation, it's just a beast to track down because there are a lot of moving parts when it comes to module cleanup, especially for extension modules.

Aside from that, the tests all pass, so I hope you're OK with my checking it in its current state for beta 1, with the aim of fixing the bug for beta 2 (I would have asked first, but time zones intervened - I'm about to head to bed, and I expect you'll have closed 3.5 to new features by the time I get up)

[ncoghlan]

Regarding the extraneous whitespace changes in modsupport.h, those are courtesy of running "make patchcheck" as part of preparing the commit.

[encukou]

Fix some refleaks

• one in PyModule_FromDefAndSpec2 - this is my fault
• one in PyType_FromSpecWithBases - I guess this is
  the first time PEP-384 built-in types are GC'd
• one in the new PEP-489 tests

There's still one more in the new testing module, _testmultiphase. I'm working on it.

[python-dev]

New changeset 7f2e6f236202 by Nick Coghlan in branch 'default':
Issue bpo-24268: Address some PEP-489 refleaks
https://hg.python.org/cpython/rev/7f2e6f236202

[ncoghlan]

Thanks, that reduces the refleaks shown for "./python -m test -R3:3 test_importlib" from 102 to 50 for me.

However, I suspect there may still be a leak in the machinery, as I'm still seeing the total number of objects growing when importing the array module and then dropping the references to it:

$ ./python -X showrefcount
Python 3.5.0a4+ (default:e729b946cc03+, May 24 2015, 00:58:18) 
[GCC 5.1.1 20150422 (Red Hat 5.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
[54384 refs, 15894 blocks]
>>> import array; del array; del sys.modules['array']
[54654 refs, 15974 blocks]
>>> import array; del array; del sys.modules['array']
[54695 refs, 15986 blocks]
>>> import array; del array; del sys.modules['array']
[54736 refs, 15997 blocks]
>>> import array; del array; del sys.modules['array']
[54777 refs, 16008 blocks]
>>> import array; del array; del sys.modules['array']
[54818 refs, 16019 blocks]
>>> import array; del array; del sys.modules['array']
[54859 refs, 16030 blocks]

[ncoghlan]

Once we sort these out, any that impact Python 3.4 (like the PyType_FromSpecAndBases one) should be backported to the maintenance branch.

I wouldn't assume the one you found in PyType_FromSpecAndBases is the only one, though - I believe the tests for this PEP are managing to exercise parts of the module cleanup machinery that it's never been practical to properly test in the past.

And now I really am heading to bed :)

[encukou]

The array module is good if you *really* drop references:

$ ./python -X showrefcount
Python 3.5.0a4+ (default, May 23 2015, 16:44:38) 
[GCC 4.9.2 20150212 (Red Hat 4.9.2-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, gc
[54618 refs, 15960 blocks]
>>> import array; del array; del sys.modules['array']; gc.collect()
6
[54851 refs, 15973 blocks]
>>> import array; del array; del sys.modules['array']; gc.collect()
6
[54851 refs, 15973 blocks]
>>> import array; del array; del sys.modules['array']; gc.collect()
6
[54851 refs, 15973 blocks]

There is a cycle between a each built-in function and its module. Modules aren't optimized for being unloaded.

[encukou]

FWIW, the remaining refleak occurs when unloading an extension module object. This is something that wasn't possible before PEP-489 -- extension modules were never deleted.

[python-dev]

New changeset a811f5561c99 by Steve Dower in branch 'default':
Issue bpo-24268: Fixes generation of init import name on Windows.
https://hg.python.org/cpython/rev/a811f5561c99

[zooba]

For the sake of getting things running again, I went ahead and fixed the format string in dynload_win.c. Feel free to change it again if that isn't what was intended.