TODO list when PEP 573 "Module State Access from C Extension Methods" will be implemented

[vstinner]

BPO	40137
Nosy	@rhettinger, @vstinner, @phsilva, @encukou, @serhiy-storchaka, @corona10, @pablogsal, @miss-islington, @shihai1991, @erlend-aasland
PRs	bpo-40077: Convert _jsonmodule to use PyType_FromSpec. #19177 [WIP] bpo-40137: Use pep573 in functools extension module. #20055 bpo-40137: Convert _functools module to use PyType_FromModuleAndSpec. #23405 bpo-40137: Fix refleak in _functools_exec() #24006 bpo-40137: Micro-optimize _PyType_GetModuleByDef() #25393 bpo-40137: Move state lookups out of the critical path #25492 bpo-40137: _PyType_GetModuleByDef() doesn't check tp_flags #25504 bpo-40137: Optimize _PyType_GetModuleByDef() loop #25505 bpo-40137: Add pycore_moduleobject.h internal header #25507
Files	patch.diff: Move get_functools_state_by_type() to lru_cache_new()

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-21.23:28:53.908>
created_at = <Date 2020-04-01.15:08:23.822>
labels = ['type-feature', 'library', '3.10']
title = 'TODO list when PEP 573 "Module State Access from C Extension Methods" will be implemented'
updated_at = <Date 2021-05-06.00:04:16.932>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-05-06.00:04:16.932>
actor = 'rhettinger'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-21.23:28:53.908>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2020-04-01.15:08:23.822>
creator = 'vstinner'
dependencies = []
files = ['49958']
hgrepos = []
issue_num = 40137
keywords = ['patch']
message_count = 28.0
messages = ['365482', '365486', '368852', '380668', '380669', '383993', '384036', '384037', '384043', '384048', '384049', '384056', '390756', '390758', '390759', '390768', '390769', '391004', '391005', '391045', '391547', '391549', '391550', '391553', '391557', '391560', '391561', '393056']
nosy_count = 10.0
nosy_names = ['rhettinger', 'vstinner', 'phsilva', 'petr.viktorin', 'serhiy.storchaka', 'corona10', 'pablogsal', 'miss-islington', 'shihai1991', 'erlendaasland']
pr_nums = ['19177', '20055', '23405', '24006', '25393', '25492', '25504', '25505', '25507']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40137'
versions = ['Python 3.10']

[vstinner]

In bpo-1635741, many C extension modules are converted to PEP-489 multiphase initialization and/or modified to get a module state. Problem: the module state cannot be accessed in some functions, and so some of these changes had to workaround the fact that PEP-573 "Module State Access from C Extension Methods" is not implemented yet.

This issue tracks C extension modules which should be modified once PEP-573 will be implemented:

• _functools: Py_CLEAR(kwd_mark); is commented in _functools_free()
  See commit eacc074

• _abc: abc_invalidation_counter is shared by all module instances. abc_data_new() requires access to abc_invalidation_counter but it doesn't have access to the module state. abc_invalidation_counter should be moved to the module state.

[vstinner]

Another minor issue: "assert(PyScanner_Check(self));" and "assert(PyEncoder_Check(self));" were removed by commit 33f15a1 when _json module got a module state.

scanner_traverse(), scanner_clear(), encoder_traverse() and encoder_clear() cannot get the module state of a module using PEP-489 multiphase initialization.

Similar issue in scanner_call() and encoder_call().

I'm not sure that the PEP-573 provide a solution for this exact issue, since the intent of the assertion is to check the type of the argument. The PEP-573 gives access to the module state through types. But here we are not sure that the object type is the expected type... :-)

Again, it's a minor issue, it can be ignored. That's why I accepted to merge the commit. Technically, I don't see how we could get the wrong type in practice.

[vstinner]

I consider that all things that could be done have already been done, so I close the issue. Thanks for Hai and Dong-hee who helped ;-)

• _functools: Py_CLEAR(kwd_mark); is commented in _functools_free()

Sadly, PEP-573 implementation is not complete enough to use it in _functools: see PR 20055.

• _abc: abc_invalidation_counter

Fixed in bpo-40566 by:

commit 77c6146
Author: Dong-hee Na <donghee.na92@gmail.com>
Date: Sat May 9 17:31:40 2020 +0900

bpo-40566: Apply PEP-573 to abc module (GH-20005)

scanner_traverse(), scanner_clear(), encoder_traverse() and encoder_clear()

tp_clear slot cannot get the defining type (PEP-573 may be extended later to allow that).

Using PyType_GetModule(Py_TYPE(self)) to access types stored in the module state and then compare Py_TYPE(self) to these types... looks badly broken :-) If we get the wrong type, PyType_GetModule(Py_TYPE(self)) will return the wrong module and so we cannot get the json state from the wrong module... It's a chicken-and-egg issue :-)

I think that it's not worth it to attempt to add back these assertions. It's very unlikely to get the wrong types in practice.

[encukou]

It should be possible to get them back using _PyType_GetModuleByDef.

[vstinner]

Oh nice, in this case, I reopen my issue :-)