PyMem_Malloc(): check that the GIL is hold in debug hooks

[vstinner]

BPO	26563
Nosy	@vstinner
Files	pymem_gil.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-03-17.08:20:02.723>
created_at = <Date 2016-03-14.22:52:53.562>
labels = []
title = 'PyMem_Malloc(): check that the GIL is hold in debug hooks'
updated_at = <Date 2016-03-18.10:04:44.032>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2016-03-18.10:04:44.032>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2016-03-17.08:20:02.723>
closer = 'vstinner'
components = []
creation = <Date 2016-03-14.22:52:53.562>
creator = 'vstinner'
dependencies = []
files = ['42165']
hgrepos = []
issue_num = 26563
keywords = ['patch']
message_count = 11.0
messages = ['261787', '261850', '261851', '261854', '261865', '261866', '261867', '261874', '261875', '261902', '261954']
nosy_count = 2.0
nosy_names = ['vstinner', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue26563'
versions = ['Python 3.6']

[vstinner]

With the issue bpo-26558, debug hooks of PyObject_Malloc() now checks that the GIL is hold.

I left PyMem_Malloc() unchanged, because I am not 100% sure that it's ok yet to implement the same check in PyMem_Malloc(). So I opened this issue.

Python 3 doc explicitly asks that the GIL is hold to call PyMem_Malloc():
https://docs.python.org/dev/c-api/memory.html#memory-interface

Python 2 doc doesn't say anything about the GIL:
https://docs.python.org/2/c-api/memory.html#memory-interface

Usually, functions prefixed by "Py" require the GIL to be hold.

In practice, currently PyMem_Malloc() is implemented with malloc() which is thread-safe.

In the issue bpo-26249, I tested numpy, lxml, Pillow and cryptography with all debug hooks enabled, including GIL checks in PyMem_Malloc() and PyObject_Malloc(). I only found one bug in numpy: numpy/numpy#7404

Attached patch changes debug hooks on PyMem_Malloc() to ensure that the GIL is hold.

[python-dev]

New changeset 58644086c195 by Victor Stinner in branch 'default':
Fail if PyMem_Malloc() is called without holding the GIL
https://hg.python.org/cpython/rev/58644086c195

[vstinner]

I pushed the patch (I added an unit test).

If too many users complain, we can revert the change since it's now a small change.

[python-dev]

New changeset ae76a1046bb9 by Victor Stinner in branch 'default':
Fix usage of PyMem_Malloc() in os.stat()
https://hg.python.org/cpython/rev/ae76a1046bb9

[python-dev]

New changeset 5b4f173f0690 by Victor Stinner in branch 'default':
faulthandler now works in non-Python threads
https://hg.python.org/cpython/rev/5b4f173f0690

New changeset d6d64168ee8c by Victor Stinner in branch 'default':
Py_FatalError: disable faulthandler earlier
https://hg.python.org/cpython/rev/d6d64168ee8c

[python-dev]

New changeset 959e58cfbde9 by Victor Stinner in branch 'default':
Fix usage of PyMem_Malloc() in overlapped.c
https://hg.python.org/cpython/rev/959e58cfbde9

[python-dev]

New changeset 73d8adc0d5ea by Victor Stinner in branch '3.5':
Fix usage of PyMem_Malloc() in overlapped.c
https://hg.python.org/cpython/rev/73d8adc0d5ea

[vstinner]

Reopen. I found and fixed two bugs in os and _overlapped modules (_overlapped is used by asyncio on Windows).

[python-dev]

New changeset ce62df22f3bf by Victor Stinner in branch '3.5':
Py_FatalError: disable faulthandler earlier
https://hg.python.org/cpython/rev/ce62df22f3bf

[vstinner]

Hum ok, all buildbots look to be happy again :-) I close the issue (again).

[python-dev]

New changeset 7b079adb0774 by Victor Stinner in branch 'default':
Enhance documentation on malloc debug hooks
https://hg.python.org/cpython/rev/7b079adb0774