Python 3.4.3 regression in ctypes.frombuffer()

[JakeMont]

BPO	25498
Nosy	@vstinner, @bitdancer, @skrah, @vadmium, @zware, @serhiy-storchaka, @eryksun
Files	pythoncrash.py: This is python code that will trigger the crash ctypes_crash.py: Trigger a crash, isolated to ctypes from_buffer. ctypes_from_buffer_2.patch: Use PyMemoryView_FromObject instead of PyMemoryView_FromBuffer. ctypes_from_buffer_3.patch ctypes_from_buffer_4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-11-13.23:07:40.457>
created_at = <Date 2015-10-28.18:06:12.369>
labels = ['ctypes', 'type-crash']
title = 'Python 3.4.3 regression in ctypes.frombuffer()'
updated_at = <Date 2015-11-16.16:46:34.614>
user = 'https://bugs.python.org/JakeMont'

bugs.python.org fields:

activity = <Date 2015-11-16.16:46:34.614>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2015-11-13.23:07:40.457>
closer = 'martin.panter'
components = ['ctypes']
creation = <Date 2015-10-28.18:06:12.369>
creator = 'JakeMont'
dependencies = []
files = ['40876', '40883', '40890', '41009', '41025']
hgrepos = []
issue_num = 25498
keywords = ['patch', '3.4regression']
message_count = 22.0
messages = ['253613', '253616', '253617', '253639', '253646', '253682', '253702', '253749', '253756', '253758', '253776', '253789', '253801', '253950', '254472', '254473', '254577', '254593', '254596', '254628', '254633', '254740']
nosy_count = 11.0
nosy_names = ['vstinner', 'jnoller', 'r.david.murray', 'skrah', 'python-dev', 'sbt', 'martin.panter', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'JakeMont']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue25498'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']

[JakeMont]

We are seeing a consistent "invalid memory access" crash in Python3.4.3.

*Test setup*
This is occurring on Ubuntu 14.04.1 LTS. It is occurring on multiple unrelated installs, but has not been tested with any other operating system. It occurs using the "Python 3.4.3 (default, Oct 14 2015, 20:28:29)" that is installed using apt-get. It does *not* appear to occur using Python 3.4.0. Other versions of python were not tested.

*Reproducing*
I was able to reduce the code needed to a pretty minimal python program, which is attached. Running this as "pyton3 ./pythoncrash.py" will trigger the crash, and a core dump if enabled.

If run under python3.4.3-dbg The error is:

python3.4-dbg: ../Objects/memoryobject.c:115: mbuf_dealloc: Assertion `self->exports == 0' failed.

In some versions of this code, it had to be run twice, since it would only occur when the pyc files were already created. The attached version does not appear to have this behavior, but run it twice, at least, if you do not see the crash.

The code supplied is "minimal", in that removing any line will cause the crash not to occur.

*More clues*
I suspect that the key is the multiprocessing.Value object, and its destruction. The other "import" statements may just be happenstance needed to create the correct memory conditions for the crash. I was able to start "stubbing out" the actual code in those imported packages, and the crash continued as long as the number of functions, classes, and class dependencies were maintained. But that became too tedious. In earlier versions of the test code, adding or removing significant numbers of function or classes could make the crash not manifest.

*Stack trace*
Running under python3.4.3-dbg, gdb gives the following stack trace:

#0 0x00007fc0425b4cc9 in __GI_raise (sig=sig@entry=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fc0425b80d8 in __GI_abort () at abort.c:89
#2 0x00007fc0425adb86 in __assert_fail_base (
fmt=0x7fc0426fe830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x707199 "self->exports == 0",
file=file@entry=0x70715a "../Objects/memoryobject.c", line=line@entry=115,
function=function@entry=0x708923 <PRETTY_FUNCTION.10355> "mbuf_dealloc") at assert.c:92
#3 0x00007fc0425adc32 in __GI___assert_fail (assertion=0x707199 "self->exports == 0",
file=0x70715a "../Objects/memoryobject.c", line=115,
function=0x708923 <PRETTY_FUNCTION.10355> "mbuf_dealloc") at assert.c:101
#4 0x00000000004beb92 in mbuf_dealloc (self=0x7fc042068058) at ../Objects/memoryobject.c:115
#5 0x00000000004cd67b in _Py_Dealloc (op=<managedbuffer at remote 0x7fc042068058>)
at ../Objects/object.c:1749
#6 0x00000000004c0ec6 in memory_clear (self=0x7fc042026bf8) at ../Objects/memoryobject.c:1079
#7 0x00000000004409c3 in delete_garbage (collectable=0x7fff21065180,
old=0x9a4f40 <generations+64>) at ../Modules/gcmodule.c:866
#8 0x0000000000440f56 in collect (generation=2, n_collected=0x7fff210651e8,
n_uncollectable=0x7fff210651f0, nofail=0) at ../Modules/gcmodule.c:1032
#9 0x000000000044144e in collect_with_callback (generation=2) at ../Modules/gcmodule.c:1140
#10 0x00000000004421f2 in PyGC_Collect () at ../Modules/gcmodule.c:1616
#11 0x00000000004226a0 in Py_Finalize () at ../Python/pythonrun.c:607
#12 0x0000000000428f38 in Py_Exit (sts=0) at ../Python/pythonrun.c:2713
#13 0x0000000000426077 in handle_system_exit () at ../Python/pythonrun.c:1812
#14 0x00000000004260a2 in PyErr_PrintEx (set_sys_last_vars=1) at ../Python/pythonrun.c:1822
#15 0x0000000000425cef in PyErr_Print () at ../Python/pythonrun.c:1718
#16 0x00000000004255c4 in PyRun_SimpleFileExFlags (fp=0x29824c0,
filename=0x7fc042442b68 "./pythoncrash.py", closeit=1, flags=0x7fff21065520)
at ../Python/pythonrun.c:1611
#17 0x0000000000424418 in PyRun_AnyFileExFlags (fp=0x29824c0,
filename=0x7fc042442b68 "./pythoncrash.py", closeit=1, flags=0x7fff21065520)
at ../Python/pythonrun.c:1292
#18 0x000000000043e9fb in run_file (fp=0x29824c0, filename=0x28b62f0 L"./pythoncrash.py",
p_cf=0x7fff21065520) at ../Modules/main.c:319
#19 0x000000000043f781 in Py_Main (argc=2, argv=0x28b5020) at ../Modules/main.c:751
#20 0x000000000041e6d6 in main (argc=2, argv=0x7fff21065798) at ../Modules/python.c:69

The full core dump is available, if it would help.

[bitdancer]

The program does not crash for me on gentoo running python 3.4 tip (--with-pydebug). It does crash running gentoo's stock 3.4.3. So, perhaps this bug has already been fixed.

[zware]

I get the crash on OSX with recent builds of 3.4 and 3.6:

Assertion failed: (self->exports == 0), function mbuf_dealloc, file Objects/memoryobject.c, line 115.
Abort trap: 6

[vadmium]

Crashes on Linux x86-64 for me with and without --with-pydebug, with the default 3.6 branch, 3.5.0, and 3.4 tip. No extra debugging output though in the --with-pydebug cases though.

[vadmium]

I traced this down to part of revision 1da9630e9b7f (Issue bpo-22896). If I revert the changes to CDataType_from_buffer() at <https://hg.python.org/cpython/rev/1da9630e9b7f/#l4.3\>, the crash no longer happens. I suspect the offending code is this, that is trying to stash the original buffer protocol supporting object into a memory view:

mv = PyMemoryView_FromBuffer(&buffer);
if (mv == NULL) {
    PyBuffer_Release(&buffer);
    return NULL;
}
/* Hack the memoryview so that it will release the buffer. */
((PyMemoryViewObject *)mv)->mbuf->master.obj = buffer.obj;
((PyMemoryViewObject *)mv)->view.obj = buffer.obj;
//~ Py_INCREF(buffer.obj);
if (-1 == KeepRef((CDataObject *)result, -1, mv))
    result = NULL;

If I enable my INCREF() line it also stops the crash, but I guess at the expense of a memory leak.

[skrah]

We should ultimately do something like I suggested in msg235256.

Everything else is a hack (N.B.: the issues that 1da9630e9b7f
tried to fix were also hacks, so it isn't really the fault of
that commit).

[vadmium]

Before the bpo-22896 changes, PyObject_AsWriteBuffer() was used, which requests a buffer with the PyBUF_WRITABLE flag. Currently we use “w*” argument parsing format, which also uses PyBUF_WRITABLE. (Incidentally, I suspect the C-contiguity check is redundant for “w*”; non-contiguous buffers trigger the “read-write” error instead.)

Now Eryksun’s patch changes to PyMemoryView_FromObject(), which requests a buffer with the PyBUF_FULL_RO flag (the most liberal), and only then checks for writability and contiguity. Could this be a problem for some kind of object that returns different buffers depending on the request flags?

I agree that the existing buffer and memory view APIs don’t seem to be very practical and well understood. PyMemoryView_FromBuffer() is also used in the “io” module. It doesn’t even try to save a reference to the underlying buffers, meaning it is possible for Python code to write into freed memory by saving the memory view object passed to the readinto() method.

[serhiy-storchaka]

Before committing any solution we first should have understood the cause of the crash. The peculiarity of example ctypes_crash.py is that the argument of ctypes.c_char.from_buffer is a memoryview. With current code we have following chain (the list creates a reference loop with itself):

list->memoryview->ManagedBuffer->memoryview->ManagedBuffer->bytearray

Using PyMemoryView_FromObject or hypothetical PyMemoryView_FromObjectEx we could get shorter chain:

list->memoryview->ManagedBuffer->bytearray

May be the cause not in current hack, but in memoryview that can't survive with breaking long chain? Or may be we just are lucky in latter case?

I share Martin's doubts about writability in Eryksun’s patch.

[skrah]

If memoryview B is created from memoryview A, then B must be
registered with the same ManagedBuffer as A, otherwise the
whole scheme breaks down (PyMemoryView_FromObject() performs
this check).

PyMemoryView_FromBuffer() is really a legacy function that
should be only used for creating temporary views within a
function. As far as I know, it has usually been used that
way from the start (since 3.0).

[skrah]

Per the docs the readonly flag must be consistent for all consumers,
so checking for writability after getting the view should be okay in
principle.