Cython compiler run crashes CPython 3.1.1 and 3.2

[scoder]

BPO	7173
Nosy	@amauryfa, @pitrou, @scoder, @vstinner, @benjaminp, @florentx
Files	crash-patch.patch: patch against Cython that shows the crash yield_crasher.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-03-07.17:11:07.583>
created_at = <Date 2009-10-20.09:21:15.666>
labels = ['interpreter-core', 'type-crash']
title = 'Cython compiler run crashes CPython 3.1.1 and 3.2'
updated_at = <Date 2010-03-07.17:11:07.582>
user = 'https://github.com/scoder'

bugs.python.org fields:

activity = <Date 2010-03-07.17:11:07.582>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2010-03-07.17:11:07.583>
closer = 'benjamin.peterson'
components = ['Interpreter Core']
creation = <Date 2009-10-20.09:21:15.666>
creator = 'scoder'
dependencies = []
files = ['15168', '16225']
hgrepos = []
issue_num = 7173
keywords = ['patch']
message_count = 20.0
messages = ['94268', '94269', '95590', '95594', '95597', '95599', '95603', '95608', '98585', '98608', '98611', '99357', '99360', '99362', '99366', '99368', '99370', '99372', '99419', '100593']
nosy_count = 7.0
nosy_names = ['amaury.forgeotdarc', 'pitrou', 'scoder', 'vstinner', 'benjamin.peterson', 'Arfrever', 'flox']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue7173'
versions = ['Python 3.1', 'Python 3.2']

[scoder]

Running the Cython compiler under Python 3.1.1 and 3.2 (SVN) corrupts
PyThreadState->exc_value by leaving a dead reference. Printing the value
then leads to a crash.

This bug is about plain Python code, no Cython built extension modules
involved.

Steps to reproduce the problem:

• get Cython from http://hg.cython.org/cython-devel (see the bz2/zip/gz
  links on the left to get an archive without doing a checkout)

• apply the attached patch, which simply prints sys.exc_info() during
  the compiler run at a place where it's known to be corrupted

• execute the following to run a single test in the test suite:

  python3.1 runtests.py --no-cpp --no-pyregr --no-doctest -vv \
  'compile\.first_assignment'

• expect a crash after printing "HERE1"

Before exc_info gets corrupted, we make heavy use of generators to
traverse the parse tree (see Cython/Compiler/TreePath.py), while being
inside of a recursive traversal of the tree already (see the
"VisitorTransform" class and its base class in
Cython/Compiler/Visitor.py). The code section that links the two is in
the class "TreeAssertVisitor" at the end of Cython/TestUtils.py, where
the patch applies. As exc_info doesn't get corrupted during the normal
recursive tree transformation traversals before that, the generator
usage in TreePath.py is likely related to the crash.

The crash was found and reproduced under two different Linux x86 systems.

[scoder]

I should add that the crash doesn't necessarily happen during the first
test run, which also converts the Cython source to Py3 using 2to3.
However, once that's done, running the test a second time crashes reliably.

[pitrou]

Have you tried using 3.1.0 and even 3.0?

[scoder]

I hadn't, but it looks like the 2to3-ed Cython also runs on 3.0 now, so
I tested that, but I failed to get the procedure below to crash for me.
And that's both in 3.0 *and* 3.1.1! :-/

But I can still provoke the crash in 3.0, 3.0.1, 3.1.1 and the latest
3.2 when running the test suite normally, so it's still there, and it's
been there for a while.

At least a debug build of the latest py3k SVN rev. 76441 (3.2a0) seems
to crash reliably for the latest Cython (cython-devel, rev.
76a814a1fc57) with the attached crash patch applied. Running the
following crashes after printing the "HERE1":

python3.2 runtests.py --no-cpp --no-pyregr --no-doctest \
     --no-cpp --no-fork --no-refnanny -vv first_assignment

with the error:
"""
python3.2: Objects/unicodeobject.c:885: PyUnicodeUCS4_FromFormatV:
Assertion `obj' failed.
Aborted
"""
which occurs when it's trying to print() the dead exception. Again, no
Cython modules have been loaded up to this point, so this is still pure
Python code that crashes the interpreter.

[amauryfa]

I don't reproduce the problem on Windows. But the class NodeTypeWriter is
not even used at all; did I miss something? The test builds a python
extension and runs it, successfully it seems.

[scoder]

The patch is supposed to apply near the end of the class
TreeAssertVisitor at the end of the file Cython/TestUtils.py, not in the
class NodeTypeWriter.

And the test doesn't run (or even import) the extension, it just builds it.

[amauryfa]

Sorry, my mistake. Now the prints are there, but the test run without
error:

Running tests against Cython 0.12.rc1
Python 3.2a0 (py3k, Nov 22 2009, 12:04:23) [MSC v.1500 32 bit (Intel)]
HERE1
(None, None, None)
HERE2
HERE1
(None, None, None)
HERE2
HERE1
(None, None, None)
HERE2
building 'first_assignment' extension
[...More output...]

[amauryfa]

I reproduce the crash on Linux. Some debug prints showed that the failing
exception object is "tp_clear'ed", because it belongs to a reference
cycle. Now if it is cleared it should not be reachable...

[vstinner]

I'm unable to reproduce the crash. Can someone write a shorter code reproducing the issue? I downloaded Cython tip and applied the patch attached to this issue.

I tried py3k, py3k compiled in pydebug mode, python 3.1 branch: none crashed. Should I clear generated files before running a new test? If yes, how?

[scoder]

Looks like this is one of those bugs that run away when you look too close...

I can get it to crash reliably for me with the latest py3k branch (and all Python 3.x release versions) when I run the test suite completely, so here's a new recipe. The intention is to keep CPython from importing Cython built modules to make sure it's not related to the C code that Cython generates. I thought it was when I hit this problem ages ago (1 or 2 years?), but I can now assure you, it's not. To reproduce, run:

python3.2 runtests.py --no-cpp --no-pyregr --no-doctest \
      --no-fork --cython-only --no-refnanny -vv 'run\.'

The "--cython-only" switch makes sure the modules are not run through the C compiler, so no binary modules are built and only the pure Python code of the Cython compiler runs here. The output is a little badly formatted, but it basically prints the name of the modules it compiles and crashes after a while (also without the patch).

You don't have to do anything for cleanup, the test runner does that.

Writing a shorter code snippet is really not easy as there seem to be various things involved here, apparently including exception handling, recursion and generators.

[scoder]

I'll add a couple of comments about the relevant parts of the code that appears to trigger the crash.

The code runs through the parse tree and applies transformations to it.

1. For node matching, we use a dispatcher (in Visitor.py) that looks up methods for node class names (or the nearest matching class in its MRO hierarchy), e.g. a plain "Node" type would be handled by a "visit_Node" method, whereas an ExprNode(Node) type would match "visit_ExprNode" if it exists and "visit_Node" otherwise. The handler then recursively calls back into the dispatch code to visit its children.

For performance reasons, we sometimes reuse existing methods for this through assignment, e.g. instead of reimplementing the default fallback "visit_Node" all over the place, we assign an existing superclass method to it as in "visit_Node = VisitorTransform.recurse_to_children". This may or may not have an impact on this problem (it shouldn't - normally).

So far, all of this appears to work perfectly fine also in Py3. It's the next step that seems to be required to trigger the crash.

2. To specify assertions in our test suite, we use an XPath-like language for node tests (poor man's Schematron, sort-of). The implementation in TreePath.py (based on "ElementPath" in ElementTree 1.3) makes heavy use of generators to traverse the tree. It is used by the TreeAssertVisitor in TestUtils.py, which is itself a recursive tree visitor (as described above).

So the situation is as follows. Cython traverses the parse tree recursively, several times in a pipeline, dispatching to methods that further recurse into the tree. One of the pipeline steps is the tree assertion visitor, which, on matching functions, applies the generator-based path language to the tree, that tries to find nodes by iterating over a chain of generators.

At some point in this process, somehow, sys.exc_info gets corrupted, and the doctest/unittest framework crashes the runtime with a segfault when trying to print an exception (raised unexpectedly by the "pure" test).

I hope this makes it clearer a) what happens and b) why it's not easy to cut this down into a simple test case. As the previous comments show, it's already hard enough to reproduce it reliably with a reduced test setup.

Please do not hesitate to contact me if you need any further information.

[pitrou]

It's still reproduceable, but without a short easy-to-study snippet it's difficult to do anything.

[scoder]

I tried several times to debug it myself, but I don't understand the exception cleanup macros in ceval.c (UNWIND_EXCEPTION_HANDLER and friends, new in Py3). If someone can get me set up to debug them, I can give it another shot. I assume there are a couple of invariants involved in them? Is there any interaction with generators that I should know about?