test_thread leaks a core dump on PPC64 AIX 3.x

[vstinner]

BPO	36273
Nosy	@vstinner, @aixtools, @tirkarthi
Files	recurse.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-02-11.00:02:54.962>
created_at = <Date 2019-03-12.16:14:04.799>
labels = ['3.8', 'tests']
title = 'test_thread leaks a core dump on PPC64 AIX 3.x'
updated_at = <Date 2020-03-25.23:15:51.389>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-03-25.23:15:51.389>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-02-11.00:02:54.962>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2019-03-12.16:14:04.799>
creator = 'vstinner'
dependencies = []
files = ['48525']
hgrepos = []
issue_num = 36273
keywords = []
message_count = 16.0
messages = ['337772', '337775', '338178', '338975', '340391', '340395', '343012', '348844', '348845', '348846', '348854', '348861', '348949', '348950', '361762', '365030']
nosy_count = 3.0
nosy_names = ['vstinner', 'Michael.Felt', 'xtreak']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36273'
versions = ['Python 3.8']

[vstinner]

PPC64 AIX 3.x:
https://buildbot.python.org/all/#/builders/10/builds/2224

0:12:47 [160/420/1] test_threading failed (env changed)
...
Ran 158 tests in 12.023s

OK (skipped=1)
Warning -- files was modified by test_threading
Before: []
After: ['core']

I don't know if it's a regression or if it occurred in the past.

Michael: Can you please try to reproduce the issue? Any idea if it's new or not?

You can try to identify which test creates a core dump using:

$ ./python -m test.bisect_cmd --fail-env-changed test_thread -v

The command is supposed to find which test creates the "core" file.

[tirkarthi]

See also https://bugs.python.org/issue35828#msg337076 where test_multiprocessing_fork seemed to leave a core dump. Maybe one test core dumps and leaves the report to the other causing env changed?

[vstinner]

David Edelsohn: Would you mind to have a look at this issue?

[aixtools]

I have looked at this briefly.

I would consider it a regression, or a spurious incident as the bot run before (https://buildbot.python.org/all/#/builders/10/builds/2223) and after (https://buildbot.python.org/all/#/builders/10/builds/2225) do not show this error. It may also be spurious - as there are other users on that bot system and the cause could be a side-effect of something else.

FYI: The error message is something similar to what I see when building using xlc - there is a core dump - but I cannot get a copy of the core dump to evaluate it.

See: https://bugs.python.org/issue35828#msg336134 - where I ask for a hint on how to get the code that reports the ENV change to save/copy a core dump.

[vstinner]

The same bug is back:
https://buildbot.python.org/all/#/builders/10/builds/2443

Warning -- files was modified by test_threading
Before: []
After: ['core']

[aixtools]

On 17/04/2019 14:02, STINNER Victor wrote:

STINNER Victor <vstinner@redhat.com> added the comment:

The same bug is back:
https://buildbot.python.org/all/#/builders/10/builds/2443

Warning -- files was modified by test_threading
Before: []
After: ['core']

----------

What can I do to "copy" the core dump before normal testing removes it?

I have had a similar message regarding the multiprocessing tests, but in
those cases I expect it is the "client" thread complaining about a
"server" thread crash.

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue36273\>

---

[aixtools]

Again - how can I get the core (dump) mentioned in the error message. When I force this situation I have several core dumps - not "the one".

[vstinner]

Again - how can I get the core (dump) mentioned in the error message. When I force this situation I have several core dumps - not "the one".

You have to contact the owner of the buildbot worker. From a build, you can click on the Worker tab. For PPC64 AIX, I see:

David Edelsohn <dje.gcc@gmail.com>

[vstinner]

test_threading has a similar issue, still on PPC64 AIX 3.x:
https://buildbot.python.org/all/#builders/10/builds/3171

test_recursion_limit (test.test_threading.ThreadingExceptionTests) ... FAIL
test_releasing_unacquired_lock (test.test_threading.ThreadingExceptionTests) ... ok
test_start_thread_again (test.test_threading.ThreadingExceptionTests) ... ok
test_init_immutable_default_args (test.test_threading.TimerTests) ... ok

======================================================================
FAIL: test_recursion_limit (test.test_threading.ThreadingExceptionTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_threading.py", line 1088, in test_recursion_limit
    self.assertEqual(p.returncode, 0, "Unexpected error: " + stderr.decode())
AssertionError: -11 != 0 : Unexpected error:

---

Ran 168 tests in 12.623s

FAILED (failures=1)
Warning -- files was modified by test_threading
Before: []
After: ['core']

[vstinner]

Same test_threading crash on POWER6 AIX 3.x:
https://buildbot.python.org/all/#/builders/161/builds/1430

This build was triggered by the change of bpo-18049 which is supposed to only impact macOS, but it does touch test_recursion_limit... can it be related to the regression?

1a057ba

test_recursion_limit (test.test_threading.ThreadingExceptionTests) ... FAIL
test_releasing_unacquired_lock (test.test_threading.ThreadingExceptionTests) ... ok
test_start_thread_again (test.test_threading.ThreadingExceptionTests) ... ok
test_init_immutable_default_args (test.test_threading.TimerTests) ... ok

======================================================================
FAIL: test_recursion_limit (test.test_threading.ThreadingExceptionTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_threading.py", line 1088, in test_recursion_limit
    self.assertEqual(p.returncode, 0, "Unexpected error: " + stderr.decode())
AssertionError: -11 != 0 : Unexpected error: 

---

Ran 168 tests in 12.623s

FAILED (failures=1)
Warning -- files was modified by test_threading
Before: []
After: ['core']

[aixtools]

On 01/08/2019 11:15, STINNER Victor wrote:

FAILED (failures=1)
Warning -- files was modified by test_threading
Before: []
After: ['core']

Thanks. I'll look again (on my bot) and other test systems.

What I assume has not been clear about my question - from the output I
see that there is, somewhere, a file named 'core' - but I can never find
it - and my assumption is that the test environment is cleaning up after
itself.

So, rephrased - is there a way to stop running test cycle when there is
an environment error, especially with a core dump, so that it can be
examined for further clues.

Michael

[vstinner]

"""
What I assume has not been clear about my question - from the output I
see that there is, somewhere, a file named 'core' - but I can never find
it - and my assumption is that the test environment is cleaning up after
itself.
"""

Right, regrtest removes all temporary directories used for tests. You should try to extract the code run by the test and run it manually.

For example for test_recursion_limit(), you can use attached recurse.py.

[aixtools]

resolved via bpo-18049