Intermittent buildbot failure and timeout on Android aarch64

[mhsmith]

Bug report

Happened twice in the last two days:

• https://buildbot.python.org/#/builders/1594/builds/84
• https://buildbot.python.org/#/builders/1594/builds/110

In both cases, the sequence of events is:

• TestAndroidOutput.test_bytes fails due to not finding the test start marker line in the logcat. A "threading._dangling was modified" warning is also shown.
  • In build 84, none of the output from the first attempt of test_bytes or test_str is shown in the buildbot log, even though test_str passed. The first line of test_rate_limit is also missing ("Initial message to reset"), but the body of that test is present.
  • In build 110, only the test_bytes output is missing.

test_bytes (test.test_android.TestAndroidOutput.test_bytes) ... FAIL
test_str (test.test_android.TestAndroidOutput.test_str) ... ok
test_rate_limit (test.test_android.TestAndroidRateLimit.test_rate_limit) ... ok

======================================================================
FAIL: test_bytes (test.test_android.TestAndroidOutput.test_bytes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.14/test/test_android.py", line 67, in assert_log
    line = self.logcat_queue.get(timeout=(deadline - time()))
  File "/data/user/0/org.python.testbed/files/python/lib/python3.14/queue.py", line 212, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.14/test/test_android.py", line 57, in setUp
    self.assert_log("I", tag, message, skip=True, timeout=5)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.14/test/test_android.py", line 69, in assert_log
    self.fail(f"line not found: {expected!r}")
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: line not found: 'test.test_android.TestAndroidOutput.test_bytes 1727286696.2114007'

----------------------------------------------------------------------
Ran 3 tests in 5.061s

FAILED (failures=1)
Warning -- threading._dangling was modified by test_android
Warning --   Before: {<weakref at 0x7414441df0; to 'threading._MainThread' at 0x73aea63cb0>}
Warning --   After:  {<weakref at 0x7414dd9d00; to 'threading._MainThread' at 0x73aea63cb0>, <weakref at 0x7414dda110; to 'threading.Thread' at 0x7414d36450>} 
test test_android failed

• The test passes on the second attempt. All output appears to be present and correct, except that the "Initial message to reset" is combined with "Line 000" because the test neglects to write a newline between them, but that's probably not relevant.
• Python reports the overall result as "FAILURE then SUCESS".
• But then it apparently hangs until the worker times out after "1800 seconds without output".

Linked PRs

• gh-124666: Improve thread cleanup in test_android #131427
• [3.13] gh-124666: Improve thread cleanup in test_android (GH-131427) #131433

[mhsmith]

It's happened 3 more times in the last week, both on the main and 3.13 branches:

• https://buildbot.python.org/#/builders/1594/builds/745
• https://buildbot.python.org/#/builders/1594/builds/754
• https://buildbot.python.org/#/builders/1590/builds/283

It's only been seen on aarch64, but this probably isn't because of an architecture difference, but because the aarch64 runner is significantly faster than the AMD64 one.

The reported failure varies between test_bytes or test_str. I still don't know why it's failing, but my guess is that the threading._dangling message and the hang at the end are both caused by the continued existence of the thread which the test spawns to read the logcat. If this thread cleanup was fixed, or the thread was marked as daemon, then a pass on the second attempt would set the buildbot status to a warning rather than an error, and it wouldn't post comments on GitHub anymore.