test_asyncio: test_cancel_gather_2() dangling thread

[vstinner]

BPO	37137
Nosy	@vstinner, @asvetlov, @pablogsal
PRs	bpo-37137: Fix test_asyncio: use TestCase.set_event_loop() #13779 bpo-37137, asyncio: add BaseEventLoop.wait_executor_on_close #13786 bpo-37137: Close pid waiters on watcher.close() #13792

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-04.12:43:35.166>
created_at = <Date 2019-06-03.01:18:08.370>
labels = ['3.8', 'tests']
title = 'test_asyncio: test_cancel_gather_2() dangling thread'
updated_at = <Date 2019-06-04.12:43:35.165>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-06-04.12:43:35.165>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-04.12:43:35.166>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2019-06-03.01:18:08.370>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 37137
keywords = ['patch']
message_count = 17.0
messages = ['344353', '344358', '344359', '344426', '344427', '344433', '344471', '344479', '344481', '344488', '344489', '344490', '344491', '344492', '344497', '344516', '344557']
nosy_count = 3.0
nosy_names = ['vstinner', 'asvetlov', 'pablogsal']
pr_nums = ['13779', '13786', '13792']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue37137'
versions = ['Python 3.8']

[vstinner]

s390x RHEL 3.x buildbot:
https://buildbot.python.org/all/#/builders/21/builds/3150

test_cancel_gather_1 (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
Ensure that a gathering future refuses to be cancelled once all ... ok
test_cancel_gather_2 (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ...
Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 4396153272064)>
ok

test_cancel_inner_future (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok
test_cancel_task_catching (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok

--

AMD64 FreeBSD 10-STABLE Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/167/builds/1181

test_cancel_gather_2 (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)

[vstinner]

Another similar error on "ARMv7 Debian buster 3.x" buildbot:
https://buildbot.python.org/all/#/builders/176/builds/664

test_proc_exited (test.test_asyncio.test_subprocess.SubprocessTransportTests) ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)

[vstinner]

AMD64 Ubuntu Shared 3.x:
https://buildbot.python.org/all/#/builders/141/builds/1912

test_proc_exited (test.test_asyncio.test_subprocess.SubprocessTransportTests) ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)
test_close (test.test_asyncio.test_unix_events.SafeChildWatcherTests) ... /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/asyncio/base_events.py:646: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
_warn(f"unclosed event loop {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

[vstinner]

https://buildbot.python.org/all/#/builders/167/builds/1188

Variant:

test_cancel_both_task_and_inner_future (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok
test_cancel_current_task (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok
test_cancel_gather_1 (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
Ensure that a gathering future refuses to be cancelled once all ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 34389124096)>
ok
test_cancel_gather_2 (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok
test_cancel_inner_future (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok
test_cancel_task_catching (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok

[vstinner]

See also bpo-34037: "asyncio: BaseEventLoop.close() shutdowns the executor without waiting causing leak of dangling threads".

[vstinner]

New changeset 49a7e34 by Victor Stinner in branch 'master':
bpo-37137: Fix test_asyncio: use TestCase.set_event_loop() (GH-13779)
49a7e34

[vstinner]

My change didn't fix the issue:

https://buildbot.python.org/all/#/builders/176/builds/680

test_proc_exited (test.test_asyncio.test_subprocess.SubprocessTransportTests) ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)

[vstinner]

I push a fix in asyncio, instead of trying to fix individual asyncio tests:

New changeset 0f0a30f by Victor Stinner in branch 'master':
bpo-34037, asyncio: add BaseEventLoop.wait_executor_on_close (GH-13786)
0f0a30f

See https://bugs.python.org/issue34037#msg344476

[vstinner]

My second attempt isn't fully successful :-( test_asyncio still fails with ENV_CHANGED :-(

https://buildbot.python.org/all/#/builders/185/builds/488

test_current_task_no_running_loop_implicit (test.test_asyncio.test_tasks.CCurrentLoopTests) ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)

[asvetlov]

I suspect that I've added this dangling thread by https://github.com/python/cpython/blob/master/Lib/asyncio/unix_events.py#L1248-L1251 :(

[asvetlov]

Working on PR

[vstinner]

I suspect that I've added this dangling thread by https://github.com/python/cpython/blob/master/Lib/asyncio/unix_events.py#L1248-L1251 :(

Can you please try to revert commit 13ed079?

[asvetlov]

We have two options:

1. Apply bpo-37137: Close pid waiters on watcher.close() #13792 to wait for dangling threads
2. Revert added watchers entirely

What do you prefer?