test_asyncio leaks dangling threads

[vstinner]

BPO	31250
Nosy	@vstinner, @1st1
PRs	bpo-31250, test_asyncio: fix dangling threads #3252 bpo-31250, test_asyncio: fix EventLoopTestsMixin.tearDown() #3264 [3.6] bpo-31250: test_asyncio: fix dangling threads #3517

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-09-12.21:18:39.212>
created_at = <Date 2017-08-21.16:33:56.001>
labels = ['3.7', 'expert-asyncio', 'tests', 'performance']
title = 'test_asyncio leaks dangling threads'
updated_at = <Date 2017-09-13.09:56:51.402>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-09-13.09:56:51.402>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-09-12.21:18:39.212>
closer = 'vstinner'
components = ['Tests', 'asyncio']
creation = <Date 2017-08-21.16:33:56.001>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 31250
keywords = ['patch']
message_count = 7.0
messages = ['300637', '301048', '301097', '301099', '301131', '301155', '301995']
nosy_count = 2.0
nosy_names = ['vstinner', 'yselivanov']
pr_nums = ['3252', '3264', '3517']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue31250'
versions = ['Python 3.6', 'Python 3.7']

[vstinner]

Using PR 3138 of bpo-31234, I noticed that test_asyncio leaks dangling threads.

Tests which randomly leak threads:
---
test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests.test_create_connection_no_inet_pton
test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests.test_create_connection_service_name
test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests.test_create_connection_bluetooth
test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests.test_create_datagram_endpoint_no_addrinfo
test.test_asyncio.test_futures.DuckTests.test_wrap_future
test.test_asyncio.test_futures.CFutureTests.test_wrap_future
test.test_asyncio.test_futures.CFutureTests.test_wrap_future_use_global_loop
test.test_asyncio.test_futures.PyFutureTests.test_wrap_future
test.test_asyncio.test_futures.PyFutureTests.test_wrap_future_use_global_loop
test.test_asyncio.test_selector_events.BaseSelectorEventLoopTests.test_sock_connect_resolve_using_socket_params
test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests.test_run_coroutine_threadsafe
test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests.test_run_coroutine_threadsafe_with_timeout
test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests.test_run_coroutine_threadsafe_task_cancelled
test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests.test_run_coroutine_threadsafe_task_factory_exception
---

Write this list into "tests" and run "python3 -m test --matchfile=tests".

haypo@selma$ ./python -u -m test -v test_asyncio --matchfile=tests
== CPython 3.7.0a0 (heads/concurrent_futures:f277fa3, Aug 21 2017, 18:11:41) [GCC 6.4.1 20170727 (Red Hat 6.4.1-1)]
(...)
test_create_connection_bluetooth (test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests) ... ok
test_create_connection_no_inet_pton (test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests) ... Warning -- threading_cleanup() detected 4 leaked threads (count: 4, dangling: 5)
ok
test_create_connection_service_name (test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests) ... Warning -- threading_cleanup() detected 4 leaked threads (count: 4, dangling: 5)
ok
test_create_datagram_endpoint_no_addrinfo (test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_wrap_future (test.test_asyncio.test_futures.CFutureTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_wrap_future_use_global_loop (test.test_asyncio.test_futures.CFutureTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_wrap_future (test.test_asyncio.test_futures.DuckTests) ... ok
test_wrap_future (test.test_asyncio.test_futures.PyFutureTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_wrap_future_use_global_loop (test.test_asyncio.test_futures.PyFutureTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_sock_connect_resolve_using_socket_params (test.test_asyncio.test_selector_events.BaseSelectorEventLoopTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_run_coroutine_threadsafe (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
Test coroutine submission from a thread to an event loop. ... ok
test_run_coroutine_threadsafe_task_cancelled (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
Test coroutine submission from a tread to an event loop ... ok
test_run_coroutine_threadsafe_task_factory_exception (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
Test coroutine submission from a tread to an event loop ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_run_coroutine_threadsafe_with_timeout (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
Test coroutine submission from a thread to an event loop ... ok
(...)

[vstinner]

I proposed PR 3252 to only fix unit tests, but not asyncio.

An alternative is to modify BaseEventLoop.close() to replace executor.shutdown(wait=False) with executor.shutdown(wait=True), but it would change the behaviour. I'm not sure if we want to wait until all threads complete on close()?

[vstinner]

New changeset 16432be by Victor Stinner in branch 'master':
bpo-31250, test_asyncio: fix dangling threads (bpo-3252)
16432be

[vstinner]

I picked the safe option: only modify unit tests.

[vstinner]

Hum, EventLoopTestsMixin.tearDown() still has a bug. I reopen the issue.

[vstinner]

New changeset e8a533f by Victor Stinner in branch 'master':
bpo-31250, test_asyncio: fix EventLoopTestsMixin.tearDown() (bpo-3264)
e8a533f

[vstinner]

New changeset 5013a5e by Victor Stinner in branch '3.6':
[3.6] bpo-31250: test_asyncio: fix dangling threads (bpo-3517)
5013a5e