test_multiprocessing_spawn.WithProcessesTestPool.test_traceback() leaks 4 handles on Windows

[vstinner]

BPO	33966
Nosy	@pfmoore, @pitrou, @vstinner, @tjguk, @zware, @zooba, @applio
PRs	[WIP] bpo-33966, multiprocessing: Fix another handle leak #7965 bpo-33966, multiprocessing: Pool wait for worker start #7966

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-19.23:55:24.876>
created_at = <Date 2018-06-26.11:55:02.617>
labels = ['3.8', 'library', 'OS-windows']
title = 'test_multiprocessing_spawn.WithProcessesTestPool.test_traceback() leaks 4 handles on Windows'
updated_at = <Date 2018-09-19.23:55:24.875>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-09-19.23:55:24.875>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-19.23:55:24.876>
closer = 'vstinner'
components = ['Library (Lib)', 'Windows']
creation = <Date 2018-06-26.11:55:02.617>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 33966
keywords = ['patch']
message_count = 4.0
messages = ['320484', '320525', '320528', '325833']
nosy_count = 7.0
nosy_names = ['paul.moore', 'pitrou', 'vstinner', 'tim.golden', 'zach.ware', 'steve.dower', 'davin']
pr_nums = ['7965', '7966']
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33966'
versions = ['Python 3.8']

[vstinner]

Using my PR 7827, the following command shows a leak of 4 Windows handles per run:

python -m test -R 3:3 test_multiprocessing_spawn -v -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_traceback

See also bpo-33929: test_multiprocessing_spawn: WithProcessesTestProcess.test_many_processes() leaks 5 handles on Windows.

[vstinner]

• Pool._repopulate_pool() creates processes with args=(self._inqueue, ...)
• Pool._inqueue is a multiprocessing.queues.SimpleQueue
• Process.__init__() of multiprocessing.popen_spawn_win32 calls reduction.dump(process_obj, to_child) which indirectly contains the SimpleQueue
• A SimpleQueue object contains: self._reader, self._writer = connection.Pipe(duplex=False)
• dump() indirectly calls reduce_connection() (ex: for SimpleQueue._reader) of multiprocessing.connection
• reduce_connection() duplicates the pipe handle

It's unclear to me who is supposed to close the duplicated pipe handles? reduce_connection() creates a "ds = resource_sharer.DupSocket(s)" object, but this object doesn't seem to call CloseHandle()?

[vstinner]

• dump() indirectly calls reduce_connection() (ex: for SimpleQueue._reader) of multiprocessing.connection
• reduce_connection() duplicates the pipe handle

Sorry, it's reduce_pipe_connection(), not reduce_connection().

It's unclear to me who is supposed to close the duplicated pipe handles? reduce_connection() creates a "ds = resource_sharer.DupSocket(s)" object, but this object doesn't seem to call CloseHandle()?

reduce_pipe_connection() creates a DupHandle object which duplicates the handle, and it's detach() method closes the handle.

The race condition is that sometimes the pool terminates a worker (worker() function of multiprocessing.pool) before the worker has time to close the duplicated handle.

The handle is closed by:

• rebuild_pipe_connection()
• dh.detach(), extract:

                return _winapi.DuplicateHandle(
                    proc, self._handle, _winapi.GetCurrentProcess(),
                    self._access, False, _winapi.DUPLICATE_CLOSE_SOURCE)

[vstinner]

I give up on that one :-( It doesn't seem easy to fix this corner case without changing the behaviour.