test_multiprocessing_spawn and test_multiprocessing_forkserver leak dangling processes

[vstinner]

BPO	31069
Nosy	@vstinner
PRs	bpo-31069, test_multiprocessing: Fix dangling process #3103 [3.6] bpo-31069, test_multiprocessing: Fix dangling process (#3103) #3104 [2.7] bpo-31069, test_multiprocessing: Fix dangling process (#3103) #3105

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-09-15.16:40:24.678>
created_at = <Date 2017-07-28.15:02:05.743>
labels = ['3.7', 'tests']
title = 'test_multiprocessing_spawn and test_multiprocessing_forkserver leak dangling processes'
updated_at = <Date 2017-09-15.16:40:24.676>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-09-15.16:40:24.676>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-09-15.16:40:24.678>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-07-28.15:02:05.743>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 31069
keywords = []
message_count = 16.0
messages = ['299392', '300118', '300122', '300137', '300334', '300337', '300343', '300345', '300348', '300349', '300350', '301339', '301969', '301977', '302061', '302273']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['3103', '3104', '3105']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue31069'
versions = ['Python 3.7']

[vstinner]

It's just a warning, but later I would like to make a buildbot fail on such warnings, so I would like to first fix all these warnings.

http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Refleaks%203.x/builds/59/steps/test/logs/stdio

0:46:17 [119/406] test_multiprocessing_spawn passed (2667 sec) -- running: test_asyncio (290 sec), test_userstring (54 sec), test_regrtest (471 sec)
beginning 6 repetitions
123456
...Warning -- Dangling processes: {<Process(Process-1235, stopped daemon)>}
...

[vstinner]

With the commit 957d0e9 (bpo-26762), _test_multiprocessing now fails with ENV_CHANGED if a warning is emitted. It allowed to catch a warning on x86-64 El Capitan 3.x:

http://buildbot.python.org/all/builders/x86-64%20El%20Capitan%203.x/builds/581/steps/test/logs/stdio

test_enable_logging (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_level (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_spawn.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
test_pickling (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
test_boundaries (test.test_multiprocessing_spawn.WithProcessesTestPoll) ... ok

[vstinner]

Another warning on x86 Tiger 3.x:

http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/1043/steps/test/logs/stdio

test_rapid_restart (test.test_multiprocessing_forkserver.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_forkserver.WithProcessesTestPicklingConnections) ... ok

[vstinner]

See also bpo-30846.

[vstinner]

I marked bpo-30846 as a duplicate of this issue. Copy of its message:

http://buildbot.python.org/all/builders/AMD64%20Debian%20PGO%203.6/builds/517/steps/test/logs/stdio

======================================================================
ERROR: test_rapid_restart (test.test_multiprocessing_fork.WithProcessesTestManagerRestart)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/test/_test_multiprocessing.py", line 2410, in test_rapid_restart
    manager.start()
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/managers.py", line 517, in start
    self._address = reader.recv()
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
(...)
Re-running test 'test_multiprocessing_fork' in verbose mode
(...)
test_rapid_restart (test.test_multiprocessing_fork.WithManagerTestManagerRestart) ... ok

[vstinner]

AppVeyor:

https://ci.appveyor.com/project/python/cpython/build/3.7.0a0.5341

test_lock_context (test.test_multiprocessing_spawn.WithProcessesTestLock) ... ok
test_rlock (test.test_multiprocessing_spawn.WithProcessesTestLock) ... ok
test_enable_logging (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_level (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_spawn.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-174, stopped daemon)>}
test_access (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok

[vstinner]

http://buildbot.python.org/all/builders/x86-64%20El%20Capitan%203.x/builds/590/steps/test/logs/stdio

test_enable_logging (test.test_multiprocessing_forkserver.WithProcessesTestLogging) ... ok
test_level (test.test_multiprocessing_forkserver.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_forkserver.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_forkserver.WithProcessesTestPicklingConnections) ... ok
test_pickling (test.test_multiprocessing_forkserver.WithProcessesTestPicklingConnections) ... ok

(...)

test_level (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_spawn.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
test_pickling (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok

[vstinner]

New changeset 17657bb by Victor Stinner in branch 'master':
bpo-31069, test_multiprocessing: Fix dangling process (bpo-3103)
17657bb

[vstinner]

New changeset 78c66a6 by Victor Stinner in branch '3.6':
bpo-31069, test_multiprocessing: Fix dangling process (bpo-3103) (bpo-3104)
78c66a6

[vstinner]

New changeset 883520a by Victor Stinner in branch '2.7':
bpo-31069, test_multiprocessing: Fix dangling process (bpo-3103) (bpo-3105)
883520a