multiprocessing.Pool should join "dead" processes

[vstinner]

BPO	31019
Nosy	@vstinner
PRs	bpo-31019: multiprocessing.Pool.terminate() now joins "dead" processes #2849 bpo-31019: Fix multiprocessing.Process.is_alive() #2875 [3.6] bpo-26762, bpo-31019: Backport multiprocessing fixes from master to 3.6 #2879 [2.7] bpo-31019: Fix multiprocessing.Process.is_alive() (#2875) #2882

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-07-26.15:55:41.607>
created_at = <Date 2017-07-24.17:36:38.219>
labels = ['3.7', 'library', 'performance']
title = 'multiprocessing.Pool should join "dead" processes'
updated_at = <Date 2017-07-26.15:55:41.605>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-07-26.15:55:41.605>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-07-26.15:55:41.607>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2017-07-24.17:36:38.219>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 31019
keywords = []
message_count = 6.0
messages = ['299002', '299170', '299177', '299189', '299245', '299246']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['2849', '2875', '2879', '2882']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue31019'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

[vstinner]

With debug patches for bpo-26762, I noticed that some unit tests of test_multiprocessing_spawn leaks "dangling" processes:
---
haypo@selma$ ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match test.test_multiprocessing_spawn.WithProcessesTestPool.test_context
== CPython 3.7.0a0 (heads/master:b364d9f, Jul 24 2017, 11:06:33) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
== Linux-4.11.9-200.fc25.x86_64-x86_64-with-fedora-25-Twenty_Five little-endian
== hash algorithm: siphash24 64bit
== cwd: /home/haypo/prog/python/master/build/test_python_20982
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
Run tests sequentially
0:00:00 load avg: 0.16 [1/1] test_multiprocessing_spawn
test_context (test.test_multiprocessing_spawn.WithProcessesTestPool) ... ok
Warning -- Dangling processes: {<SpawnProcess(SpawnPoolWorker-6, stopped[SIGTERM] daemon)>}
Dangling processes: {<SpawnProcess(SpawnPoolWorker-6, stopped[SIGTERM] daemon)>}

----------------------------------------------------------------------
Ran 1 test in 1.342s

OK
test_multiprocessing_spawn failed (env changed)

1 test altered the execution environment:
test_multiprocessing_spawn

Total duration: 1 sec
Tests result: ENV CHANGED
---

multiprocessing.Pool.terminate() doesn't call the join() method of a Process object if its is_alive() method returns false. But in practice, avoid the join() creates "dangling" processes.

Attached pull request fixes the warning: Pool.terminate() now calls join() on all processes including "dead" processes.

[vstinner]

Hum, I don't think that #2849 fixes the root issue.

The root issue is that calling Process.is_alive() doesn't update the _children list if the process completes. I wrote a fix for that:
#2875

[vstinner]

New changeset 2db6482 by Victor Stinner in branch 'master':
bpo-31019: Fix multiprocessing.Process.is_alive() (bpo-2875)
2db6482

[vstinner]

New changeset d0adfb2 by Victor Stinner in branch '3.6':
[3.6] bpo-26762, bpo-31019: Backport multiprocessing fixes from master to 3.6 (bpo-2879)
d0adfb2

[vstinner]

New changeset b65cb8a by Victor Stinner in branch '2.7':
bpo-31019: Fix multiprocessing.Process.is_alive() (bpo-2875) (bpo-2882)
b65cb8a

[vstinner]

I fixed multiprocessing.Process.is_alive() in 2.7, 3.6 and master. I close the issue.