multiprocessing.Pool hangs when issuing KeyboardInterrupt

[vlasovskikh]

BPO	8296
Nosy	@vsajip, @pitrou, @vlasovskikh, @untitaker
Files	test_map_keyboard_interrput.py: Unit test fix-sigint.diff: First patch fixing the problem

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-08-27.15:24:23.054>
created_at = <Date 2010-04-03.03:37:14.183>
labels = ['type-bug', 'library']
title = 'multiprocessing.Pool hangs when issuing KeyboardInterrupt'
updated_at = <Date 2014-08-24.01:49:59.849>
user = 'https://github.com/vlasovskikh'

bugs.python.org fields:

activity = <Date 2014-08-24.01:49:59.849>
actor = 'brian.curtin'
assignee = 'none'
closed = True
closed_date = <Date 2011-08-27.15:24:23.054>
closer = 'vinay.sajip'
components = ['Library (Lib)']
creation = <Date 2010-04-03.03:37:14.183>
creator = 'vlasovskikh'
dependencies = []
files = ['16784', '16785']
hgrepos = []
issue_num = 8296
keywords = ['patch']
message_count = 15.0
messages = ['102219', '102220', '102221', '102222', '102223', '102479', '102481', '114817', '114821', '114888', '114900', '114976', '143077', '143080', '221549']
nosy_count = 12.0
nosy_names = ['vinay.sajip', 'pitrou', 'jnoller', 'asksol', 'vlasovskikh', 'gdb', 'Albert.Strasheim', 'sbt', 'gkcn', 'mdengler', 'myint', 'untitaker']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8296'
versions = ['Python 2.6', 'Python 2.7']

[vlasovskikh]

multiprocessing.Pool methods map, imap, etc. are said to be able to normally handle exceptions. But it seems that it is true only for synchronous exceptions inside their first func arguments.

When (typically during a long-running parallel map) a user hits ^C, an asynchronous KeyboardInterrupt isn't handled properly and leads to the interpreter hangup. More precisely, children processes become <defunct> (on Linux), so the only way to terminate the whole program is to issue the KILL signal.

As stopping a program with ^C while running potentially long parallel computations is probably quite a common scenario, the interpreter should not hang up in such a case.

I'm using Python 2.6.5 (r265:79063, Mar 23 2010, 04:44:21) [GCC 4.4.3] on linux2. I've also tried to use the current multiprocessing.pool module from the current (2.7) trunk with my 2.6.5 installation, but the bug persists.

[briancurtin]

Do you have a test case which can reproduce the issue?

[vlasovskikh]

Yes, here is my test case.

[briancurtin]

You might want to take a look here: http://jessenoller.com/2009/01/08/multiprocessingpool-and-keyboardinterrupt/

[vlasovskikh]

Yes, I've come up with the same solution by myself, but it cannot cover all the cases of the bug. It works only for cases when ^C is hit during a call to the users' function: http://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool/2561809#2561809

If the user is "lucky", he may hit ^C during getting or putting data into the queues in multiprocessing.pool.worker. To reproduce such a case, you may insert sleep(10) before task = get() or put((job, i, result)), for example. I've encountered such cases just by running test examples several times.

[vlasovskikh]

Despite of several workarounds available on the Web, the problem persists. Almost any exception that is rised in worker function while putting or getting tasks from queues result in Pool hang up. Currently, worker is only aware of Exception descendants rised inside of the map function parameter.

I've written a unit test that checks if KeyboardInterrupts are handled normally. The source code may be included in [Lib/test/test_multiprocessing.py](https://github.com/python/cpython/blob/main/Lib/test/test_multiprocessing.py).

[vlasovskikh]

Here is a patch that fixes this problem. Basically, it catches all the BaseExceptions that could happen during: a) getting a task from the inqueue, b) calling a user function, c) putting a task into the outqueue. The exception handler puts the exception to the outqueue.

It can be cleanly applied on top of revision 78790.

[AlbertStrasheim]

Any chance of getting this patch applied? Thanks.

[asksol]

This is related to our discussions at bpo-9205 as well (http://bugs.python.org/issue9205), as the final patch there will also fix this issue.

[asksol]

On closer look your patch is also ignoring SystemExit. I think it's beneficial to honor SystemExit, so a user could use this as a means to replace the current process with a new one.

If we keep that behavior, the real problem here is that the
result handler hangs if the process that reserved a job is gone, which is going to be handled
by bpo-9205. Should we mark it as a duplicate?

[jnoller]

If we keep that behavior, the real problem here is that the
result handler hangs if the process that reserved a job is gone, which is going to be handled
by bpo-9205. Should we mark it as a duplicate?

I would tend to agree with your assessment; we're better served just
gracefully handling everything per 9205