Issue5331
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2009-02-20 16:06 by zzzeek, last changed 2022-04-11 14:56 by admin. This issue is now closed.
| Messages (6) | |||
|---|---|---|---|
| msg82534 - (view) | Author: mike bayer (zzzeek) * | Date: 2009-02-20 16:06 | |
this occurs for me running on Mac OSX Leopard. The equivalent code
using "processing" in python 2.5 works fine, which is how I found this
bug - my code hung when upgraded to 2.6. Basically initiating a
multiprocessing.Pool inside of multiprocessing.Process hangs the
application. Below is code which illustrates the issue on both py2.6
and py3.0:
from multiprocessing import Pool, Process
import sys
def f(x):
return x*x
def go():
pool = Pool(processes=4)
x = pool.map(f, [1, 2, 3, 4, 5, 6, 7])
sys.stdout.write("result: %s\n" % x)
def go2():
x = map(f, [1,2,3,4,5,6,7])
sys.stdout.write("result: %s\n" % x)
# pool alone, fine
go()
# process alone, fine
p = Process(target=go2)
p.start()
p.join()
# use both, hangs
p = Process(target=go)
p.start()
p.join()
|
|||
| msg82540 - (view) | Author: David W. Lambert (LambertDW) | Date: 2009-02-20 18:57 | |
Fails also on this system.
$ p3
Python 3.0.1 (r301:69556, Feb 13 2009, 23:52:55)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
(...)
>>> # use both hangs
(...)
>>> p.start()
>>> p.join()
(hmm la do diii laaa...)^C
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/process.py",
line 119, in join
res = self._popen.wait(timeout)
File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",
line 118, in wait
return self.poll(0)
File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",
line 107, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
>>>
|
|||
| msg89192 - (view) | Author: OG7 (OG7) | Date: 2009-06-10 08:58 | |
Apparently the pool workers die all at once, just after the pool creates
them and before the pool is used.
I added a few lines to multiprocessing/pool.py to get the stack and the
exception backtrace.
except (EOFError, IOError):
import traceback
debug(traceback.format_exc())
debug(''.join(traceback.format_stack()))
debug('worker got EOFError or IOError -- exiting')
break
INFO::Rule dispatcher::multiprocessing::child process calling self.run()
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3082559488
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3082104832
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3081826304
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3081822208
INFO::PoolWorker-3:1::multiprocessing::child process calling self.run()
DEBUG::PoolWorker-3:1::multiprocessing::Traceback (most recent call last):
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 57, in worker
task = get()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/queues.py",
line 339, in get
return recv()
IOError: [Errno 9] Bad file descriptor
DEBUG::PoolWorker-3:1::multiprocessing:: File
"/home/who/var/co/git-svn/what-base/correlator/bin/correlator", line 30,
in <module>
what.corr.actors.main.main()
File
"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/main.py",
line 47, in main
rrp.start()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 109, in start
self._popen = Popen(self)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",
line 99, in __init__
code = process_obj._bootstrap()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 93, in run
self._target(*self._args, **self._kwargs)
File
"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/rule_dispatcher.py",
line 26, in main
initargs=(manager.agg_msgs_queue, ))
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/__init__.py",
line 232, in Pool
return Pool(processes, initializer, initargs)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 107, in __init__
w.start()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 109, in start
self._popen = Popen(self)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",
line 99, in __init__
code = process_obj._bootstrap()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 93, in run
self._target(*self._args, **self._kwargs)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 61, in worker
debug(''.join(traceback.format_stack()))
DEBUG::PoolWorker-3:1::multiprocessing::worker got EOFError or IOError
-- exiting
INFO::PoolWorker-3:1::multiprocessing::process shutting down
DEBUG::PoolWorker-3:1::multiprocessing::running all "atexit" finalizers
with priority >= 0
|
|||
| msg89194 - (view) | Author: OG7 (OG7) | Date: 2009-06-10 10:42 | |
It seems the root cause is at http://bugs.python.org/issue5155 . A workaround is to use a duplex Pipe in SimpleQueue. |
|||
| msg89901 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2009-06-30 03:04 | |
Patch attached to issue 5313, please review |
|||
| msg89945 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2009-06-30 17:14 | |
Committed in r73708 on trunk |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:56:45 | admin | set | github: 49581 |
| 2009-06-30 17:14:56 | jnoller | set | status: open -> closed resolution: fixed messages: + msg89945 |
| 2009-06-30 03:04:14 | jnoller | set | messages: + msg89901 |
| 2009-06-10 10:42:31 | OG7 | set | messages: + msg89194 |
| 2009-06-10 08:58:09 | OG7 | set | nosy:
+ OG7 messages: + msg89192 |
| 2009-03-29 14:38:33 | jnoller | set | priority: normal |
| 2009-02-21 01:53:21 | benjamin.peterson | set | assignee: jnoller nosy: + jnoller |
| 2009-02-20 18:57:27 | LambertDW | set | nosy:
+ LambertDW messages: + msg82540 |
| 2009-02-20 16:06:50 | zzzeek | create | |
➜
