Skip to content

multiprocessing hangs when Pool used within Process #49581

Closed
Closed
multiprocessing hangs when Pool used within Process#49581
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-crashA hard crash of the interpreter, possibly with a core dump

Description

@zzzeek
zzzeek
mannequin
opened on Feb 20, 2009
BPO 5331

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-06-30.17:14:56.090>
created_at = <Date 2009-02-20.16:06:50.657>
labels = ['library', 'type-crash']
title = 'multiprocessing hangs when Pool used within Process'
updated_at = <Date 2009-06-30.17:14:56.088>
user = 'https://bugs.python.org/zzzeek'

bugs.python.org fields:

activity = <Date 2009-06-30.17:14:56.088>
actor = 'jnoller'
assignee = 'jnoller'
closed = True
closed_date = <Date 2009-06-30.17:14:56.090>
closer = 'jnoller'
components = ['Library (Lib)']
creation = <Date 2009-02-20.16:06:50.657>
creator = 'zzzeek'
dependencies = []
files = []
hgrepos = []
issue_num = 5331
keywords = []
message_count = 6.0
messages = ['82534', '82540', '89192', '89194', '89901', '89945']
nosy_count = 4.0
nosy_names = ['OG7', 'LambertDW', 'jnoller', 'zzzeek']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue5331'
versions = ['Python 2.6', 'Python 3.0']

Activity

  1. zzzeek commented on Feb 20, 2009

    @zzzeek
    zzzeekmannequin
    on Feb 20, 2009
    MannequinAuthor

    this occurs for me running on Mac OSX Leopard. The equivalent code
    using "processing" in python 2.5 works fine, which is how I found this
    bug - my code hung when upgraded to 2.6. Basically initiating a
    multiprocessing.Pool inside of multiprocessing.Process hangs the
    application. Below is code which illustrates the issue on both py2.6
    and py3.0:

        from multiprocessing import Pool, Process
    import sys

    def f(x):
        return x*x

    def go():
        pool = Pool(processes=4)              
        x = pool.map(f, [1, 2, 3, 4, 5, 6, 7])
        sys.stdout.write("result: %s\n" % x)

    def go2():
        x = map(f, [1,2,3,4,5,6,7])
        sys.stdout.write("result: %s\n" % x)

    # pool alone, fine
    go()

    # process alone, fine
    p = Process(target=go2)
    p.start()
    p.join()

    # use both, hangs
    p = Process(target=go)
    p.start()
    p.join()
  2. Imagezzzeek
    added
    stdlibStandard Library Python modules in the Lib/ directory
    type-crashA hard crash of the interpreter, possibly with a core dump
    on Feb 20, 2009

  3. lambertdw commented on Feb 20, 2009

    @lambertdw
    lambertdwmannequin
    on Feb 20, 2009
    Mannequin

    Fails also on this system.

    $ p3
Python 3.0.1 (r301:69556, Feb 13 2009, 23:52:55) 
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
(...)
>>> # use both hangs
(...)
>>> p.start()
>>> p.join()
(hmm la  do diii   laaa...)^C
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/process.py",
line 119, in join
    res = self._popen.wait(timeout)
  File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",
line 118, in wait
    return self.poll(0)
  File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",
line 107, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
>>>
  4. Imagebenjaminp
    assigned
    jnoller
    on Feb 21, 2009

  5. OG7 commented on Jun 10, 2009

    @OG7
    OG7mannequin
    on Jun 10, 2009
    Mannequin

    Apparently the pool workers die all at once, just after the pool creates
    them and before the pool is used.
    I added a few lines to multiprocessing/pool.py to get the stack and the
    exception backtrace.

            except (EOFError, IOError):
            import traceback
            debug(traceback.format_exc())
            debug(''.join(traceback.format_stack()))
            debug('worker got EOFError or IOError -- exiting')
            break
    INFO::Rule dispatcher::multiprocessing::child process calling self.run()
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3082559488
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3082104832
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3081826304
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3081822208
INFO::PoolWorker-3:1::multiprocessing::child process calling self.run()
DEBUG::PoolWorker-3:1::multiprocessing::Traceback (most recent call last):
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 57, in worker
    task = get()
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/queues.py",
line 339, in get
    return recv()
IOError: [Errno 9] Bad file descriptor

    DEBUG::PoolWorker-3:1::multiprocessing:: File
    "/home/who/var/co/git-svn/what-base/correlator/bin/correlator", line 30,
    in <module>
    what.corr.actors.main.main()
    File
    "/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/main.py",
    line 47, in main
    rrp.start()
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
    line 109, in start
    self._popen = Popen(self)
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",
    line 99, in __init__
    code = process_obj._bootstrap()
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
    line 236, in _bootstrap
    self.run()
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
    line 93, in run
    self._target(*self._args, **self._kwargs)
    File
    "/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/rule_dispatcher.py",
    line 26, in main
    initargs=(manager.agg_msgs_queue, ))
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/init.py",
    line 232, in Pool
    return Pool(processes, initializer, initargs)
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
    line 107, in __init__
    w.start()
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
    line 109, in start
    self._popen = Popen(self)
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",
    line 99, in __init__
    code = process_obj._bootstrap()
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
    line 236, in _bootstrap
    self.run()
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
    line 93, in run
    self._target(*self._args, **self._kwargs)
    File
    "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
    line 61, in worker
    debug(''.join(traceback.format_stack()))

    DEBUG::PoolWorker-3:1::multiprocessing::worker got EOFError or IOError
    -- exiting
    INFO::PoolWorker-3:1::multiprocessing::process shutting down
    DEBUG::PoolWorker-3:1::multiprocessing::running all "atexit" finalizers
    with priority >= 0

  6. OG7 commented on Jun 10, 2009

    @OG7
    OG7mannequin
    on Jun 10, 2009
    Mannequin

    It seems the root cause is at http://bugs.python.org/issue5155 .
    A workaround is to use a duplex Pipe in SimpleQueue.

  7. jnoller commented on Jun 30, 2009

    @jnoller
    jnollermannequin
    on Jun 30, 2009
    Mannequin

    Patch attached to bpo-5313, please review

  8. jnoller commented on Jun 30, 2009

    @jnoller
    jnollermannequin
    on Jun 30, 2009
    Mannequin

    Committed in r73708 on trunk

  9. Imagejnoller
    closed this as completedon Jun 30, 2009
  10. Imageezio-melotti
    transferred this issue fromon Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions