`concurrent.futures.ProcessPoolExecutor` swallows tracebacks

[cool-RR]

BPO	21817
Nosy	@brianquinlan, @pitrou, @cool-RR, @PCManticore, @MojoVampire
Files	issue21817.patch issue21817_1.patch issue21817_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-01-17.19:03:59.900>
created_at = <Date 2014-06-20.22:13:22.610>
labels = ['type-feature', 'library']
title = '`concurrent.futures.ProcessPoolExecutor` swallows tracebacks'
updated_at = <Date 2015-01-17.19:03:59.899>
user = 'https://github.com/cool-RR'

bugs.python.org fields:

activity = <Date 2015-01-17.19:03:59.899>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2015-01-17.19:03:59.900>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2014-06-20.22:13:22.610>
creator = 'cool-RR'
dependencies = []
files = ['35719', '37738', '37744']
hgrepos = []
issue_num = 21817
keywords = ['patch']
message_count = 9.0
messages = ['221128', '221218', '225649', '225660', '234174', '234181', '234186', '234191', '234192']
nosy_count = 6.0
nosy_names = ['bquinlan', 'pitrou', 'cool-RR', 'Claudiu.Popa', 'python-dev', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21817'
versions = ['Python 3.5']

[cool-RR]

When you use concurrent.futures.ProcessPoolExecutor and an exception is raised in the created process, it doesn't show you the traceback. This makes debugging very annoying.

Example file:

    #!python
    
    import sys
    import concurrent.futures 
    
    def f():
        # Successful assert:
        assert True
        return g()
    
    def g():
        # Hard-to-find failing assert:
        assert False
    
        
    if __name__ == '__main__':
        with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
            assert isinstance(executor, concurrent.futures.Executor)
            future = executor.submit(f)
            future.result()
        print('Main process finished')
        
        
If you run this under Windows, you get this: 

    Traceback (most recent call last):
      File "./bug.py", line 20, in <module>
        future.result()
      File "c:\python34\lib\concurrent\futures\_base.py", line 402, in result
        return self.__get_result()
      File "c:\python34\lib\concurrent\futures\_base.py", line 354, in __get_result
        raise self._exception
    AssertionError

This is the traceback of the main process, while we want the traceback of the process that failed.

[PCManticore]

Hello. Here's a patch based on c4f92b597074, which adds something similar to multiprocessing.pool.
The output after the patch is:

concurrent.futures.process.RemoteTraceback:
"""
Traceback (most recent call last):
  File "D:\Projects\cpython\lib\concurrent\futures\process.py", line 153, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "D:\Projects\cpython\PCbuild\a.py", line 9, in f
    return g()
  File "D:\Projects\cpython\PCbuild\a.py", line 13, in g
    assert False
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "a.py", line 20, in <module>
    future.result()
  File "D:\Projects\cpython\lib\concurrent\futures\_base.py", line 402, in result
    return self.__get_result()
  File "D:\Projects\cpython\lib\concurrent\futures\_base.py", line 354, in __get_result
    raise self._exception
AssertionError

It's a little better than the current output, even though it's a little verbose.

[PCManticore]

Any type of feedback regarding this approach will be appreciated.

[cool-RR]

Hi Claudiu, sorry for the silence.

This output looks great. I'd love to see that go into Python.

[PCManticore]

Here's the updated version of this patch.

[pitrou]

Hi, here are some comments about the patch:

• RemoteTraceback, ExceptionWithTraceback, rebuild_exc should be private (i.e. with a leading underscore)
• in test_traceback, you can use the context manager form of assertRaises instead of the try..except..else construct

[PCManticore]

Thanks, Antoine! Here's the new version, with your comments addressed.

[python-dev]

New changeset a36b402b099b by Antoine Pitrou in branch 'default':
Issue bpo-21817: When an exception is raised in a task submitted to a ProcessPoolExecutor, the remote traceback is now displayed in the parent process.
https://hg.python.org/cpython/rev/a36b402b099b

[pitrou]

Thank you very much. Everything is now committed.