In multiprocessing.pool, ExceptionWithTraceback should derive from Exception

[myint]

BPO	20980
Nosy	@bitdancer
Files	exception_with_traceback.diff thread_pool_exception_test.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-03-23.18:42:37.862>
created_at = <Date 2014-03-19.14:53:54.411>
labels = ['type-bug', 'library']
title = 'In multiprocessing.pool, ExceptionWithTraceback should derive from Exception'
updated_at = <Date 2014-03-23.18:42:37.861>
user = 'https://bugs.python.org/myint'

bugs.python.org fields:

activity = <Date 2014-03-23.18:42:37.861>
actor = 'sbt'
assignee = 'sbt'
closed = True
closed_date = <Date 2014-03-23.18:42:37.862>
closer = 'sbt'
components = ['Library (Lib)']
creation = <Date 2014-03-19.14:53:54.411>
creator = 'myint'
dependencies = []
files = ['34518', '34520']
hgrepos = []
issue_num = 20980
keywords = ['patch']
message_count = 6.0
messages = ['214101', '214102', '214106', '214247', '214273', '214571']
nosy_count = 4.0
nosy_names = ['r.david.murray', 'python-dev', 'sbt', 'myint']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20980'
versions = ['Python 3.4', 'Python 3.5']

[myint]

In multiprocessing.pool, ExceptionWithTraceback is not derived from Exception. Thus when it is raised, we get the exception, "TypeError: exceptions must derive from BaseException". Attached is a patch that fixes this.

See below example traceback that prompted me to notice this:

Traceback (most recent call last):
  File "/Users/myint/Library/Python/3.4/bin/yolk", line 9, in <module>
    load_entry_point('yolk3k==0.7.3', 'console_scripts', 'yolk')()
  File "/Users/myint/Library/Python/3.4/lib/python/site-packages/yolk/cli.py", line 1123, in main
    my_yolk.run()
  File "/Users/myint/Library/Python/3.4/lib/python/site-packages/yolk/cli.py", line 208, in run
    return getattr(self, action)()
  File "/Users/myint/Library/Python/3.4/lib/python/site-packages/yolk/cli.py", line 240, in show_updates
    for (project_name, version, newest) in _updates(pkg_list, self.pypi):
  File "/Users/myint/Library/Python/3.4/lib/python/site-packages/yolk/cli.py", line 1104, in _updates
    names):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 255, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 594, in get
    raise self._value
TypeError: exceptions must derive from BaseException

[bitdancer]

We must have a missing test case, then. One should be added.

[myint]

I didn't quite understand the multiprocessing test cases, but attached is a standalone test case that reproduces the problem. The problem only shows up when using ThreadPool.

$ python thread_pool_exception_test.py
Traceback (most recent call last):
  File "thread_pool_exception_test.py", line 7, in <module>
    pool.map(exception, [1])
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 255, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 594, in get
    raise self._value
TypeError: exceptions must derive from BaseException

[myint]

This bug was introduced in bpo-13831.

[sbt]

We should only wrap the exception with ExceptionWithTraceback in the process case where it will be pickled and then unpickled.

[python-dev]

New changeset df6a6951b2c9 by Richard Oudkerk in branch '3.4':
Issue bpo-20980: Stop wrapping exception when using ThreadPool.
http://hg.python.org/cpython/rev/df6a6951b2c9