some objects pickled by Python 3.x are not unpicklable in Python 2.x because of incorrect REVERSE_IMPORT_MAPPING

[kmike]

BPO	18473
Nosy	@pitrou, @vstinner, @avassalotti, @serhiy-storchaka
Files	fix_issue18473.patch pickle_fix_import.patch pickle_fix_import_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/avassalotti'
closed_at = <Date 2015-03-31.16:48:43.737>
created_at = <Date 2013-07-16.11:22:47.023>
labels = ['type-bug', 'library']
title = 'some objects pickled by Python 3.x are not unpicklable in Python 2.x because of incorrect REVERSE_IMPORT_MAPPING'
updated_at = <Date 2015-03-31.17:23:32.444>
user = 'https://bugs.python.org/kmike'

bugs.python.org fields:

activity = <Date 2015-03-31.17:23:32.444>
actor = 'serhiy.storchaka'
assignee = 'alexandre.vassalotti'
closed = True
closed_date = <Date 2015-03-31.16:48:43.737>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2013-07-16.11:22:47.023>
creator = 'kmike'
dependencies = []
files = ['37145', '37210', '38669']
hgrepos = []
issue_num = 18473
keywords = ['patch', 'needs review']
message_count = 15.0
messages = ['193160', '193161', '230828', '231021', '231269', '232659', '239145', '239678', '239685', '239686', '239687', '239712', '239713', '239735', '239739']
nosy_count = 7.0
nosy_names = ['pitrou', 'vstinner', 'alexandre.vassalotti', 'python-dev', 'serhiy.storchaka', 'kmike', 'doug.royal']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue18473'
versions = ['Python 3.4', 'Python 3.5']

[kmike]

I think REVERSE_IMPORT_MAPPING is wrong (see http://hg.python.org/cpython/file/7272ef213b7c/Lib/_compat_pickle.py#l80 ). It relies on dictionary items order and maps 'collections' module to either UserString or UserList. This makes it impossible to unpickle collections module classes pickled by Python 3.x when using Python 2.x.

>>> import collections
>>> dct = collections.defaultdict()
>>> pickle.dumps(dct, protocol=1)
b'cUserString\ndefaultdict\nq\x00)Rq\x01.'
>>> pickle.dumps(dct, protocol=2, fix_imports=False)
b'ccollections\ndefaultdict\nq\x00)Rq\x01.'

I think pickling of instances of classes from other modules could also suffer. I don't think it is a good idea to map io to cStringIO or StringIO, or to map http.server to either BaseHTTPServer, SimpleHTTPServer or CGIHTTPServer.

Originally found this issue here: https://github.com/nltk/nltk/pull/441/files#L2R67

[kmike]

my copy-paste went wrong, last statement in the example should be

>>> pickle.dumps(dct, protocol=1, fix_imports=False)
b'ccollections\ndefaultdict\nq\x00)Rq\x01.'

[dougroyal]

This patch only addresses the proven errors with UserList, UserString, and collections.

[serhiy-storchaka]

I think that both UserDict.UserDict and UserDict.IterableUserDict should be mapped to collections.UserDict. And reverse mapping should map collections.UserDict to UserDict.IterableUserDict.

There are similar issues with other "multiple to single" mappings (e.g. to "io", "dbm", "http.server").

[serhiy-storchaka]

Here is a patch which fixes multiple reversible mappings.

Mappings for StringIO and cStringIO is removed at all because cStringIO.StringIO is not pickleable on 2.7 at all and pickled StringIO.StringIO is not compatible with 3.x (bpo-22890).

[serhiy-storchaka]

Ping.

[serhiy-storchaka]

Updated patch fixes more bugs.

Added support of unicode, izip_longest, abstract collections, ChainMap, multiprocessing exceptions, some socket and multiprocessing functions and types, xml.etree.ElementTree (C implementation). Added support of urllib and urllib2 (only public interface). Fixed reverse mapping of OSError subclasses, str, bz2, different dbm implementations, pickle. Added special sanity tests for _compat_pickle mappings (test that mappings are not ambiguous and mainly reversible, that 3.x names exist, etc).

[python-dev]

New changeset 8d86dfe53b97 by Serhiy Storchaka in branch '3.4':
Issue bpo-18473: Fixed 2to3 and 3to2 compatible pickle mappings.
https://hg.python.org/cpython/rev/8d86dfe53b97

New changeset 5980e81219ed by Serhiy Storchaka in branch 'default':
Issue bpo-18473: Fixed 2to3 and 3to2 compatible pickle mappings.
https://hg.python.org/cpython/rev/5980e81219ed

[vstinner]

Buildbots are unhappy. Example:

http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/11437/steps/test/logs/stdio

======================================================================
ERROR: test_name_mapping (test.test_pickle.CompatPickleTests) [(('tkinter.filedialog', 'FileDialog'), ('FileDialog', 'FileDialog'))]
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 241, in getmodule
    return sys.modules[module]
KeyError: 'tkinter.filedialog'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 286, in test_name_mapping
    attr = getattribute(module3, name3)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 247, in getattribute
    obj = getmodule(module)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 243, in getmodule
    __import__(module)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/tkinter/__init__.py", line 38, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: No module named '_tkinter'

[vstinner]

A different error:

http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/2880/steps/test/logs/stdio

======================================================================
FAIL: test_class_nested_enum_and_pickle_protocol_four (test.test_enum.TestEnum)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_enum.py", line 580, in test_class_nested_enum_and_pickle_protocol_four
    protocol=(0, 3))
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_enum.py", line 82, in test_pickle_exception
    dumps(obj, protocol=protocol)
AssertionError: PicklingError not raised

[vstinner]

A third kind of error:

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/5992/steps/test/logs/stdio

======================================================================
ERROR: test_import (test.test_pickle.CompatPickleTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pickle.py", line 241, in getmodule
    return sys.modules[module]
KeyError: 'multiprocessing.popen_fork'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pickle.py", line 266, in test_import
    getmodule(module)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pickle.py", line 243, in getmodule
    __import__(module)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\multiprocessing\popen_fork.py", line 13, in <module>
    class Popen(object):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\multiprocessing\popen_fork.py", line 25, in Popen
    def poll(self, flag=os.WNOHANG):
AttributeError: module 'os' has no attribute 'WNOHANG'

[python-dev]

New changeset 5c5eb374e296 by Serhiy Storchaka in branch '3.4':
Issue bpo-18473: Fixed pickle compatibility tests for optional modules.
https://hg.python.org/cpython/rev/5c5eb374e296

New changeset 29b2b2d8e36f by Serhiy Storchaka in branch 'default':
Issue bpo-18473: Fixed pickle compatibility tests for optional modules.
https://hg.python.org/cpython/rev/29b2b2d8e36f