Pickle protocol 2.0 not loading in python 3.5

[anilredshift]

BPO	26013
Nosy	@pitrou, @vstinner, @avassalotti, @serhiy-storchaka
Files	b.py: demonstration of bug unpickle_broken_import.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-01-19.10:02:59.279>
created_at = <Date 2016-01-05.07:39:44.938>
labels = ['type-bug', 'library']
title = 'Pickle protocol 2.0 not loading in python 3.5'
updated_at = <Date 2016-01-19.10:02:59.278>
user = 'https://bugs.python.org/anilredshift'

bugs.python.org fields:

activity = <Date 2016-01-19.10:02:59.278>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-01-19.10:02:59.279>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2016-01-05.07:39:44.938>
creator = 'anilredshift'
dependencies = []
files = ['41503', '41504']
hgrepos = []
issue_num = 26013
keywords = ['patch']
message_count = 9.0
messages = ['257524', '257526', '257541', '257557', '257558', '258540', '258544', '258545', '258546']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'alexandre.vassalotti', 'python-dev', 'serhiy.storchaka', 'anilredshift']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26013'
versions = ['Python 3.5', 'Python 3.6']

[anilredshift]

Pickles created with python 3.4.X will not load with python 3.5.X if they include a collections.OrderedDict

To reproduce this issue, simply create a pickle of an OrderedDict on python 3.4.3 with protocol=2 and try to open it on 3.5. I have included a simple script to demonstrate this issue.

I believe this is related to this bug: https://bugs.python.org/issue18473

As to the real-world implications: The python package Celery uses protocol=2 by default when serializing with pickle, and thus a celery web running 3.5 cannot receive the results of a worker running 3.4
For celery specifically, there is a workaround by setting the PICKLE_PROTOCOL environment variable, but this is a core python issue.

P.S. This is the first bug I've filed with python so please let me know if there's something else I should be including.

Thanks!

[serhiy-storchaka]

Thank you for your report Anil.

Python 3.4.3 creates incorrect pickle with protocol 2 (it can't be load in Python 2). bpo-18473 fixed pickling with protocol 2 in Python 3, but broke loading broken pickles created in unpatched versions of Python 3.

Here is a patch that allows to load such broken pickles.

But it is too late to fix Python 3.4, it can get only security fixes. You have Celery specific workaround. More general workaround is to update _compact_pickle.IMPORT_MAPPING, e.g.:

import _compact_pickle
_compact_pickle.IMPORT_MAPPING.update({
    'UserDict': 'collections',
    'UserList': 'collections',
    'UserString': 'collections',
    'whichdb': 'dbm',
    'StringIO':  'io',
    'cStringIO': 'io',
})

Note that you have to set mapping not just for one module name like 'UserList', but for all 'UserDict', 'UserList', and 'UserString', because it is not predicable to what module name 'collections' is mapped in Python 3.4.3.

[anilredshift]

Hi Serhiy,

I have done some more investigation this morning and I have come across two distinct issues.

The first is that pickles (specifically the pickle in my earlier message) created in python 3.0->3.4.3 do not load on python 2.7.10. The exception is 3.5.1, which produces a pickle that DOES load on 2.7.10
This is unfortunate but not a regression

The second issue is that pickles created by python 3.0->3.4.3 do NOT load in python 3.5

This is a regression, in the sense that the pickle created by 3.0->3.4.3 is compatible with every other version of python (I haven't tested every single combination here, but several)

From the language perspective, it may be that 3.4.X has incorrect code, but from a compatibility perspective, 3.5 is breaking the promise that pickles are compatible across versions of python.

I write this as justification that python 3.5 should fix this regression in compatibility with 3.0 -> 3.4.3. At the very least 3.5 should have a shim behavior to fallback and allow it to import these pickles correctly.

Thanks,
Anil

[serhiy-storchaka]

I agree, and the provided is purposed to fix this issue. In the previous message I just proposed a workaround for those who can't wait 3.5.2.

[anilredshift]

Ah, sorry I misunderstood. Thanks for the quick turnaround!

[python-dev]

New changeset 270add17f203 by Serhiy Storchaka in branch '3.5':
Issue bpo-26013: Added compatibility with broken protocol 2 pickles created
https://hg.python.org/cpython/rev/270add17f203

New changeset 35ff0976b211 by Serhiy Storchaka in branch 'default':
Issue bpo-26013: Added compatibility with broken protocol 2 pickles created
https://hg.python.org/cpython/rev/35ff0976b211

[vstinner]

Buildbots are unhappy.

http://buildbot.python.org/all/builders/PPC64%20Fedora%203.5/builds/344/steps/test/logs/stdio

======================================================================
FAIL: test_reverse_import_mapping (test.test_pickle.CompatPickleTests) [('cStringIO', 'io')]
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/test/test_pickle.py", line 353, in test_reverse_import_mapping
    (module3, module2))
AssertionError: No reverse mapping from 'io' to 'cStringIO'

[python-dev]

New changeset 7215d13baa2d by Serhiy Storchaka in branch '3.5':
Added exceptins for testing non-reversible import mapping for Issue bpo-26013.
https://hg.python.org/cpython/rev/7215d13baa2d

New changeset 16cfc1652844 by Serhiy Storchaka in branch 'default':
Added exceptins for testing non-reversible import mapping for Issue bpo-26013.
https://hg.python.org/cpython/rev/16cfc1652844

[serhiy-storchaka]

Thank you Victor.