Persistent id in pickle with protocol version 0

[serhiy-storchaka]

BPO	17711
Nosy	@avassalotti, @serhiy-storchaka
Files	fix_bad_persid.patch fix_bad_persid_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/avassalotti'
closed_at = <Date 2016-10-25.18:41:54.290>
created_at = <Date 2013-04-13.11:41:49.521>
labels = ['extension-modules', 'type-bug', 'library']
title = 'Persistent id in pickle with protocol version 0'
updated_at = <Date 2016-10-25.18:41:54.289>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2016-10-25.18:41:54.289>
actor = 'serhiy.storchaka'
assignee = 'alexandre.vassalotti'
closed = True
closed_date = <Date 2016-10-25.18:41:54.290>
closer = 'serhiy.storchaka'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2013-04-13.11:41:49.521>
creator = 'serhiy.storchaka'
dependencies = []
files = ['29842', '38128']
hgrepos = []
issue_num = 17711
keywords = ['patch']
message_count = 9.0
messages = ['186705', '186789', '186816', '186881', '186894', '235881', '268851', '269874', '270619']
nosy_count = 3.0
nosy_names = ['alexandre.vassalotti', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17711'
versions = ['Python 3.5', 'Python 3.6']

[serhiy-storchaka]

Python 2 allows pickling and unpickling non-ascii persistent ids. In Python 3 C implementation of pickle saves persistent ids with protocol version 0 as utf8-encoded strings and loads as bytes.

>>> import pickle, io
>>> class MyPickler(pickle.Pickler):
...     def persistent_id(self, obj):
...         if isinstance(obj, str):
...             return obj
...         return None
... 
>>> class MyUnpickler(pickle.Unpickler):
...     def persistent_load(self, pid):
...         return pid
... 
>>> f = io.BytesIO(); MyPickler(f).dump('\u20ac'); data = f.getvalue()
>>> MyUnpickler(io.BytesIO(data)).load()
'€'
>>> f = io.BytesIO(); MyPickler(f, 0).dump('\u20ac'); data = f.getvalue()
>>> MyUnpickler(io.BytesIO(data)).load()
b'\xe2\x82\xac'
>>> f = io.BytesIO(); MyPickler(f, 0).dump('a'); data = f.getvalue()
>>> MyUnpickler(io.BytesIO(data)).load()
b'a'

Python implementation in Python 3 doesn't works with non-ascii persistant ids at all.

[avassalotti]

In protocol 0, the persistent ID is restricted to alphanumeric strings because of the problems that arise when the persistent ID contains newline characters. _pickle likely should be changed to use the ASCII decoded. And perhaps, we should check for embedded newline characters too.

[serhiy-storchaka]

Even for alphanumeric strings Python 3 have a bug. It saves strings and load bytes objects.

[avassalotti]

Here's a patch that fix the bug.

[serhiy-storchaka]

I think a string with character codes < 256 will be better for test_protocol0_is_ascii_only(). It can be latin1 encoded (Python 2 allows any 8-bit strings).

PyUnicode_AsASCIIString() can be slower than _PyUnicode_AsStringAndSize() (actually PyUnicode_AsUTF8AndSize()) because the latter can use cached value. You can check if the persistent id only contains ASCII characters by checking PyUnicode_GET_LENGTH(pid_str) == size.

And what are you going to do with the fact that in Python 2 you can pickle non-ascii persistent ids, which will not be able to unpickle in Python 3?

[serhiy-storchaka]

The patch is updated to current sources. Also optimized writing ASCII strings and fixed tests.

[serhiy-storchaka]

Ping.

[serhiy-storchaka]

Ping again.

[python-dev]

New changeset f6a41552a312 by Serhiy Storchaka in branch '3.5':
Issue bpo-17711: Fixed unpickling by the persistent ID with protocol 0.
https://hg.python.org/cpython/rev/f6a41552a312

New changeset df8857c6f3eb by Serhiy Storchaka in branch 'default':
Issue bpo-17711: Fixed unpickling by the persistent ID with protocol 0.
https://hg.python.org/cpython/rev/df8857c6f3eb