Support pickling objects with __new__ with keyword arguments with protocol 2+

[serhiy-storchaka]

BPO	24164
Nosy	@pitrou, @vstinner, @avassalotti, @serhiy-storchaka
Files	pickle_new_ex_protocol_2.patch pickle_new_ex_protocol_2_doc.patch pickle_new_ex_protocol_2_doc_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-10-13.18:29:17.151>
created_at = <Date 2015-05-11.10:49:50.457>
labels = ['extension-modules', 'interpreter-core', 'type-feature', 'library']
title = 'Support pickling objects with __new__ with keyword arguments with protocol 2+'
updated_at = <Date 2015-11-17.11:15:19.259>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2015-11-17.11:15:19.259>
actor = 'python-dev'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-10-13.18:29:17.151>
closer = 'serhiy.storchaka'
components = ['Extension Modules', 'Interpreter Core', 'Library (Lib)']
creation = <Date 2015-05-11.10:49:50.457>
creator = 'serhiy.storchaka'
dependencies = []
files = ['39341', '40751', '40771']
hgrepos = []
issue_num = 24164
keywords = ['patch']
message_count = 19.0
messages = ['242890', '251830', '251857', '252743', '252744', '252773', '252809', '252811', '252854', '252855', '252856', '252897', '252916', '252924', '252945', '252946', '252950', '252951', '254793']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'alexandre.vassalotti', 'Arfrever', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24164'
versions = ['Python 3.6']

[serhiy-storchaka]

Pickling of objects of classes whose __new__ mandates the use of keyword-only arguments is supported with protocol 4 (using a new opcode NEWOBJ_EX). But it is possible to implement this feature with protocol 2+ (less efficiently than with NEWOBJ_EX). __new_ex__ is pickled as partial(cls.__new__, cls, *args, **kwargs). Pickled data is compatible with older Python releases up to 2.7 (bpo-5228).

Proposed patch adds support of __new__ with keyword arguments with protocols 2 and 3.

[vstinner]

"Pickling of objects of classes whose __new__ mandates the use of keyword-only arguments is supported with protocol 4 (using a new opcode NEWOBJ_EX)."

Hum, can you please write a short example of such class which can only be pickled by the protocol 4 currently? Just for my information.

I understand that some objects cannot be serialized by pickle with protocol lower than 4, whereas your change makes possible to serialize them on Python 3, and it will be possible to deserialize them on Python 2 and Python 3.

If I understood correctly, the change makes sense.

I reviewed the patch, it looks good to me.

[serhiy-storchaka]

Hum, can you please write a short example of such class which can only be pickled by the protocol 4 currently? Just for my information.

For now there are no such classes in the stdlib. No one implements __getnewargs_ex__. But an alternative implementation of pickling for methodcaller could use it (I implemented methodcaller pickling in bpo-22955 in different way, via __reduce_ex__, but used the same trick for passing keyword arguments to constructor).

Note that multiprocessing uses default protocol 3 (bpo-23403), and this is not configurable.

[python-dev]

New changeset bc5894a3a0e6 by Serhiy Storchaka in branch 'default':
Issue bpo-24164: Objects that need calling __new__ with keyword arguments,
https://hg.python.org/cpython/rev/bc5894a3a0e6

[serhiy-storchaka]

Thank you for your review Victor.

[vstinner]

Buildbots failed.

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/2807/steps/test/logs/stdio

======================================================================
FAIL: test_reduce (test.test_descr.PicklingTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_descr.py", line 4745, in test_reduce
    obj.__reduce_ex__(proto)
AssertionError: ValueError not raised

[python-dev]

New changeset df33dbbef7bb by Serhiy Storchaka in branch 'default':
Issue bpo-24164: Fixed test_descr: __getnewargs_ex__ now is supported in protocols 2 and 3.
https://hg.python.org/cpython/rev/df33dbbef7bb

[serhiy-storchaka]

Thank you Victor. Tests are fixed.

I think the documentation needs to be updated. Here is a patch for pickle documentation.