Support out-of-band pickling for builtin types

[jakirkham]

BPO	40718
Nosy	@pitrou, @jakirkham

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-05-21.19:25:54.076>
labels = ['3.8', '3.9', '3.10', 'performance', '3.11', 'library']
title = 'Support out-of-band pickling for builtin types'
updated_at = <Date 2021-09-30.00:30:58.504>
user = 'https://github.com/jakirkham'

bugs.python.org fields:

activity = <Date 2021-09-30.00:30:58.504>
actor = 'jakirkham'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-05-21.19:25:54.076>
creator = 'jakirkham'
dependencies = []
files = []
hgrepos = []
issue_num = 40718
keywords = []
message_count = 1.0
messages = ['369533']
nosy_count = 2.0
nosy_names = ['pitrou', 'jakirkham']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue40718'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[jakirkham]

It would be nice (where possible) to support out-of-band pickling of builtin bytes-like types. This would allow binary data from these objects to be shipped along separately zero-copy and later reconstructed during unpickling.

It seems that bytes, bytearray, and array would be good candidates for this behavior. Not sure if mmap or memoryview would make sense as it might not be clear on how to reconstruct them during unpickling, but if someone sees a way those would be nice to support too.

To illustrate this a bit, here is the behavior with a bytes object today:

In [1]: import pickle

In [2]: b = b"abc"

In [3]: l = []

In [4]: p = pickle.dumps(b, protocol=5, buffer_callback=l.append)

In [5]: l
Out[5]: []

With this change, we would see this behavior instead:

In [1]: import pickle

In [2]: b = b"abc"

In [3]: l = []

In [4]: p = pickle.dumps(b, protocol=5, buffer_callback=l.append)

In [5]: l
Out[5]: [<pickle.PickleBuffer at 0x10efe7540>]

(This is my first Python bug submission. So apologies if I got turned around here. Please go easy on me :)