Improve pickling of typing types

[serhiy-storchaka]

BPO	33652
Nosy	@gvanrossum, @ned-deily, @serhiy-storchaka, @ilevkivskyi, @miss-islington
PRs	bpo-33652: Improve pickle support in the typing module. #7123 [3.7] bpo-33652: Improve pickle support in the typing module. (GH-7123) #7132 bpo-33652: Remove __getstate__ and __setstate__ methods in typing. #7144 [3.7] bpo-33652: Remove __getstate__ and __setstate__ methods in typing. (GH-7144) #7148

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-28.12:57:46.782>
created_at = <Date 2018-05-26.07:10:15.083>
labels = ['3.8', '3.7', 'library']
title = 'Improve pickling of typing types'
updated_at = <Date 2018-05-28.12:57:46.781>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-05-28.12:57:46.781>
actor = 'levkivskyi'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-28.12:57:46.782>
closer = 'levkivskyi'
components = ['Library (Lib)']
creation = <Date 2018-05-26.07:10:15.083>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 33652
keywords = ['patch']
message_count = 8.0
messages = ['317727', '317752', '317753', '317766', '317768', '317837', '317839', '317846']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'ned.deily', 'serhiy.storchaka', 'levkivskyi', 'miss-islington']
pr_nums = ['7123', '7132', '7144', '7148']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33652'
versions = ['Python 3.7', 'Python 3.8']

[serhiy-storchaka]

The following PR makes pickles for typing types more portable. Type variables no longer use _find_name() and can be unpickled in 3.6. Subscripted generics no longer expose internals and can be unpickled in 3.6 and future versions with changed internal implementation.

Before this PR:

>>> import pickle, pickletools, typing                                            
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.T, 4)))              
    0: \x80 PROTO      4
    2: \x95 FRAME      30
   11: \x8c SHORT_BINUNICODE 'typing'
   19: \x94 MEMOIZE    (as 0)
   20: \x8c SHORT_BINUNICODE '_find_name'
   32: \x93 STACK_GLOBAL
   33: h    BINGET     0
   35: \x8c SHORT_BINUNICODE 'T'
   38: \x86 TUPLE2
   39: R    REDUCE
   40: .    STOP
highest protocol among opcodes = 4
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.Union[int, str], 4)))
    0: \x80 PROTO      4
    2: \x95 FRAME      198
   11: \x8c SHORT_BINUNICODE 'copyreg'
   20: \x8c SHORT_BINUNICODE '_reconstructor'
   36: \x93 STACK_GLOBAL
   37: \x8c SHORT_BINUNICODE 'typing'
   45: \x94 MEMOIZE    (as 0)
   46: \x8c SHORT_BINUNICODE '_GenericAlias'
   61: \x93 STACK_GLOBAL
   62: \x8c SHORT_BINUNICODE 'builtins'
   72: \x94 MEMOIZE    (as 1)
   73: \x8c SHORT_BINUNICODE 'object'
   81: \x93 STACK_GLOBAL
   82: N    NONE
   83: \x87 TUPLE3
   84: R    REDUCE
   85: }    EMPTY_DICT
   86: (    MARK
   87: \x8c     SHORT_BINUNICODE '_inst'
   94: \x88     NEWTRUE
   95: \x8c     SHORT_BINUNICODE '_special'
  105: \x89     NEWFALSE
  106: \x8c     SHORT_BINUNICODE '_name'
  113: N        NONE
  114: \x8c     SHORT_BINUNICODE '__origin__'
  126: h        BINGET     0
  128: \x8c     SHORT_BINUNICODE 'Union'
  135: \x93     STACK_GLOBAL
  136: \x8c     SHORT_BINUNICODE '__args__'
  146: h        BINGET     1
  148: \x8c     SHORT_BINUNICODE 'int'
  153: \x93     STACK_GLOBAL
  154: h        BINGET     1
  156: \x8c     SHORT_BINUNICODE 'str'
  161: \x93     STACK_GLOBAL
  162: \x86     TUPLE2
  163: \x8c     SHORT_BINUNICODE '__parameters__'
  179: )        EMPTY_TUPLE
  180: \x8c     SHORT_BINUNICODE '__slots__'
  191: N        NONE
  192: \x8c     SHORT_BINUNICODE '__module__'
  204: h        BINGET     0
  206: u        SETITEMS   (MARK at 86)
  207: b    BUILD
  208: .    STOP
highest protocol among opcodes = 4

With this PR:

>>> import pickle, pickletools, typing                                                              
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.T, 4)))                       
    0: \x80 PROTO      4
    2: \x95 FRAME      13
   11: \x8c SHORT_BINUNICODE 'typing'
   19: \x8c SHORT_BINUNICODE 'T'
   22: \x93 STACK_GLOBAL
   23: .    STOP
highest protocol among opcodes = 4
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.Union[int, str], 4)))       
    0: \x80 PROTO      4
    2: \x95 FRAME      66
   11: \x8c SHORT_BINUNICODE '_operator'
   22: \x8c SHORT_BINUNICODE 'getitem'
   31: \x93 STACK_GLOBAL
   32: \x8c SHORT_BINUNICODE 'typing'
   40: \x8c SHORT_BINUNICODE 'Union'
   47: \x93 STACK_GLOBAL
   48: \x8c SHORT_BINUNICODE 'builtins'
   58: \x94 MEMOIZE    (as 0)
   59: \x8c SHORT_BINUNICODE 'int'
   64: \x93 STACK_GLOBAL
   65: h    BINGET     0
   67: \x8c SHORT_BINUNICODE 'str'
   72: \x93 STACK_GLOBAL
   73: \x86 TUPLE2
   74: \x86 TUPLE2
   75: R    REDUCE
   76: .    STOP
highest protocol among opcodes = 4

If there is a chance it would be nice to merge these changes into 3.7. Otherwise either pickles created in 3.7.0 will be incompatible not only with older Python versions, but with future Python versions too, or we will need to add complex code for supporting specific 3.7.0 pickles in future versions.

[ilevkivskyi]

New changeset 09f3221 by Ivan Levkivskyi (Serhiy Storchaka) in branch 'master':
bpo-33652: Improve pickle support in the typing module. (GH-7123)
09f3221

[miss-islington]

New changeset d498625 by Miss Islington (bot) in branch '3.7':
bpo-33652: Improve pickle support in the typing module. (GH-7123)
d498625

[serhiy-storchaka]

There is a question -- what to do with all these __getstate__ and __setstate__ methods? They are part of the pickle protocol, but they are not used when define __reduce__. And they are not needed for supporting compatibility with older versions, because in 3.6 pickleable generic types were pickled by name. I think it is better to remove these methods.

[ilevkivskyi]

Yes, these are just legacy from times when TypeVars were serialized by value, not by identity like now. I think it should be safe to remove them. Would you like to make a PR?

[ilevkivskyi]

New changeset 97b523d by Ivan Levkivskyi (Serhiy Storchaka) in branch 'master':
bpo-33652: Remove __getstate__ and __setstate__ methods in typing. (GH-7144)
97b523d

[miss-islington]

New changeset 98b089e by Miss Islington (bot) in branch '3.7':
bpo-33652: Remove __getstate__ and __setstate__ methods in typing. (GH-7144)
98b089e

[ilevkivskyi]

I think this can be closed now.