Idle: disable pickleability of user code objects

[pppery]

BPO	21986
Nosy	@loewis, @terryjreedy, @kbkaiser, @pitrou, @serwy, @PCManticore, @serhiy-storchaka, @pppery
Files	idlelib.patch issue21986.patch test.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2014-10-10.23:39:54.202>
created_at = <Date 2014-07-16.00:13:43.806>
labels = ['expert-IDLE', 'type-bug']
title = 'Idle: disable pickleability of user code objects'
updated_at = <Date 2014-10-13.07:21:34.837>
user = 'https://github.com/pppery'

bugs.python.org fields:

activity = <Date 2014-10-13.07:21:34.837>
actor = 'serhiy.storchaka'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2014-10-10.23:39:54.202>
closer = 'terry.reedy'
components = ['IDLE']
creation = <Date 2014-07-16.00:13:43.806>
creator = 'ppperry'
dependencies = []
files = ['35968', '35969', '36855']
hgrepos = []
issue_num = 21986
keywords = ['patch']
message_count = 20.0
messages = ['223161', '223173', '223176', '223193', '223201', '223205', '223208', '223243', '223244', '223255', '228887', '228906', '228912', '228917', '228932', '229046', '229048', '229139', '229214', '229225']
nosy_count = 9.0
nosy_names = ['loewis', 'terry.reedy', 'kbk', 'pitrou', 'roger.serwy', 'Claudiu.Popa', 'python-dev', 'serhiy.storchaka', 'ppperry']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21986'
versions = ['Python 3.4', 'Python 3.5']

[pppery]

In IDLE:

>>> code = compile("dummy_code", "<test>", "exec")
>>> pickle.dumps(code)
"cidlelib.rpc\nunpickle_code\np0\n(S'c\\x00\\x00\\x00\\x00\\x00\\x00 \\x00\\x00\\x01\\x00\\x00\\x00@\\x00\\x00\\x00s\\x08\\x00\\x00\\x00e\\x00\\x00\\x01d\\x00\\x00S(\\x01\\x00\\x00\\x00N(\\x01\\x00\\x00\\x00t\\n\\x00\\x00\\x00dummy_code(\\x00\\x00\\x00\\x00(\\x00\\x00\\x00\\x00(\\x00\\x00\\x00\\x00s\\x06\\x00\\x00\\x00<test>t\\x08\\x00\\x00\\x00<module>\\x01\\x00\\x00\\x00s\\x00\\x00\\x00\\x00'\np1\ntp2\nRp3\n."


Outside of IDLE:
>>> code = compile("dummy_code", "<test>", "exec")
>>> pickle.dumps(code)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "C:\Python27\lib\copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle code objects

Also, the error probably should be a pickle.PicklingError, not a TypeError.

[terryjreedy]

Code is really a code object, so the compile does not seem to be the problem. In 2.7, on Win7, I get exactly the same output after removing the possibly spurious space in the string posted. (ppperry), what system (OS) are you using. (In the future, please report both system and Python version).

3.4 gives a mild variation
b'\x80\x03cidlelib.rpc\nunpickle_code\nq\x00CZ\xe3\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00@\x00\x00\x00s\x08\x00\x00\x00e\x00\x00\x01d\x00\x00S)\x01N)\x01Z\ndummy_code\xa9\x00r\x01\x00\x00\x00r\x01\x00\x00\x00z\x06<test>\xda\x08<module>\x01\x00\x00\x00s\x00\x00\x00\x00q\x01\x85q\x02Rq\x03.'

I ran both test_pickle and test_pickletools from both the command line

python -m test -v test_pickletools
and from an Idle editor with F5, with no errors and apparently identical output.

import pickle
code = compile("dummy_code", "this is a file name", "exec")
print(type(code))
print('pickle: ', pickle.dumps(code))
>>> produces
<class 'code'>
pickle:  b'\x80\x03cidlelib.rpc ... \x13this is a file name\...'

which shows that the bytes come from picke, not from a garbled traceback.

Since the bytes look like an actual pickle, I tried unpickling and it works:

import pickle
code = compile("'a string'", "", "eval")
pick = pickle.dumps(code)
code2 = pickle.loads(pick)
print(eval(code), eval(code2))
>>>
a string a string

import pickle
code1 = compile("print('a string')", "", "exec")
pick = pickle.dumps(code1)
code2 = pickle.loads(pick)
exec(code1)
exec(code2)
>>> 
a string
a string

I do not see any Idle bug here, so this might be closed unless you want to make this a pickle enhancement issue to work on code objects in the standard interpreter. The 3.4 interpreter traceback gives more info on why the pickle fails there.

>>> code0 = compile("'abc'", '', 'eval')
>>> pick = pickle.dumps(code0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_pickle.PicklingError: Can't pickle <class 'code'>: attribute lookup code on builtins failed

In Idle, __builtins__.code fails, just as in the interpreter, so pickle is doing something different, and more successful, in the slightly altered Idle user-process execution environment.

[PCManticore]

It works in IDLE because it registers a custom pickling for code objects, in idlelib.rpc:

  copyreg.pickle(types.CodeType, pickle_code, unpickle_code)

where pickle_code / unpickle_code calls marshal.dumps/loads.

Although, I admit that this is weird. If idlelib.rpc is using this for transferring data between RPC instances, that's okay, but leaking the behaviour in the IDLE's interactive interpreter is not that okay, because leads to different results and expectancies between IDLE and Python's interactive interpreter.

[loewis]

I agree with Claudiu. IDLE should pickle with a private dispatch_table.

[PCManticore]

Maybe something like the attached patch. It doesn't have tests, though, I didn't find any tests for the idlelib.rpc anyway.

[serhiy-storchaka]

Instead of copying dispatch_table, use ChainMap.

[PCManticore]

Thanks, Serhiy.

[terryjreedy]

There is no unittest module for rpc yet.

Should the pickle test include a test that pickling a code object fails, and with the proper exception? Is ppperry correct about PicklingError?

Chainmap is new in 3.3 so 2.7 would need the copy version of the patch.