Issue38770
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2019-11-11 21:10 by Saim Raza, last changed 2022-04-11 14:59 by admin.
| Messages (3) | |||
|---|---|---|---|
| msg356388 - (view) | Author: Saim Raza (Saim Raza) | Date: 2019-11-11 21:10 | |
If the __qualname__ of a class is set to have a circular reference to itself, pickle behaves differently based on protocol. Following script demonstrates the issue:
======================================================
from __future__ import print_function
import pickle, sys
class Foo:
__name__ = __qualname__ = "Foo.ref"
pass
Foo.ref = Foo
print(sys.version_info)
for proto in range(0, pickle.HIGHEST_PROTOCOL + 1):
print("{}:".format(proto), end=" ")
try:
pkl = pickle.dumps(Foo, proto)
print("Dump OK,", end=" ")
assert(pickle.loads(pkl) is Foo)
print("Load OK,")
except Exception as err:
print(repr(err))
======================================================
OUTPUT:
Python2.7:
sys.version_info(major=2, minor=7, micro=16, releaselevel='final', serial=0)
0: Dump OK, Load OK,
1: Dump OK, Load OK,
2: Dump OK, Load OK,
Python3.7:
sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)
0: RecursionError('maximum recursion depth exceeded while pickling an object')
1: RecursionError('maximum recursion depth exceeded while pickling an object')
2: RecursionError('maximum recursion depth exceeded while pickling an object')
3: RecursionError('maximum recursion depth exceeded while pickling an object')
4: Dump OK, Load OK,
======================================================
This was introduced as a side effect of issue#23611 (?). I can think of the following approaches to fix the issue and make the behavior consistent:
1. Check if the class has a self-reference and raise an error for all protocols.
2. Use memoization to handle self-references. I am not sure what should be dumped in this case. In the example above `Foo` will exist in the namespace but not `Foo.ref`.
3. Dump such classes similar to Python 2 pickle and Python 3 pickle protocol >= 4.
I had a stab at pickle.py and had a bit of success in doing point 3 above. Posting this issue for discussions. I would be happy to submit a PR for this issue.
Thanks,
Saim Raza
|
|||
| msg366951 - (view) | Author: Furkan Onder (furkanonder) * | Date: 2020-04-21 23:19 | |
I ran your script and didn't get RecursionError. The issue seems to be fixed.
Python 3.8.2 (default, Apr 8 2020, 14:31:25)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>>
>>> import pickle, sys
>>>
>>> class Foo:
... __name__ = __qualname__ = "Foo.ref"
... pass
...
>>> Foo.ref = Foo
>>>
>>> print(sys.version_info)
sys.version_info(major=3, minor=8, micro=2, releaselevel='final', serial=0)
>>> for proto in range(0, pickle.HIGHEST_PROTOCOL + 1):
... print("{}:".format(proto), end=" ")
... try:
... pkl = pickle.dumps(Foo, proto)
... print("Dump OK,", end=" ")
... assert(pickle.loads(pkl) is Foo)
... print("Load OK,")
... except Exception as err:
... print(repr(err))
...
0: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed")
1: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed")
2: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed")
3: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed")
4: Dump OK, Load OK,
5: Dump OK, Load OK,
>>>
|
|||
| msg366956 - (view) | Author: Furkan Onder (furkanonder) * | Date: 2020-04-21 23:57 | |
Ahh. I misunderstood the problem. Pickle behaves differently when it is a circular reference. If you have a solution, I am waiting with curiosity. |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:59:23 | admin | set | github: 82951 |
| 2020-04-21 23:57:29 | furkanonder | set | messages: + msg366956 |
| 2020-04-21 23:19:10 | furkanonder | set | nosy:
+ furkanonder messages: + msg366951 |
| 2019-11-11 21:36:07 | serhiy.storchaka | set | assignee: serhiy.storchaka |
| 2019-11-11 21:10:17 | Saim Raza | create | |
➜