Reloading tokenize breaks tokenize.open()

[takluyver]

BPO	23615
Nosy	@brettcannon, @ncoghlan, @vstinner, @ericsnowcurrently, @takluyver, @serhiy-storchaka
Files	tokenize-reloadable.patch tokenize-reloadable-B.patch tokenize-reloadable-B2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-03-11.15:35:00.110>
created_at = <Date 2015-03-09.01:21:08.683>
labels = ['type-bug', 'library']
title = 'Reloading tokenize breaks tokenize.open()'
updated_at = <Date 2015-03-11.21:54:20.069>
user = 'https://github.com/takluyver'

bugs.python.org fields:

activity = <Date 2015-03-11.21:54:20.069>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-03-11.15:35:00.110>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2015-03-09.01:21:08.683>
creator = 'takluyver'
dependencies = []
files = ['38423', '38425', '38426']
hgrepos = []
issue_num = 23615
keywords = ['patch']
message_count = 11.0
messages = ['237588', '237782', '237784', '237787', '237788', '237790', '237798', '237870', '237876', '237904', '237906']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'ncoghlan', 'vstinner', 'python-dev', 'eric.snow', 'takluyver', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23615'
versions = ['Python 3.4', 'Python 3.5']

[takluyver]

Issue bpo-22599 changed tokenize.open() from using builtins.open() to having a module-level reference to _builtin_open, stored by doing _builtin_open = open. However, on reloading the module, _builtin_open is pointed to tokenize.open from the last execution of the code, breaking tokenize.open():

>>> import tokenize
>>> tokenize.open
<function open at 0x7f15b660fc80>
>>> tokenize._builtin_open
<built-in function open>
>>> import imp; imp.reload(tokenize)
<module 'tokenize' from '/home/takluyver/miniconda3/envs/py34/lib/python3.4/tokenize.py'>
>>> tokenize.open
<function open at 0x7f15b660fbf8>
>>> tokenize._builtin_open
<function open at 0x7f15b660fc80>
>>> tokenize.open('foo.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/takluyver/miniconda3/envs/py34/lib/python3.4/tokenize.py", line 438, in open
    buffer = _builtin_open(filename, 'rb')
TypeError: open() takes 1 positional argument but 2 were given

The actual case where I'm seeing this error is nose logging a failure in a test suite, so it's not clear what is reloading tokenize, but it appears that something is. This just started failing when our Windows test system updated to Python 3.4.3.

[takluyver]

Patch attached to fix this.

[serhiy-storchaka]

The same issue exists in Lib/bz2.py, Lib/tarfile.py, Tools/freeze/bkfile.py.

May be write a code as

from builtins import open as _builtin_open

?

[takluyver]

-B.patch as Serhiy suggests, for tokenize only for the time being.

[serhiy-storchaka]

LGTM. Do you want to write a patch for other cases Thomas?

[takluyver]

Fixed the other three cases you pointed out (-B2.patch).

[serhiy-storchaka]

LGTM.

[brettcannon]

LGTM as well. You want to commit, Serhiy? If not assign to me and I will get to it on Friday.

[python-dev]

New changeset 383ba3699084 by Serhiy Storchaka in branch '3.4':
Issue bpo-23615: Modules bz2, tarfile and tokenize now can be reloaded with
https://hg.python.org/cpython/rev/383ba3699084

New changeset 6e736a57a482 by Serhiy Storchaka in branch 'default':
Issue bpo-23615: Modules bz2, tarfile and tokenize now can be reloaded with
https://hg.python.org/cpython/rev/6e736a57a482

New changeset 36bd5add9732 by Serhiy Storchaka in branch '2.7':
Issue bpo-23615: Module tarfile is now can be reloaded with imp.reload().
https://hg.python.org/cpython/rev/36bd5add9732

[vstinner]

Is there a method to detect other reload bugs?

[serhiy-storchaka]

Only grep.