Can't use lib2to3 with embeddable zip file.

[OlegN]

BPO	24960
Nosy	@gpshead, @pfmoore, @ncoghlan, @benjaminp, @dstufft, @ppinard, @Lukas00500
PRs	bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files #4977 [3.6] bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (GH-4977) #4979 [2.7] bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (GH-4977) #4980 make PatternCompiler use the packaged grammar if possible (more bpo-24960) #5034 [3.6] make PatternCompiler use the packaged grammar if possible (more bpo-24960) (GH-5034) #5036 [2.7] make PatternCompiler use the packaged grammar if possible (more bpo-24960) (GH-5034) #5037
Files	pgen.py pgen.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-12-22.20:52:37.021>
created_at = <Date 2015-08-29.17:36:22.794>
labels = ['3.7', 'library', 'performance']
title = "Can't use lib2to3 with embeddable zip file."
updated_at = <Date 2018-04-03.21:10:50.278>
user = 'https://bugs.python.org/OlegN'

bugs.python.org fields:

activity = <Date 2018-04-03.21:10:50.278>
actor = 'Lukasz00500'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-22.20:52:37.021>
closer = 'benjamin.peterson'
components = ['Library (Lib)']
creation = <Date 2015-08-29.17:36:22.794>
creator = 'Oleg N'
dependencies = []
files = ['46866', '46869']
hgrepos = []
issue_num = 24960
keywords = ['patch']
message_count = 18.0
messages = ['249329', '249804', '255441', '269765', '269779', '285753', '285756', '292529', '293780', '293855', '294105', '308940', '308941', '308942', '309153', '309154', '309155', '314896']
nosy_count = 13.0
nosy_names = ['gregory.p.smith', 'paul.moore', 'ncoghlan', 'benjamin.peterson', 'dstufft', 'Marcus.Smith', 'Oleg N', 'Florian Hassanen', 'fourplusone', 'S\xc3\xa9bastien Taylor', 'ppinard', 'kirk.d.sayre', 'Lukasz00500']
pr_nums = ['4977', '4979', '4980', '5034', '5036', '5037']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue24960'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

[OlegN]

Error: [Errno 2] No such file or directory: '...\\python35.zip\\lib2to3\\Grammar.txt'. Python35.zip from this archive: https://www.python.org/ftp/python/3.5.0/python-3.5.0rc2-embed-amd64.zip. Path is correct.

[dstufft]

This looks more like lib2to3 doesn't support running from a .zip archive.

[FlorianHassanen]

got this traceback:

Traceback (most recent call last):
  File "setup.py", line 169, in <module>
    dist = setuptools.setup(**setup_params)
  File "distutils\core.py", line 134, in setup
  File "...\setuptools\setuptools\dist.py", line 299, in parse_command_line
    result = _Distribution.parse_command_line(self)
  File "distutils\dist.py", line 469, in parse_command_line
  File "...\setuptools\setuptools\dist.py", line 679, in handle_display_options
    return _Distribution.handle_display_options(self, option_order)
  File "distutils\dist.py", line 672, in handle_display_options
  File "...\setuptools\setuptools\dist.py", line 439, in print_commands
    cmdclass = ep.resolve()
  File "...\setuptools\pkg_resources\__init__.py", line 2383, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "...\setuptools\setuptools\command\build_py.py", line 10, in <module>
    from setuptools.lib2to3_ex import Mixin2to3
  File "...\setuptools\setuptools\lib2to3_ex.py", line 12, in <module>
    from lib2to3.refactor import RefactoringTool, get_fixers_from_package
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "lib2to3\refactor.py", line 27, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "lib2to3\fixer_util.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "lib2to3\pygram.py", line 32, in <module>
  File "lib2to3\pgen2\driver.py", line 121, in load_grammar
  File "lib2to3\pgen2\pgen.py", line 385, in generate_grammar
  File "lib2to3\pgen2\pgen.py", line 15, in __init__
FileNotFoundError: [Errno 2] No such file or directory: '...\\python-3.5.0-embed-amd64\\python35.zip\\lib2to3\\Grammar.txt'

triggered e.g. by "python setup.py --help-commands" (when building setuptools)

workaround is to extract "...\python-3.5.0-embed-amd64\python35.zip" into a __directory__ named exactly the same (hence a directory named "python35.zip" which replaces the original zip file)

[fourplusone]

The problem is indeed that lib2to3 fails to load the Grammar.txt file if it is located inside a .zip archive.

I think one way to fix it is using of the "get_data" function of the loader https://docs.python.org/3/library/importlib.html#importlib.abc.ResourceLoader.get_data

However, this would make caching the Grammar more difficult, as one cannot write back the pickled dict to a zip file

Alternatively one could generate an importable *.pyc file from the grammar during the build process

[ncoghlan]

(issue title updated to indicate this appears to be lib2to3 related and added Benjamin Peterson to the nosy list)

The build process already generates a _sysconfigdata.py file in order to pass along build time details to the sysconfig module in a way that isn't dependent on having otherwise build-time only files accessible at runtime.

Doing something similar with a _lib2to3grammar.py file seems like it could be a reasonable approach to making lib2to3 zip archive friendly, but I'm not familiar enough with pgen2's internals to say how feasible that would be.

[pfmoore]

I'm not sure I see why you'd want to run lib2to3 with the embedded distribution anyway. Surely you'd do any 2to3 conversion in a normal development installation, and then simply bundle the resulting converted scripts with your application and the embedded interpreter?

[SbastienTaylor]

Are there existing tools for bundling the python35.zip (and adding additional packages)? I personally ran into this issue by using pip to add additional modules to my embeddable python but if there's a better way to migrate from devel to embeddable python I'd rather not have to use pip for this.

[ppinard]

As Sébastien Taylor, I ran into the same problem. The workaround I found was to unzip the content of python35.zip and put it in the Lib/ folder.

[kirkdsayre]

I've been fighting with this issue today trying to import networkx in the embedded Python shipped with IDA-Pro. I have implemented a local fix to pgen.py that allows networkx to import and be used in IDAPython scripts. I added a check to see if a .zip archive appears in the path of the file loaded in the constructor and if so I use the zipfile package to open the archived file rather than the standard file open() function.

I've uploaded the modified pgen.py in case any one is interested.

[kirkdsayre]

Here is a more robust version of my local fix for this problem.

[gpshead]

re: Paul's "I'm not sure I see why you'd want to run lib2to3 with the embedded distribution anyway. Surely you'd do any 2to3 conversion in a normal development installation"

lib2to3 is a library useful for doing things with Python code. Not just a stand alone tool.

Kirk: Could you please post your change as a pull request on https://github.com/python/cpython/ rather than attaching individual files here. a full file isn't a reviewable change.

Also, Grammar.txt is loaded and saved as a cached pickle file for speed. When packaging up an embedded stdlib bundle I recommend shipping that pickle. (loaded in pgen2/grammar.py).

both probably need to use https://docs.python.org/3.6/library/pkgutil.html#pkgutil.get_data to read the file out of the package rather than assuming it lives on a filesystem.

using the zipfile module within lib2to3 to try and figure out when it should look somewhere other than a filesystem is the wrong approach.