Embedded 3.6.0 distribution cannot run pyz files

[pfmoore]

BPO	29319
Nosy	@warsaw, @pfmoore, @ncoghlan, @larryhastings, @tjguk, @ned-deily, @ericfrederich, @zware, @zooba
PRs	bpo-29723: Consistently configure sys.path[0] #575 [3.6] bpo-29723: Consistently configure sys.path[0] #636 [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552 [3.5] bpo-32551: Consistently configure sys.path[0] (GH-575) #5197

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-03-12.11:41:29.355>
created_at = <Date 2017-01-19.12:11:31.582>
labels = ['type-bug', '3.7', 'OS-windows']
title = 'Embedded 3.6.0 distribution cannot run pyz files'
updated_at = <Date 2018-04-19.20:42:09.033>
user = 'https://github.com/pfmoore'

bugs.python.org fields:

activity = <Date 2018-04-19.20:42:09.033>
actor = 'barry'
assignee = 'none'
closed = True
closed_date = <Date 2017-03-12.11:41:29.355>
closer = 'ncoghlan'
components = ['Windows']
creation = <Date 2017-01-19.12:11:31.582>
creator = 'paul.moore'
dependencies = []
files = []
hgrepos = []
issue_num = 29319
keywords = []
message_count = 32.0
messages = ['285783', '285785', '285789', '285800', '285801', '285811', '285814', '285822', '285825', '285904', '287008', '287009', '287011', '287017', '287022', '289181', '289182', '289194', '289198', '289304', '289332', '289349', '289356', '289357', '289363', '289369', '289496', '290120', '290203', '290208', '310132', '310494']
nosy_count = 10.0
nosy_names = ['barry', 'paul.moore', 'ncoghlan', 'larry', 'tim.golden', 'ned.deily', 'python-dev', 'eric.frederich', 'zach.ware', 'steve.dower']
pr_nums = ['575', '636', '552', '5197']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29319'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[pfmoore]

Trying to run a pyz file using the embedded distribution for 3.6.0, I get an error "Could not import runpy module".

To reproduce, see below:

type .\main.py
print('Hello, world')
zip test.pyz __main__.py
adding: __main__.py (172 bytes security) (stored 0%)
unzip .\python-3.6.0-embed-amd64.zip
...
.\python.exe .\test.pyz
Could not import runpy module
ModuleNotFoundError: No module named 'runpy'

Running with the standard interpreter works fine:

py -V
Python 3.6.0
py .\test.pyz
Hello, world

[pfmoore]

I just checked, and 3.6.0b1 (the only prerelease version I had available) has the same problem. 3.5.2 works fine.

I thought I'd had similar code working during the beta cycle, but I can't demonstrate that any more, so maybe my recollection is wrong :-(

[pfmoore]

Confirmed that it works with alpha 2, 3 and 4. But fails with beta 1 and the release version.

[zooba]

Does running with -v provide any more hints?

I'm also interested in whether the alphas work when you don't have a full install on the same machine. PC/getpathp.c changed for b1. But that's probably less important.

Perhaps runpy just never made it into the python36.zip file?

[pfmoore]

Sorry I should have thought of trying -v. The output (included below) doesn't seem to offer many hints, though. runpy.pyc is in python36.zip, I checked that.

I'll see if I can find a machine without Python installed to test that case.

>.\python.exe -v .\test.pyz
import _frozen_importlib # frozen
import _imp # builtin
import sys # builtin
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
# zipimport: found 609 names in 'C:\\Work\\Scratch\\test\\python36.zip'
import 'zlib' # <class '_frozen_importlib.BuiltinImporter'>
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import codecs # loaded from Zip C:\Work\Scratch\test\python36.zip\codecs.pyc
# zipimport: zlib available
# zipimport: zlib available
import encodings.aliases # loaded from Zip C:\Work\Scratch\test\python36.zip\encodings\aliases.pyc
import encodings # loaded from Zip C:\Work\Scratch\test\python36.zip\encodings\__init__.pyc
# zipimport: zlib available
# zipimport: zlib available
import encodings.utf_8 # loaded from Zip C:\Work\Scratch\test\python36.zip\encodings\utf_8.pyc
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# zipimport: zlib available
# zipimport: zlib available
import encodings.latin_1 # loaded from Zip C:\Work\Scratch\test\python36.zip\encodings\latin_1.pyc
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
import _weakrefset # loaded from Zip C:\Work\Scratch\test\python36.zip\_weakrefset.pyc
import abc # loaded from Zip C:\Work\Scratch\test\python36.zip\abc.pyc
import io # loaded from Zip C:\Work\Scratch\test\python36.zip\io.pyc
Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32
# zipimport: zlib available
# zipimport: zlib available
import encodings.cp437 # loaded from Zip C:\Work\Scratch\test\python36.zip\encodings\cp437.pyc
# zipimport: found 1 names in '.\\test.pyz'
Could not import runpy module
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 948, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'runpy'
# clear builtins._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.__interactivehook__
# clear sys.flags
# clear sys.float_info
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup[2] removing builtins
# cleanup[2] removing sys
# cleanup[2] removing _frozen_importlib
# cleanup[2] removing _imp
# cleanup[2] removing _warnings
# cleanup[2] removing _thread
# cleanup[2] removing _weakref
# cleanup[2] removing _frozen_importlib_external
# cleanup[2] removing _io
# cleanup[2] removing marshal
# cleanup[2] removing nt
# cleanup[2] removing winreg
# cleanup[2] removing zipimport
# cleanup[2] removing zlib
# cleanup[2] removing encodings
# destroy encodings
# cleanup[2] removing codecs
# cleanup[2] removing _codecs
# cleanup[2] removing encodings.aliases
# cleanup[2] removing encodings.utf_8
# cleanup[2] removing _signal
# cleanup[2] removing __main__
# destroy __main__
# cleanup[2] removing encodings.latin_1
# cleanup[2] removing io
# destroy io
# cleanup[2] removing abc
# destroy abc
# cleanup[2] removing _weakrefset
# destroy _weakrefset
# cleanup[2] removing encodings.cp437
# destroy zipimport
# destroy zlib
# destroy _signal
# cleanup[3] wiping _frozen_importlib
# destroy _frozen_importlib_external
# cleanup[3] wiping _imp
# cleanup[3] wiping _warnings
# cleanup[3] wiping _thread
# cleanup[3] wiping _weakref
# cleanup[3] wiping _io
# cleanup[3] wiping marshal
# cleanup[3] wiping nt
# cleanup[3] wiping winreg
# cleanup[3] wiping codecs
# cleanup[3] wiping _codecs
# cleanup[3] wiping encodings.aliases
# cleanup[3] wiping encodings.utf_8
# cleanup[3] wiping encodings.latin_1
# cleanup[3] wiping encodings.cp437
# cleanup[3] wiping sys
# cleanup[3] wiping builtins
# destroy _imp
# destroy io
# destroy _warnings
# destroy marshal
# destroy nt
# destroy _thread
# destroy _weakref
# destroy winreg
# destroy _frozen_importlib

[zooba]

I just tried it and it makes no difference.

Omitting the "._pth" file seems to fix it, but apart from bpo-29326 (benign) sys.path is fine.

Still digging, but don't worry about testing on other machines.

[zooba]

Found it in Modules/main.c in RunMainFromImporter():

    /* argv0 is usable as an import source, so put it in sys.path[0]
       and import __main__ */
    sys_path = PySys_GetObject("path");
    if (sys_path == NULL) {
        PyErr_SetString(PyExc_RuntimeError, "unable to get sys.path");
        goto error;
    }
    if (PyList_SetItem(sys_path, 0, argv0)) {
        argv0 = NULL;
        goto error;
    }
    Py_INCREF(argv0);

When running with a ._pth file, we force the -I option, which removes the empty entry at sys.path[0]. Instead, it becomes the path to the .zip file with the standard library, so when RunMainFromImporter overwrites it blindly, it's cutting out the only hope it has of finding runpy.

You can see this in a normal install by doing:

py -I -c "import sys; print(sys.path)"
[correct output]

    py -Ii test.pyz
    [output from test.pyz]
    >>> import sys; sys.path
    [incorrect output]

So we need to stop blindly overwriting sys.path[0] here. I'm not sure what the best approach would be, but maybe Nick has a preference? Perhaps we should pass the zip file path into runpy._run_module_as_main so it can initialize __path__ with it rather than making a global change? Or maybe an insert rather than a set is the right way and I'm over-thinking this.

[pfmoore]

Nice! Thanks for finding this. I don't suppose there's any chance this would qualify as a bugfix for 3.6.1? I've been holding off on working on https://github.com/pfmoore/pylaunch until 3.6 because 3.5 doesn't handle being in a subdirectory very well. It'd be a shame if this meant I could only really support 3.7+

But it is a pretty minor use case, so I'll live either way.

FWIW, I'm inclined to think we shouldn't be blindly overwriting things, at the very least we should check that what we're overwriting is what we expect. But my knowledge of this part of the interpreter is pretty much zero, so I'll defer to the experts.

[zooba]

I'd say it definitely qualifies as a bug fix, even in 3.5 (which repros), assuming we don't break any existing APIs in the process.

[ncoghlan]

I think this is just a straight-up interaction bug where we never accounted for the combination of -I with directory and zipfile execution back when -I was added.

From a bug fix perspective, I think the right thing to do in all affected versions would be to add a check to RunMainFromImporter such that it does an insert rather than a set when -I is used.

For 3.7+, it may make sense to change the signature of runpy._run_module_as_main to accept the path entry as Steve suggests, as that could also resolve the quirk where we'll run a __main__ module from *anywhere* on sys.path, rather than only from the given path entry.

The core of the current approach was implemented back in the 2.6 time frame, and then tweaked a bit in 2.7, and hence predates all the niceties offered by the importlib API.

However, the latter change would be a separate RFE rather than a bug fix.