3.6.1rc1 adds the current directory to sys.path when running a subdirectory's __main__.py; previous versions did not

[nedbat]

BPO	29723
Nosy	@ncoghlan, @nedbat, @ned-deily, @ericsnowcurrently, @eryksun, @zooba
PRs	WIP: bpo-29723: Set sys.path[0] correctly for importer-as-main #571 bpo-29723: Consistently configure sys.path[0] #575 [3.6] bpo-29723: Consistently configure sys.path[0] #636 bpo-29723: Add missing NEWS entry #638

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2017-03-12.11:38:57.611>
created_at = <Date 2017-03-05.13:31:48.921>
labels = ['type-bug', '3.7', 'release-blocker']
title = "3.6.1rc1 adds the current directory to sys.path when running a subdirectory's __main__.py; previous versions did not"
updated_at = <Date 2018-01-16.07:08:50.064>
user = 'https://github.com/nedbat'

bugs.python.org fields:

activity = <Date 2018-01-16.07:08:50.064>
actor = 'serhiy.storchaka'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2017-03-12.11:38:57.611>
closer = 'ncoghlan'
components = []
creation = <Date 2017-03-05.13:31:48.921>
creator = 'nedbat'
dependencies = []
files = []
hgrepos = []
issue_num = 29723
keywords = []
message_count = 26.0
messages = ['289009', '289011', '289044', '289046', '289049', '289053', '289117', '289122', '289201', '289208', '289214', '289215', '289237', '289254', '289269', '289271', '289275', '289276', '289384', '289493', '289495', '289500', '290119', '290202', '290204', '290207']
nosy_count = 6.0
nosy_names = ['ncoghlan', 'nedbat', 'ned.deily', 'eric.snow', 'eryksun', 'steve.dower']
pr_nums = ['571', '575', '636', '638']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29723'
versions = ['Python 3.6', 'Python 3.7']

[nedbat]

3.6.1rc1 adds the current directory to sys.path when running a subdirectory's __main__.py
Previous versions, including 3.6.0, did not. Is this intentional?

$ pwd
/Users/ned/foo
$ cat main361/__main__.py
import pprint, sys
pprint.pprint(sys.path)
$ for ver in 2.7.13 3.4.6 3.5.3 3.6.0 3.6.1rc1; do
>     py=/usr/local/pythonz/pythons/CPython-$ver/bin/python
>     $py -V
>     $py main361
> done
Python 2.7.13
['main361',
 '/usr/local/pythonz/pythons/CPython-2.7.13/lib/python27.zip',
 '/usr/local/pythonz/pythons/CPython-2.7.13/lib/python2.7',
 '/usr/local/pythonz/pythons/CPython-2.7.13/lib/python2.7/plat-darwin',
 '/usr/local/pythonz/pythons/CPython-2.7.13/lib/python2.7/plat-mac',
 '/usr/local/pythonz/pythons/CPython-2.7.13/lib/python2.7/plat-mac/lib-scriptpackages',
 '/usr/local/pythonz/pythons/CPython-2.7.13/lib/python2.7/lib-tk',
 '/usr/local/pythonz/pythons/CPython-2.7.13/lib/python2.7/lib-old',
 '/usr/local/pythonz/pythons/CPython-2.7.13/lib/python2.7/lib-dynload',
 '/usr/local/pythonz/pythons/CPython-2.7.13/lib/python2.7/site-packages']
Python 3.4.6
['main361',
 '/usr/local/pythonz/pythons/CPython-3.4.6/lib/python34.zip',
 '/usr/local/pythonz/pythons/CPython-3.4.6/lib/python3.4',
 '/usr/local/pythonz/pythons/CPython-3.4.6/lib/python3.4/plat-darwin',
 '/usr/local/pythonz/pythons/CPython-3.4.6/lib/python3.4/lib-dynload',
 '/usr/local/pythonz/pythons/CPython-3.4.6/lib/python3.4/site-packages']
Python 3.5.3
['main361',
 '/usr/local/pythonz/pythons/CPython-3.5.3/lib/python35.zip',
 '/usr/local/pythonz/pythons/CPython-3.5.3/lib/python3.5',
 '/usr/local/pythonz/pythons/CPython-3.5.3/lib/python3.5/plat-darwin',
 '/usr/local/pythonz/pythons/CPython-3.5.3/lib/python3.5/lib-dynload',
 '/usr/local/pythonz/pythons/CPython-3.5.3/lib/python3.5/site-packages']
Python 3.6.0
['main361',
 '/usr/local/pythonz/pythons/CPython-3.6.0/lib/python36.zip',
 '/usr/local/pythonz/pythons/CPython-3.6.0/lib/python3.6',
 '/usr/local/pythonz/pythons/CPython-3.6.0/lib/python3.6/lib-dynload',
 '/usr/local/pythonz/pythons/CPython-3.6.0/lib/python3.6/site-packages']
Python 3.6.1rc1
['main361',
 '/Users/ned/foo',
 '/usr/local/pythonz/pythons/CPython-3.6.1rc1/lib/python36.zip',
 '/usr/local/pythonz/pythons/CPython-3.6.1rc1/lib/python3.6',
 '/usr/local/pythonz/pythons/CPython-3.6.1rc1/lib/python3.6/lib-dynload',
 '/usr/local/pythonz/pythons/CPython-3.6.1rc1/lib/python3.6/site-packages']
$

[nedbat]

BTW, I don't know if this is relevant, but PyPy has added the current directory in this situation for a long time.

[ned-deily]

I think this is a result of the changes introduced with bpo-29319. Perhaps a more prominent NEWS entry is needed? Nick? Ned, does this change in behavior cause problems?

[eryksun]

That's not the current directory. It's the 'script' directory (i.e. the parent of main361), as set by PySys_SetArgv when Py_IsolatedFlag isn't set.

The old behavior was for RunMainFromImporter to unconditionally replace this directory in sys.path with the import source for "__main__.py", which is the correct behavior, except I think it should try to get an absolute path. The resolution for bpo-29319 changed this such that when sys.path[0] is non-empty it now inserts the import source at index 0 rather than replace index 0. It seems to me that this new logic should only be used in isolated mode. Otherwise it should be replacing this bogus script directory like it used to.

[nedbat]

I don't have a specific problem that it causes. It's just a difference from any previous CPython version.

About the PyPy thing: it has always behaved this way:

$ for ver in 4.0.1 5.1.1 5.4.0 5.6.0; do py=/usr/local/pythonz/pythons/PyPy-$ver/bin/pypy; $py -V; $py main361; done
Python 2.7.10 (5f8302b8bf9f, Nov 18 2015, 10:38:03)
[PyPy 4.0.1 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
['main361',
 '/Users/ned/foo',
 '/usr/local/pythonz/pythons/PyPy-4.0.1/lib_pypy/__extensions__',
 '/usr/local/pythonz/pythons/PyPy-4.0.1/lib_pypy',
 '/usr/local/pythonz/pythons/PyPy-4.0.1/lib-python/2.7',
 '/usr/local/pythonz/pythons/PyPy-4.0.1/lib-python/2.7/lib-tk',
 '/usr/local/pythonz/pythons/PyPy-4.0.1/lib-python/2.7/plat-darwin',
 '/usr/local/pythonz/pythons/PyPy-4.0.1/lib-python/2.7/plat-mac',
 '/usr/local/pythonz/pythons/PyPy-4.0.1/lib-python/2.7/plat-mac/lib-scriptpackages',
 '/usr/local/pythonz/pythons/PyPy-4.0.1/site-packages']
Python 2.7.10 (b0a649e90b66, Apr 28 2016, 08:33:07)
[PyPy 5.1.1 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
['main361',
 '/Users/ned/foo',
 '/usr/local/pythonz/pythons/PyPy-5.1.1/lib_pypy/__extensions__',
 '/usr/local/pythonz/pythons/PyPy-5.1.1/lib_pypy',
 '/usr/local/pythonz/pythons/PyPy-5.1.1/lib-python/2.7',
 '/usr/local/pythonz/pythons/PyPy-5.1.1/lib-python/2.7/lib-tk',
 '/usr/local/pythonz/pythons/PyPy-5.1.1/lib-python/2.7/plat-darwin',
 '/usr/local/pythonz/pythons/PyPy-5.1.1/lib-python/2.7/plat-mac',
 '/usr/local/pythonz/pythons/PyPy-5.1.1/lib-python/2.7/plat-mac/lib-scriptpackages',
 '/usr/local/pythonz/pythons/PyPy-5.1.1/site-packages']
Python 2.7.10 (77392ad26350, Aug 28 2016, 05:07:44)
[PyPy 5.4.0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
['main361',
 '/Users/ned/foo',
 '/usr/local/pythonz/pythons/PyPy-5.4.0/lib_pypy/__extensions__',
 '/usr/local/pythonz/pythons/PyPy-5.4.0/lib_pypy',
 '/usr/local/pythonz/pythons/PyPy-5.4.0/lib-python/2.7',
 '/usr/local/pythonz/pythons/PyPy-5.4.0/lib-python/2.7/lib-tk',
 '/usr/local/pythonz/pythons/PyPy-5.4.0/lib-python/2.7/plat-darwin',
 '/usr/local/pythonz/pythons/PyPy-5.4.0/lib-python/2.7/plat-mac',
 '/usr/local/pythonz/pythons/PyPy-5.4.0/lib-python/2.7/plat-mac/lib-scriptpackages',
 '/usr/local/pythonz/pythons/PyPy-5.4.0/site-packages']
Python 2.7.12 (aff251e54385, Nov 09 2016, 17:25:49)
[PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
['main361',
 '/Users/ned/foo',
 '/usr/local/pythonz/pythons/PyPy-5.6.0/lib_pypy/__extensions__',
 '/usr/local/pythonz/pythons/PyPy-5.6.0/lib_pypy',
 '/usr/local/pythonz/pythons/PyPy-5.6.0/lib-python/2.7',
 '/usr/local/pythonz/pythons/PyPy-5.6.0/lib-python/2.7/lib-tk',
 '/usr/local/pythonz/pythons/PyPy-5.6.0/lib-python/2.7/plat-darwin',
 '/usr/local/pythonz/pythons/PyPy-5.6.0/lib-python/2.7/plat-mac',
 '/usr/local/pythonz/pythons/PyPy-5.6.0/lib-python/2.7/plat-mac/lib-scriptpackages',
 '/usr/local/pythonz/pythons/PyPy-5.6.0/site-packages']
$ for ver in 2.4.0 5.5.0; do py=/usr/local/pythonz/pythons/PyPy3-$ver/bin/pypy3; $py -V; $py main361; done
Python 3.2.5 (b2091e973da6, Oct 19 2014, 18:30:58)
[PyPy 2.4.0 with GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)]
['main361',
 '/Users/ned/foo',
 '/usr/local/pythonz/pythons/PyPy3-2.4.0/lib_pypy/__extensions__',
 '/usr/local/pythonz/pythons/PyPy3-2.4.0/lib_pypy',
 '/usr/local/pythonz/pythons/PyPy3-2.4.0/lib-python/3',
 '/usr/local/pythonz/pythons/PyPy3-2.4.0/lib-python/3/lib-tk',
 '/usr/local/pythonz/pythons/PyPy3-2.4.0/lib-python/3/plat-darwin',
 '/usr/local/pythonz/pythons/PyPy3-2.4.0/lib-python/3/plat-mac',
 '/usr/local/pythonz/pythons/PyPy3-2.4.0/lib-python/3/plat-mac/lib-scriptpackages',
 '/usr/local/pythonz/pythons/PyPy3-2.4.0/site-packages']
Python 3.3.5 (619c0d5af0e5, Oct 08 2016, 22:08:19)
[PyPy 5.5.0-alpha0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
['main361',
 '/Users/ned/foo',
 '/usr/local/pythonz/pythons/PyPy3-5.5.0/lib_pypy/__extensions__',
 '/usr/local/pythonz/pythons/PyPy3-5.5.0/lib_pypy',
 '/usr/local/pythonz/pythons/PyPy3-5.5.0/lib-python/3',
 '/usr/local/pythonz/pythons/PyPy3-5.5.0/lib-python/3/lib-tk',
 '/usr/local/pythonz/pythons/PyPy3-5.5.0/lib-python/3/plat-darwin',
 '/usr/local/pythonz/pythons/PyPy3-5.5.0/lib-python/3/plat-mac',
 '/usr/local/pythonz/pythons/PyPy3-5.5.0/lib-python/3/plat-mac/lib-scriptpackages',
 '/usr/local/pythonz/pythons/PyPy3-5.5.0/site-packages']
$

[eryksun]

I don't have a specific problem that it causes.

Adding an arbitrary directory ahead of the standard library in sys.path is a problem waiting to happen. This also happens when the import source is a zip file, e.g. a .pyz app.

[ncoghlan]

I think Eryk's diagnosis is correct: this is a straight up bug in the original patch, where it's missing the check to only use the new logic when in isolated mode (as it's compensating for the fact that there is no extra sys.path[0] entry inserted in that case).

I mentioned that on the original issue http://bugs.python.org/issue29319#msg285904 but hadn't noticed that the relevant check was missing from the applied patch.

[ned-deily]

Thanks for the analysis, Eryk and Nick. Then this sounds like a release blocker problem that should be fixed for 3.6.1. Steve, can you take a look, please?

[zooba]

It's actually "adding" the current directory by not replacing the empty string that's normally there, presumably because it's already been resolved into a path at this stage. The behavior on Windows is correct, so I expect it's actually a difference between getpath.c and getpathp.c, rather than an isolated/non-isolated mode issue.

I'll try and take a look - my plate is fairly full over the next few weeks though, so if anyone is able to help out I'd appreciate it, especially since it's not my usual platform.

[eryksun]

It's actually "adding" the current directory

It's the script directory, which gets added on all platforms when PySys_SetArgv is called with Py_IsolatedFlag == 0. In this case we're running "__main__.py" from a directory or zip file, and we don't want its parent directory in sys.path.

The behavior on Windows is correct

Here's what I see on a current build of 3.6:

    C:\Temp>python_d -i main361
    >>> import sys
    >>> sys.version
    '3.6.0+ (default, Mar  8 2017, 06:51:22) [MSC v.1900 64 bit (AMD64)]'
    >>> sys.path[:2]
    ['main361', 'C:\\Temp']

C:\Temp doesn't belong in sys.path in this case. When we're not in isolated mode, RunMainFromImporter should set the __main__.py import source as index 0 rather than insert it.

[ncoghlan]

Ah, interesting, I didn't know there was a difference between the platforms in when the placeholder got resolved to a full path.

However, after browsing the code and running some local tests, it seems that injecting sys.path[0] isn't handled by Py_GetPath() or PySys_SetPath(), regardless of OS.

Instead, it's handled as a side effect of calling PySys_SetArgV(), with PySys_SetArgVEx() adding a flag to disable the side effect (for the benefit of the isolated mode implementation).

The actual side effect itself is implemented in sys_update_path: https://github.com/python/cpython/blob/3.6/Python/sysmodule.c#L2162

So in the case of running ./python Tools (which exercises the paths of interest here) with some strategically placed printf() and PySys_FormatStdout() calls, I get:

$ ./python Tools
module_search_path: /usr/local/lib/python36.zip:/home/ncoghlan/devel/py36/Lib:/home/ncoghlan/devel/py36/Lib:/home/ncoghlan/devel/py36/build/lib.linux-x86_64-3.6
sys.path[0]: '/home/ncoghlan/devel/py36'
sys.path: ['/home/ncoghlan/devel/py36', '/usr/local/lib/python36.zip', '/home/ncoghlan/devel/py36/Lib', '/home/ncoghlan/devel/py36/build/lib.linux-x86_64-3.6', '/home/ncoghlan/.local/lib/python3.6/site-packages']
/home/ncoghlan/devel/py36/python: can't find '__main__' module in 'Tools'

The first line is from Py_GetPath(), the second is from sys_update_path(), the third is from RunMainFromImporter (before it makes any sys.path changes), and the last is the expected error because our Tools directory isn't executable.

In this scenario, we want the "Tools" entry to *overwrite* sys.path[0].

While in isolated mode I get:

$ ./python -I Tools
module_search_path: /usr/local/lib/python36.zip:/home/ncoghlan/devel/py36/Lib:/home/ncoghlan/devel/py36/Lib:/home/ncoghlan/devel/py36/build/lib.linux-x86_64-3.6
sys.path: ['/usr/local/lib/python36.zip', '/home/ncoghlan/devel/py36/Lib', '/home/ncoghlan/devel/py36/build/lib.linux-x86_64-3.6']
/home/ncoghlan/devel/py36/python: can't find '__main__' module in 'Tools'

In this scenario, we want the "Tools" entry to be inserted *before* sys.path[0].

Note that this has been buggy since the -I switch was introduced, but the entry we've been overwriting has been the one for the stdlib-as-a-zip-archive, so it didn't cause any problems for anyone using the default directory-based installation layout.

However, we can't reliably figure out which to do based on the contents of sys.path, we need to look at Py_IsolatedFlag instead. (I'm not sure why the current check is working on Windows, as sys_update_path() attempts to resolve an absolute path entry from the archive or directory name there as well)

[ncoghlan]

So a potentially more robust fix here would be to always call PySys_SetArgVEx(argc, argv, 0) rather than the plain PySys_SetArgV when we know we're going to be relying on RunMainFromImporter.

That way RunMainFromImporter could just *always* insert at the front, and not have to try to guess whether or not a different sys.path[0] entry had already been populated.

This would involve splitting RunMainFromImporter into two pieces (one to check whether the given filename is a valid sys.path entry, the second to actually modify sys.path and run __main__), but that's entirely feasible.

[zooba]

C:\Temp doesn't belong in sys.path in this case

Hang on, why not? If I were running a module.py then it would be, so why is a package\main.py different (and not able to import itself or its siblings)? The package is the "script" being run here, yes?

Or is __main__.py different from __init__.py in this regard? (That is, it isn't *part* of the package, it's just a loose script that happens to inherit the name of its directory)