subprocess PATH semantics and portability

[dabrahams]

BPO	8557
Nosy	@mark-summerfield, @bitdancer, @briancurtin, @dabrahams, @eryksun, @henryiii, @asottile
Files	probe.py: demonstrates portable Popen wrapper

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-04-28.10:44:53.142>
labels = ['type-feature', '3.8', '3.9', '3.10', 'docs']
title = 'subprocess PATH semantics and portability'
updated_at = <Date 2021-05-13.20:49:33.930>
user = 'https://github.com/dabrahams'

bugs.python.org fields:

activity = <Date 2021-05-13.20:49:33.930>
actor = 'Henry Schreiner'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2010-04-28.10:44:53.142>
creator = 'dabrahams'
dependencies = []
files = ['17180']
hgrepos = []
issue_num = 8557
keywords = []
message_count = 21.0
messages = ['104422', '104429', '104437', '104527', '104611', '104646', '104647', '104738', '104752', '104766', '104902', '104904', '104908', '104909', '104912', '104925', '105867', '262382', '262399', '320098', '388155']
nosy_count = 10.0
nosy_names = ['mark', 'r.david.murray', 'brian.curtin', 'docs@python', 'dabrahams', 'RubyTuesdayDONO', 'eryksun', 'pepalogik', 'Henry Schreiner', 'Anthony Sottile']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue8557'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[dabrahams]

On POSIX systems, the PATH environment variable is always used to
look up directory-less executable names passed as the first argument to Popen(...), but on Windows, PATH is only considered when shell=True is also passed.

Actually I think it may be slightly weirder than that when
shell=False, because the following holds for me:

C:\>rem ##### Prepare minimal PATH #####
C:\>set "PATH=C:\Python26\Scripts;C:\Python26;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem"

C:\>rem ##### Prepare a minimal, clean environment #####
C:\>virtualenv --no-site-packages e:\zzz
New python executable in e:\zzz\Scripts\python.exe
Installing setuptools................done.

C:\>rem ##### Show that shell=True makes the difference in determining whether PATH is respected #####
C:\>python
Python 2.6.5 (r265:79096, Mar 19 2010, 18:02:59) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.Popen(['python', '-c', 'import sys; print sys.executable'])
<subprocess.Popen object at 0x0000000001DBE080>
>>> C:\Python26\python.exe

>>> subprocess.Popen(['python', '-c', 'import sys; print sys.executable'], env={'PATH':r'e:\zzz\Scripts'})
<subprocess.Popen object at 0x0000000001F05A90>
>>> C:\Python26\python.exe

>>> subprocess.Popen(['python', '-c', 'import sys; print sys.executable'], env={'PATH':r'e:\zzz\Scripts'}, shell=True)
<subprocess.Popen object at 0x0000000001F05B00>
>>> e:\zzz\Scripts\python.exe

That is, it looks like the environment at the time Python is invoked is what counts unless I pass shell=True. I don't even seem to be able to override this behavior by changing os.environ: you can clear() it and pass env={} and subprocess.Popen(['python']) still succeeds.

This is a very important problem for portable code and one that took me hours to suss out. I think:

a) the current behavior needs to be documented
b) it needs to be fixed if possible
c) otherwise, shell=True should be the default

[dabrahams]

It's worse than I thought; there isn't even one setting for shell that works everywhere. This is what happens on POSIX (tested on Mac and Ubuntu):

$ mkdir /tmp/xxx
$ cd /tmp/xxx
xxx $ virtualenv /tmp/zzz
xxx $ python
Python 2.6.5 (r265:79063, Mar 23 2010, 08:10:08) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess import *
>>> p = Popen(['python', '-c', 'import sys;print sys.executable'], 
...           stdin=PIPE,stdout=PIPE,stderr=PIPE,
...           env={'PATH':'/tmp/zzz/bin'})
>>> stdout,stderr = p.communicate(None)
>>> print stdout
/tmp/zzz/bin/python

>> print stderr

>>> p = Popen(['python', '-c', 'import sys;print sys.executable'], shell=True,
...           stdin=PIPE,stdout=PIPE,stderr=PIPE,
...           env={'PATH':'/tmp/zzz/bin'})
>>> stdout,stderr = p.communicate(None)
>>> print stdout

>> print stderr

[mark-summerfield]

IMO there's another problem with subprocess portablity---the lack of control over encodings: see bpo-6135.

[bitdancer]

Changing the default value of shell is not an option anyway.

The behavior you describe is exactly what one should expect: the environment in which the executable is located is the environment of the process calling Popen, not the environment passed to Popen. The environment passed to Popen is the environment in which the subprocess executes. When using shell=True, this is the environment in which the shell executes, and the *shell* then looks up the executable in that new environment. As far as I know this behavior is the same on both Windows and Unix, and thus is not a portability issue. (How the shell works can be a portability issue, though.)

I'm not sure that this needs to be documented explicitly, as it is a logical consequence of how subprocesses work, but if you want to suggest a doc update I'll take a look at it.

I suspect your Unix example is about the fragility of the rules for computing sys.executable (there's an issue in the tracker about that...you may get a different result on trunk), but I haven't checked it.

[dabrahams]

I wrote a Python script (enclosed) to methodically test how these things work, that doesn't rely on peculiarities of sys.executable. The tests did reveal some notable differences on *nix and 'doze:

• When shell=False on windows you must launch the process using a full filename (e.g. "foo.exe", not just "foo", pass --invoke-filename to the script to enable that). This may seem obvious to you, but for me it was surprising that one executable lookup function (looking in PATH) is in effect but not the other (extending unqualified executable names). This should be spelled out in the docs.

• On *nix, with shell=False and the executable is neither in the PATH in the environment at the time of Python's launch nor in os.environ at the time of Popen, passing Popen an explicit env whose PATH includes the executable is enough to cause it to be found. Not so on 'doze.

• On 'doze, when the executable is in the PATH of os.environ but not in that of Popen's explicit env argument, even with shell=False, no Exception is raised (but returncode is nonzero)

[bitdancer]

Well, it seems I was mistaken when I thought I knew how this worked :)
Checking the os.exec documentation linked from the subprocess page, I see that when an environment is supplied PATH is indeed checked in it. The documentation for CreateProcess, however, indicates that PATH is ignored, and any extension must be supplied explicitly.

At the very least the docs should be updated to clarify that execvpe is used when an environment is passed on posix, and to link to the CreateProcess docs. A discussion of PATH could perhaps be put in a note or footnote (probably footnote, there are enough notes already in those docs!)

I'm not sure how one creates a good portability story out of these pieces. It doesn't seem as though there is any way to harmonize the two, since we are dealing with the semantics of system calls over which we have no control.

For reference, here is (a?) link to CreateProcess docs that I found via Google:

http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx

It doesn't look like the kind of link that one could trust to be stable, though, so I'm not sure if we should include it in the docs.

I'm adding Brian Curtin as nosy to see if he knows whether or not there are permalink-like links to the relevant Microsoft documentation that we could use.

[briancurtin]

You could take the "(VS8.5)" part out of the link which will give the latest version, which may not always be the relevant version (although I doubt this specific API would change).

That's about the best permalink-like feature you'll find, but overall, leaving the link as-is is pretty safe in my experience.

[dabrahams]

@r.david.murray: did you try running my test? I think it shows that we are pretty darned close to fully portable. I believe we could fix Popen to make it fully portable pretty easily. In fact, there may be a pure-python fix. Documenting the differences would also not be hard. I would discourage you from relying *solely* on a description such as "uses execvpe on POSIX" to describe the semantics. Aside from being a nonportable description, it doesn't help anyone who isn't familiar with the POSIX system calls.

[bitdancer]

I didn't run the script. I have now, but I'm not clear from its output what each test is actually doing, and don't really have the time to figure it out from the code right now.

I think it is probably more efficient to just ask you what your suggestion is for making things more portable?

As for the docs, the docs link to the os.exec python docs, which explain the PATH semantics. Linking to the Microsoft documentation would equivalently explain the Windows semantics. An explicit footnote discussing the differences in PATH behavior in the subprocess context would probably be helpful.

[dabrahams]

I've uploaded a new probe.py that contains a win32 Popen wrapper that I think acts just like *nix's Popen w.r.t. PATH and environment (pass --fix to demonstrate). I suggest using this or an equivalent wrapper for Win32, and documenting the fact that with shell=False, filename extensions need to be supplied explicitly on windows.