[doc] subprocess security considerations needs a Windows-specific exception

The documentation at https://docs.python.org/3/library/subprocess.html#security-considerations says that "this implementation will never implicitly call a system shell".

While this is technically true, on Windows the underlying CreateProcess API may create a system shell, which then exposes arguments to shell parsing. This happens when passed a .bat or .cmd file.

PSRT review of the issue determined that we can't safely detect and handle this situation without causing new issues and making it more complex for users to work around when they want to intentionally launch a batch file without shell processing. For the two cases of untrusted input, an untrusted application/argv[0] is already vulnerable, and an untrusted argument/argv[1:] is safe provided argv[0] is controlled. However, we do need to inform developers of the inconsistency so they can check their own use.

We'll use this issue to ensure we get good wording. First proposal in the next comment.

Thanks to RyotaK for reporting responsibly to the Python Security Response Team.

Linked PRs

Starting proposal:

On Windows, batch files (*.bat or *.cmd) may be launched by the operating system in a system shell. This could result in arguments being parsed according to shell rules, but without any escaping added by Python. If you are intentionally launching a batch file with arguments from untrusted sources, consider passing shell=True to allow Python to escape special characters.

Is the situation really as vague as 'may be' (unless forced by shell=True? Is it repeatable (deterministic) for the same call?

@zooba

@zooba Thanks for opening the issue!

Regarding the wording, using shell=True doesn't escape arguments properly.

subprocess.run([".\\test.bat", "&calc.exe"], shell=True)

The above snippet will spawn the following process, which still executes calc.exe:

C:\WINDOWS\system32\cmd.exe /c ".\test.bat &calc.exe"

@terryjreedy It's always launched using cmd.exe because batch files can't be parsed without it

Is the situation really as vague as 'may be' (unless forced by shell=True? Is it repeatable (deterministic) for the same call?

If we assume that the file system can change at any time, then it's not deterministic, even for the same call. But assuming nobody modifies the file you are executing, it should always use the same mechanism.

I do know more details about when/why it might vary, but we don't want to document them. There's also always the potential that Microsoft could change the behaviour and/or add an option to change the behaviour, and we don't want our docs to be invalidated by that. The main aim is to encourage users to pass shell=True if they know they're launching a batch file, and to be clear that we won't escape anything if they launch a batch file with shell=False.

Regarding the wording, using shell=True doesn't escape arguments properly.

I know we've discussed this at various points over the years... it ought to be a long-standing bug if it's not working right now. But we might need to make sure we're correctly handling command characters (and compatibly across platforms).

I do know more details about when/why it might vary, but we don't want to document them.

I guess I can add that the main issue is that a .bat or .cmd extension doesn't guarantee that it's actually a batch file, and that factors into how it may be run. If the extension is one of those and it's actually a batch file with no deliberate tricks, it'll always be run under cmd.exe.

Yep. IIRC Windows and related software has had a notorious number of security bugs related to unfortunate magic file type detection based on contents rather than what the name, extension, or mime type says... We can't prevent a fundamental platform issue on the Python side.

In defence of this case, by passing the path to CreateProcess you're kind of implying that you think it's an executable (akin to CPython's concrete C API). Do the same thing with other APIs and it'll use different tests (e.g. ShellExecute is more like our abstract C API).

But it's the fallback behaviour in this (and other) cases that cause problems. Which might also be why I'm instinctively against adding "abstract" behaviour to our concrete C APIs... 🤔

pythongh-114539: Clarify implicit launching of shells by subprocess (p…

[3.12] gh-114539: Clarify implicit launching of shells by subprocess (G…

Main change is committed to active branches - security fix branches will be merged by the RMs. There's no further need to keep this open.

If issues in shell quoting are discovered (they will always exist, FWIW), please open a new issue.

[3.10] gh-114539: Clarify implicit launching of shells by subprocess (G…

[3.9] gh-114539: Clarify implicit launching of shells by subprocess (G…

[3.8] gh-114539: Clarify implicit launching of shells by subprocess (G…

I think #117996 made the doc https://docs.python.org/3/library/subprocess.html self-contradictory.

First, regarding whether shell=True should be used to call a batch file.

At https://docs.python.org/3/library/subprocess.html#subprocess.Popen, it says shell=True should not be used:

You do not need shell=True to run a batch file or console-based executable.

But in https://docs.python.org/3/library/subprocess.html#security-considerations, it says shell=True should be used:

If you are intentionally launching a batch file with arguments from untrusted sources, consider passing shell=True to allow Python to escape special characters.

Second, regarding who should escape special characters.

In the first paragraph of https://docs.python.org/3/library/subprocess.html#security-considerations, it says the application should do it:

If the shell is invoked explicitly, via shell=True, it is the application’s responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid shell injection vulnerabilities.

But in the second paragraph, it says Python will do it:

If you are intentionally launching a batch file with arguments from untrusted sources, consider passing shell=True to allow Python to escape special characters.

In the source code, whether list2cmdline is invoked is determined by the type of args, not shell=True:

cpython/Lib/subprocess.py

Lines 1459 to 1471 in 81c739e

    
           if isinstance(args, str): 
        
               pass 
        
           elif isinstance(args, bytes): 
        
               if shell: 
        
                   raise TypeError('bytes args is not allowed on Windows') 
        
               args = list2cmdline([args]) 
        
           elif isinstance(args, os.PathLike): 
        
               if shell: 
        
                   raise TypeError('path-like args is not allowed when ' 
        
                                   'shell is true') 
        
               args = list2cmdline([args]) 
        
           else: 
        
               args = list2cmdline(args)

shell=True only quotes args with ":

cpython/Lib/subprocess.py

Line 1538 in 81c739e

args = '{} /c "{}"'.format (comspec, args)

@Ry0taK

@Ry0taK's comment reminds me of PowerShell/PowerShell#1995 (comment).

For example, I have a D:\test.cmd:

echo %*

Regardless of whether shell=True is used, calc.exe is launched:

import subprocess

subprocess.run([r'D:\test.cmd', 'test&calc.exe'])
subprocess.run([r'D:\test.cmd', 'test&calc.exe'], shell=True)

When calling CreateProcess:

cpython/Lib/subprocess.py

Line 1547 in 81c739e

hp, ht, pid, tid = _winapi.CreateProcess(executable, args,

Without shell=True, args is D:\test.cmd test&calc.exe
With shell=True, args is C:\Windows\system32\cmd.exe /c "D:\test.cmd test&calc.exe"

Another interesting discovery: even if args contains double quotes ", for example:

subprocess.run('echo "a"', shell=True)

cpython/Lib/subprocess.py

Line 1538 in 81c739e

args = '{} /c "{}"'.format (comspec, args)

will set args to

C:\Windows\system32\cmd.exe /c "echo "a""

This didn't corrupt the command, and it still prints

"a"

[3.11] gh-114539: Clarify implicit launching of shells by subprocess (G…

Sidenotes for a more ridiculous case (a byproduct when composing venv test cases):

Since & is among valid filename charset on Windows, in theory you can have executable files called &.bat and &.exe

import subprocess
out = subprocess.check_output(['&.exe', '-echoargs', '1']) # fine
print(out)
out = subprocess.check_output(['&.bat', '-echoargs', '1']) # will never succeed
print(out)
# workaround by constructing cmdline manually
subprocess.check_output('"&.bat"')

The funniest thing is that after they peek argv[0] and notice it's a Batch script, they implicitly change the cmdline mode which breaks the atomicity of argv[0] and contradicts the original peek.

Also regarding who should escape special characters: when using shell=True and passing arguments in a list, and there is a literal ampersand in one of the argument, there doesn't seem to be a way to call it properly, as you can't really "quote it yourself".

I.e.:

subprocess.run(['my.exe', 'amper&sand'], shell=False)  # works
subprocess.run('my.exe "amper&sand"', shell=False)     # works

subprocess.run(['my.exe', 'amper&sand'], shell=True)   # does not work; it will treat the ampersand as a command separator
subprocess.run(['my.exe', '"amper&sand"'], shell=True) # doesn't not work, the argument will have quotes around it
subprocess.run('my.exe "amper&sand"', shell=True)      # works

I don't quite understand how this happens.

My understanding is that if you use a list as args, all the elements will be properly quoted by Python, regardless it's shell=True or not. And that's indeed the case for things like white space (you don't need to quote it yourself even with shell=True).

But apparently Python suddenly stops quoting ampersand when shell=True?

(This is without involving any quirks due to batch file, just with normal sane programs.)

Navigation Menu

Repository navigation

[doc] subprocess security considerations needs a Windows-specific exception #114539

Linked PRs

8 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	if isinstance(args, str):
	pass
	elif isinstance(args, bytes):
	if shell:
	raise TypeError('bytes args is not allowed on Windows')
	args = list2cmdline([args])
	elif isinstance(args, os.PathLike):
	if shell:
	raise TypeError('path-like args is not allowed when '
	'shell is true')
	args = list2cmdline([args])
	else:
	args = list2cmdline(args)

[doc] subprocess security considerations needs a Windows-specific exception #114539

Description

Linked PRs

Activity

zooba commented on Jan 24, 2024

terryjreedy commented on Jan 25, 2024

Ry0taK commented on Jan 25, 2024

zooba commented on Jan 25, 2024

zooba commented on Jan 25, 2024

gpshead commented on Jan 25, 2024

zooba commented on Jan 25, 2024

8 remaining items

zooba commented on Apr 17, 2024

jiasli commented on Aug 8, 2024

jiasli commented on Aug 8, 2024

jiasli commented on Aug 8, 2024

y5c4l3 commented on Sep 28, 2024

fireattack commented on Jul 25, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions