Repository navigation
Soft deprecate os.spawn*(), os.popen() and os.system() functions #120743
Description
See the discussion for the rationale: Is time to remove os module spawn* functions?
See also the discussion: How to deal with unsafe/broken os.spawn* arg handling behavior on Windows (Nov 2022).
Advantages of the subprocess module:
- subprocess restores the signal handlers (reliable behavior).
- subprocess closes all file descriptors (more secure).
- subprocess doesn't use a shell by default,
shell=Truemust be used explicitly. - subprocess is the defacto standard way to spawn processes in Python.
Examples of os functions issues:
- os.popen() and os.system() always use
shell=True, it cannot be disabled. (higher risk of shell code injection) - os.popen().close() return value depends on the platform. (not portable)
- os.popen().close() return value is a "wait status", not an exit code: waitstatus_to_exitcode() must be used to get a return code.
- There are 8 os.spawn*() functions, it's more error prone and harder to use than subprocess APIs.
- os.spawn*() functions return an exit code 1 if the program is not found instead of raising an exception. It's harder to distinguish if the program exists and has an exit code of 1, or if it doesn't exist.
- The os.spawn*() functions are not safe for use in multithreaded programs on POSIX systems as they use fork()+exec() from Python there.
Activity
vstinner commentedon Jun 19, 2024
Links:
pythongh-120743: Soft deprecate os.popen() function
vstinner commentedon Jun 19, 2024
Reminder: https://docs.python.org/dev/glossary.html#term-soft-deprecated
gpshead commentedon Jun 24, 2024
As a soft-deprecation with docs only, no warning raised, and no planned removal... I like this for all of the listed APIs. Including the widely used
os.system(). Our goal is to always point users towards the best way to do things.adorilson commentedon Jun 25, 2024
Could we do that as soon as possible?
For a pattern issue, the
deprecation noteis only at the section's end, right?However, the spawn* functions, for instance, have a long section (almost two pages), so I think this notice must be earlier.
We have a note suggesting subprocess use, but it doesn't mention the deprecation state.
So, what do you guys think about:
These functions are soft deprecated. See note at the bottom.in a highlighted way, if possible.gh-120743: Soft deprecate os.popen() function (#120744)
vstinner commentedon Jul 1, 2024
Implemented as the change d44c550.
vstinner commentedon Jul 1, 2024
I don't think that we have to be more aggressive about deprecating these functions. A soft deprecation mention at the end of the doc should be enough for now.
eryksun commentedon Jul 1, 2024
On Windows, bear in mind that sharing file descriptors with a child process is non-obvious if you're not using
os.spawn*(). You have to get the OS handle value viamscvrt.get_osfhandle(). Make it inheritable, at least temporarily. Spawn the child process viasubprocess.Popenwithclose_fds=False(or use theSTARTUPINFOhandle list). Communicate the handle value to the child (as opposed to the fd value). Then implement the child process to wrap the handle back as a new fd via C_open_osfhandle()(e.g.msvcrt.open_osfhandle()). Withos.spawn*(), if the child process uses CRT file descriptors, then one just has to make the file descriptor inheritable and communicate the fd value to the child (e.g. with a command-line argument or environment variable).vstinner commentedon Jul 1, 2024
On Windows, I prefer to pass handles to child processes :-) https://docs.python.org/dev/library/subprocess.html#subprocess.STARTUPINFO.lpAttributeList
eryksun commentedon Jul 1, 2024
If one's current workflow is based around file descriptors, then switching from
spawn*()tosubprocess.Popenentails adapting the code in both the Python script as well as every affected child script or application to use the CRT's_get_osfhandle()and_open_osfhandle()functions, such as viamsvcrt.get_osfhandle()andmsvcrt.open_osfhandle(). If you don't control the source code for the spawned processes, the latter is obviously impossible.Even from Python script to a child Python script, we have nothing implemented to make this workflow obvious and simple.
fd_listlist has to be constructed manually by duplicating each file descriptor viaos.dup(). Then build thehandle_listby callingmsvcrt.get_osfhandle()on each file descriptor infd_list, making it inheritable viaos.set_handle_inheritable(), and appending it tohandle_list. Next create a startup record viasubprocess.STARTUPINFO(lpAttributeList={"handle_list": handle_list})and spawn the child process with thisstartupinfoparameter, as well as with the handle values, and whether each should be opened in append mode, passed to the child in some way (e.g. command line, environment variable, pipe, shared memory, etc). Finally, close all of the duped file descriptors infd_list.handle_listin order to buildfd_list. Make each handle non-inheritable viaos.set_handle_inheritable()and, depending on whether it was opened in append mode, open a file descriptor viamsvcrt.open_osfhandle(h, os.O_NOINHERIT)ormsvcrt.open_osfhandle(h, os.O_APPEND | os.O_NOINHERIT), and append the opened fd tofd_list.pythongh-120743: Soft deprecate os.popen() function (python#120744)
pythongh-120743: Soft deprecate os.popen() function (python#120744)
pythongh-120743: Soft deprecate os.popen() function (python#120744)