subprocess.Popen() is sometimes slower in python3 under illumos

[AlexanderPyhalov]

BPO	37790
Nosy	@pitrou, @vstinner, @MojoVampire, @nickhendo
Files	1.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-04-24.00:28:51.566>
created_at = <Date 2019-08-07.20:36:00.343>
labels = ['library']
title = 'subprocess.Popen() is sometimes slower in python3 under illumos'
updated_at = <Date 2020-05-19.14:39:22.058>
user = 'https://bugs.python.org/AlexanderPyhalov'

bugs.python.org fields:

activity = <Date 2020-05-19.14:39:22.058>
actor = 'movement'
assignee = 'none'
closed = True
closed_date = <Date 2020-04-24.00:28:51.566>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2019-08-07.20:36:00.343>
creator = 'Alexander.Pyhalov'
dependencies = []
files = ['48549']
hgrepos = []
issue_num = 37790
keywords = []
message_count = 15.0
messages = ['349197', '349207', '349211', '349857', '349893', '349894', '349897', '350064', '351343', '351352', '367162', '367181', '367182', '367185', '369358']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'movement', 'Alexander.Pyhalov', 'josh.r', 'nickhendo']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue37790'
versions = ['Python 3.5']

[AlexanderPyhalov]

We've moved illumos-gate wsdiff python tool from Python 2 to Python 3.
The tool does the following - for each file from old and new proto area compares file attributes to find differences in binary otput (spawning elfdump, dump and other utilities).
Performance has degraded in two times when commands.getstatusoutput() was replaced by subprocess.getstatusoutput().
os.popen() now is Popen() wrapper, so it also has poor performance.
Even naive popen() implementation using os.system() and os.mkstemp() behaved much more efficiently (about two times faster e.g. 20 minuts vs 40 minutes for single tool pass).

[MojoVampire]

Have you tried switching to using Popen itself (or run, which keeps it to one layer of convenience wrapping)?

subprocess.getstatusoutput is three layers of wrapping (it calls check_output, which in turn calls run, which in turn calls Popen), and (unlike Python 2) has to decode all the output. run would avoid two layers of wrapping, can optionally receive the raw bytes instead of decoding to str, and avoids needing to wrap the whole thing in a shell (which system, older popen, and getstatusoutput all do).

Beyond that, it looks like when 3.8 releases, Popen should get *much* faster if the call meets certain conditions, see https://docs.python.org/3.8/whatsnew/3.8.html#optimizations for details. If you can make your use cases conform to those requirements (e.g. by using shutil.which to determine the absolute paths to your utilities instead of relying on PATH lookup), the speed up should eliminate (or more than eliminate) the performance regression you're seeing (per bpo-35537, on macOS, which got the least benefit, it was still a 1.7x improvement; on other OSes, the multiplier hits 61x or 106x).

[AlexanderPyhalov]

I've tried to rewrite subporcess.getstatusoutput() usage with subprocess.Popen() and switch to shell=False, it didn't help, so I doubti it getstatusoutput() overhead, it's Popen() issue.

[pitrou]

Can you post a simple script showcasing the performance degradation?

[AlexanderPyhalov]

I couldn't reproduce entire test, as wsdiff script is rather large, but here's the simplified version. If Popen is used more often, difference is much more significant.

# Using workaround
$ python3.5 ~/tmp/1.py 1
10.780487537384033
# Without workaround
$ python3.5 ~/tmp/1.py
13.347045660018921
# Using python2.7
$ python2.7 ~/tmp/1.py
9.83385586739

[AlexanderPyhalov]

Even if I use
import subprocess
process = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True,
universal_newlines=True)
return process.stdout.read()
difference in times are the same.

[pitrou]

I don't seen any significant difference here (Ubuntu 18.04):

$ time python2 sp.py 
10.3433089256

real 0m10,362s
user 0m6,565s
sys 0m4,372s

$ time python3 sp.py 
11.746907234191895

real 0m11,781s
user 0m7,356s
sys 0m5,239s

[vstinner]

This issue lacks a lot of information:

• What is your operating system (name and version)? On Linux, what is your Linux kernel version?
• Which Python version did you try?
• Which command are you running?
• Do you use a shell?
• Do you use bytes (default) or Unicode (universal_newlines=True or text=True)?
• Can you provide a minimum reproducer? I don't know how to use msg349894: cmd is not defined.

Attached 1.py uses 2 commands, appararently both use a shell:

• "find /usr/bin -type f 2>/dev/null"
• "objdump '%s'"

"objdump '%s'" is unsafe and can lead to shell injection: try to avoid the usage of a shell. Use subprocess.Popen directly, or an helper which doesn't use shell=True.

[nickhendo]

We have recently bumped into a similar problem. Using FreeBSD, subprocess calls were taking more than 10 times the usual time to execute after migrating to python3.6. After some digging, the default for 'close_fds' was changed to 'True'. On linux, this actually made things faster, but for unix, much slower. Passing 'close_fds=False' solved this for us.

[vstinner]

I create bpo-38061 "FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()".

[vstinner]

We've moved illumos-gate wsdiff python tool from Python 2 to Python 3.

I guess that you're talking about https://github.com/illumos/illumos-gate/blob/master/usr/src/tools/scripts/wsdiff.py

This project is part of illumos: "An open-source Unix operating system". It's a fork of Solaris.

Python does not officially support Solaris not Illumos. If someone provides a patch, we can review it. But I'm not aware of anyone working actively on supporting Illumos.

In the meanwhile, there are two workarounds:

• Use subprocess.Popen(close_fds=False): it should be safe since PEP-446 has been implemented in Python 3.4... except for C extension modules and C libraries which don't implement it.
• Reduce the maximum number of file descriptors.

On Linux and FreeBSD (if fdescfs kernel module is loaded and mounted), Python 3 iterates on /proc/self/fd/. On FreeBSD, the closefrom() syscall can be used: PR 19697. But I'm not aware of any more efficient solution for Illumos.

Since the latest comment of the original poster was last August, I close the issue.

[movement]

closefrom() is on both Solaris and illumos too - and might even have originated there as an API - so if that's the issue, it should be trivially fixable