How to use `subprocess` command with pipes

Question

I want to use subprocess.check_output() with ps -A | grep 'process_name'. I tried various solutions but so far nothing worked. Can someone guide me how to do it?

related: How do I use subprocess.Popen to connect multiple processes by pipes? — jfs
– jfs, Commented Jun 16, 2013 at 1:34
there is psutil that allows to get process info in a portable manner. — jfs
– jfs, Commented Jun 16, 2013 at 1:35

buhtz · Accepted Answer · 2024-05-31 10:49:57Z

641

To use a pipe with the subprocess module, you can pass shell=True but be aware of the Security Considerations. It is discouraged using shell=True. In most cases there are better solutions for the same problem.

However, this isn't really advisable for various reasons, not least of which is security. Instead, create the ps and grep processes separately, and pipe the output from one into the other, like so:

ps = subprocess.Popen(('ps', '-A'), stdout=subprocess.PIPE)
output = subprocess.check_output(('grep', 'process_name'), stdin=ps.stdout)
ps.wait()

In your particular case, however, the simple solution is to call subprocess.check_output(('ps', '-A')) and then str.find on the output.

edited May 31, 2024 at 10:49

buhtz

12.5k22 gold badges95 silver badges196 bronze badges

answered Nov 11, 2012 at 14:58

Taymon

25.8k9 gold badges65 silver badges85 bronze badges

Sign up to request clarification or add additional context in comments.

16 Comments

Nicolas Over a year ago

+1 for separating the output/input to avoid using shell=True

Serge Over a year ago

Don't forget, error subprocess.CalledProcessError: Command '('grep', 'process_name')' returned non-zero exit status 1 just means that nothing was found by grep, so it's normal behaviour.

Papouche Guinslyzinho Over a year ago

Why do we need the ps.wait() for when we already have the output. ps.wait.__doc__ waits for the child to terminate but the content of the child seems already placed into the output variable

Taymon Over a year ago

@MakisH You're looking at string.find, which has been deprecated in favor of str.find (i.e., the method find on str objects).

jfs Over a year ago

note: if grep dies prematurely; ps may hang indefinitely if it produces enough output to fill its OS pipe buffer (because you haven't called ps.stdout.close() in the parent). Swap the starting order, to avoid it

|

Guillaume Jacquenot · Accepted Answer · 2019-10-08 11:39:00Z

89

Or you can always use the communicate method on the subprocess objects.

cmd = "ps -A|grep 'process_name'"
ps = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
output = ps.communicate()[0]
print(output)

The communicate method returns a tuple of the standard output and the standard error.

edited Oct 8, 2019 at 11:39

Guillaume Jacquenot

11.8k6 gold badges45 silver badges50 bronze badges

answered Nov 11, 2012 at 16:35

jkalivas

1,1931 gold badge9 silver badges11 bronze badges

7 Comments

Paolo Over a year ago

I think using communicate is better than wait. There is such warning: "This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that."

EnemyBagJones Over a year ago

To clarify Paolo's comment above, the warning is for wait, not for communicate - i.e. it's the reason he says communicate is better.

Miguel Ortiz Over a year ago

The output of ps.communicate()[0] in python3 returns a bytes object.

tripleee Over a year ago

You are reinventing subprocess.check_output, not too poorly but unattractively. As the documentation suggests, you should avoid the low-level Popen when the library already provides higher-level functions which take care of all this plumbing in a single line of code, often with better behavior for boundary conditions.

Stephen Miller Over a year ago

@JvO. That makes very little sense in this context and will probably only confuse you and require that you actively separate stdout from stderr, which you may not even be able to do reliably. Why not trust that communicate will do what it is designed to do? It would be better to set both stdout and stderr to subprocess.PIPE. The communicate method will then return a tuple with stdout at index 0 and stderr at index 1. That is how it should be done.

|

Alan W. Smith · Accepted Answer · 2022-10-02 18:47:56Z

56

Using input from subprocess.run you can pass the output of one command into a second one.

import subprocess
    
ps = subprocess.run(['ps', '-A'], check=True, capture_output=True)
processNames = subprocess.run(['grep', 'process_name'],
                              input=ps.stdout, capture_output=True)
print(processNames.stdout.decode('utf-8').strip())

edited Oct 2, 2022 at 18:47

Alan W. Smith

25.7k5 gold badges73 silver badges102 bronze badges

answered Nov 12, 2020 at 0:50

anaken78

8346 silver badges8 bronze badges

9 Comments

MightyInSpirit Over a year ago

NOTE: capture_output will only work for Python 3.7.9 and above.

CervEd Over a year ago

what does check do and what's the purpose of capture_output?

tripleee Over a year ago

@CervEd Both of these are clearly documented. capture_output is a shorthand for the option combination stdout=supprocess.PIPE, stderr=subprocess.PIPE and check=True raises an error if the subprocess did not return a success (zero) status.

CervEd Over a year ago

@tripleee they are documented, somewhere in the unwieldy Python documentation, but there's no detail in the answer as to why they are included. check=True is for example not strictly necessary but capture_output=True is for the answer to work. The reason for using these options should be included as a part of the answer

Eldritch Cheese Over a year ago

A downside to this approach is that capture_output will read all of the process's stdout into memory. For small programs like ps, this may be fine, but for larger analysis pipelines this should be avoided.

|

AlcubierreDrive · Accepted Answer · 2013-06-16 01:12:21Z

32

See the documentation on setting up a pipeline using subprocess: http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline

I haven't tested the following code example but it should be roughly what you want:

query = "process_name"
ps_process = Popen(["ps", "-A"], stdout=PIPE)
grep_process = Popen(["grep", query], stdin=ps_process.stdout, stdout=PIPE)
ps_process.stdout.close()  # Allow ps_process to receive a SIGPIPE if grep_process exits.
output = grep_process.communicate()[0]

edited Jun 16, 2013 at 1:12

answered Jun 16, 2013 at 0:48

AlcubierreDrive

3,6642 gold badges33 silver badges45 bronze badges

2 Comments

Alvin Over a year ago

Upon checking this failed, see the answer below by Taymon for something that works without mucking around

RightmireM Over a year ago

subprocess.check_output doesn't appear to exist in Python 2.6.9

Shooe · Accepted Answer · 2012-11-13 10:34:47Z

3

Also, try to use 'pgrep' command instead of 'ps -A | grep 'process_name'

answered Nov 13, 2012 at 10:34

Shooe

752 bronze badges

1 Comment

Shooe Over a year ago

if you want get process id, obviously

cyraxjoe · Accepted Answer · 2022-03-23 19:50:17Z

3

You can try the pipe functionality in sh.py:

import sh
print sh.grep(sh.ps("-ax"), "process_name")

edited Mar 23, 2022 at 19:50

cyraxjoe

5,7514 gold badges32 silver badges43 bronze badges

answered Nov 12, 2012 at 5:54

amoffat

7245 silver badges14 bronze badges

4 Comments

tripleee Over a year ago

The link is dead.

cyraxjoe Over a year ago

Not anymore, link updated.

tripleee Over a year ago

It's dead again!

Mohammad Banisaeid Over a year ago

This is the link: github.com/amoffat/sh

Brent · Accepted Answer · 2020-08-14 15:14:57Z

1

command = "ps -A | grep 'process_name'"
output = subprocess.check_output(["bash", "-c", command])

answered Aug 14, 2020 at 15:14

Brent

4,3315 gold badges37 silver badges69 bronze badges

7 Comments

Charles Duffy Over a year ago

Why not use shell=True and let that prepend ['sh', '-c']? Nothing in this code requires bash. (That said, it's significantly better practice to avoid using a shell at all; this use case is a reasonable one, but as soon as arguments start to get parameterized -- like taking the process_name as a parameter -- security concerns come in).

Brent Over a year ago

It's useful in that you don't have to split the string, which gets complicated when you have quoted white space.

Charles Duffy Over a year ago

Huh? subprocess.check_output(command, shell=True) doesn't require you to split the string. Popen converts any string into a list containing only that string -- thus, [command] -- so with shell=True you get ['sh', '-c'] prepended to that list, so you end up with ['sh', '-c', command], exactly what your code does here except for the sh/bash difference.

Charles Duffy Over a year ago

...for that matter, if you did try to split the string into a list as well as using shell=True, only the first element of that list would be treated as code; you'd get something like ['sh', '-c', 'ps', '-A', '|', 'grep', 'process_name']. That's not a useful thing to do: when invoked that way, the shell runs ps with $0 being -A, $1 being |, etc... but since the command ps doesn't look at $0, $1, etc., all that extra content is simply ignored.

Charles Duffy Over a year ago

If you read Lib/subprocess.py, you'll see that there literally is no difference between subprocess.check_output(["sh", "-c", command]) and subprocess.check_output(command, shell=True). The code is clear and simple -- this is not a place where there can be a devil hiding in the details somewhere.

|

relent95 · Accepted Answer · 2024-07-26 02:42:48Z

1

I'm answering an old question, because nobody mentioned the shlex. In Unix, you can construct a command using pipes safely with the shlex.join(), like the following example.

import shlex
import subprocess

print(subprocess.check_output(
    shlex.join(['ps', '-ef']) + ' | ' +
    shlex.join(['grep', 'ps -ef']), # The arguments will be quoted.
    shell=True
).decode('utf-8'))

answered Jul 26, 2024 at 2:42

relent95

5,2513 gold badges21 silver badges26 bronze badges

Comments

Raúl Salinas-Monteagudo · Accepted Answer · 2023-05-12 05:14:17Z

I think that launching a shell just to enjoy the pipelining is not as elegant as it could be.

The following code uses native subprocess pipeline support and it works indeed.

You could easily modify it to add more than two processes to the pipeline.

#!/usr/bin/env python3

import subprocess


def ps_grep(pattern):
    # First command-line
    ps_command = ["ps", "-A"]

    # Second command-line
    grep_command = ["grep", pattern]

    # Launch first process
    ps_process = subprocess.Popen(ps_command, stdout=subprocess.PIPE)

    # Launch second process and connect it to the first one
    grep_process = subprocess.Popen(
        grep_command, stdin=ps_process.stdout, stdout=subprocess.PIPE
    )

    # Let stream flow between them
    output, _ = grep_process.communicate()

    return output.decode()


if __name__ == "__main__":
    print(ps_grep("python"))

Collectives™ on Stack Overflow

How to use `subprocess` command with pipes

9 Answers 9

16 Comments

7 Comments

9 Comments

2 Comments

1 Comment

4 Comments

7 Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

9 Answers 9

16 Comments

7 Comments

9 Comments

2 Comments

1 Comment

4 Comments

7 Comments

Comments

Comments

Linked

Related