Process.status() returns 'running' for a ZombieProcess

[the-gigi]

This is on a Mac OSX El Capitan Here is a quick session to demonstrate. Process 70 on my machine right now is indeed a zombie process.

Python 2.7.10 (default, Jun 10 2015, 19:43:32)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.__version__
'4.0.0'
>>> p = psutil.Process(pid=70)
>>> p.status()
'running'
>>> p.name()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/gigi/.virtualenvs/work/lib/python2.7/site-packages/psutil/__init__.py", line 564, in name
    cmdline = self.cmdline()
  File "/Users/gigi/.virtualenvs/work/lib/python2.7/site-packages/psutil/__init__.py", line 617, in cmdline
    return self._proc.cmdline()
  File "/Users/gigi/.virtualenvs/work/lib/python2.7/site-packages/psutil/_psosx.py", line 207, in wrapper
    raise ZombieProcess(self.pid, self._name, self._ppid)
psutil.ZombieProcess: psutil.ZombieProcess process still exists but it's a zombie (pid=70)

I followed the code to the status() method in _psxosx.py to see why it returns 'running'

    @wrap_exceptions
    def status(self):
        code = cext.proc_status(self.pid)
        # XXX is '?' legit? (we're not supposed to return it anyway)
        return PROC_STATUSES.get(code, '?')

The cext.proc_status(self.pid) returns 2, which is the code for 'running'.

[giampaolo]

Mmmm if that is true it's the kernel which is returning the bad info.
There's a test for this:

psutil/psutil/tests/test_process.py

Line 1287 in 5c772a1

@unittest.skipUnless(POSIX, 'posix only')

....which explicitly tests the case assert p.status() == ZOMBIE:

psutil/psutil/tests/test_process.py

Line 1325 in 5c772a1

call_until(lambda: zproc.status(),

Can you run tests (make test) and see if that fails?

[giampaolo]

Bump?

[zachriggle]

Many tests currently fail immediately OS X 10.11 El Capitain.

I'd recommend adding a Travis CI target for OSX.

$ uname -a
Darwin riggle-macbookpro2 15.4.0 Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64
$ make test
...
======================================================================
FAIL: test__all__ (test_misc.TestMisc)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_misc.py", line 156, in test__all__
    self.fail('%r not in psutil.__all__' % name)
AssertionError: 'x' not in psutil.__all__

======================================================================
FAIL: test_vmem_free (test_osx.TestSystemAPIs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
AssertionError: 68771840L != 27615232 within 512000 delta

======================================================================
FAIL: test_fetch_all (test_process.TestFetchAllProcesses)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1594, in test_fetch_all
    self.fail(''.join(failures))
AssertionError:
    ======================================================================
    FAIL: test_exe (proc=psutil.Process(pid=82273, name='cupsd'), ret='/usr/sbin/cupsd'))
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1579, in test_fetch_all
        meth(ret, p)
      File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1614, in exe
        self.assertTrue(os.access(ret, os.X_OK))
      File "/Users/riggle/.pyenv/versions/2.7.9/lib/python2.7/unittest/case.py", line 422, in assertTrue
        raise self.failureException(msg)
    AssertionError: False is not true


======================================================================
FAIL: test_proc_cwd (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 2021, in test_proc_cwd
    self.assertEqual(encode_path(p.cwd()), funny_directory)
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmptYaTwJ/%C0%80' != '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmptYaTwJ/\xc0\x80'

======================================================================
FAIL: test_proc_exe (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1983, in test_proc_exe
    self.assertEqual(encode_path(os.path.basename(p.exe())), b"\xc0\x80")
AssertionError: '%C0%80' != '\xc0\x80'

======================================================================
FAIL: test_proc_open_files (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 2036, in test_proc_open_files
    self.assertIn(funny_file, encode_path(path))
AssertionError: '/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmptYaTwJ/\xc0\x80' not found in '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmptYaTwJ/%C0%80'

======================================================================
FAIL: test_prog_w_funky_name (test_process.TestProcess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 744, in test_prog_w_funky_name
    os.path.normcase(funky_path))
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu1GIghwfoo bar )' != '/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu1GIghwfoo bar )'

======================================================================
FAIL: test_proc_cwd (test_process.TestUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1905, in test_proc_cwd
    self.assertEqual(p.cwd(), tdir)
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-e\xcc\x80-RaOftt' != '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-\xc3\xa8-RaOftt'

======================================================================
FAIL: test_proc_open_files (test_process.TestUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1919, in test_proc_open_files
    self.assertEqual(os.path.normcase(path), os.path.normcase(self.uexe))
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psuExQYoPe\xcc\x80' != '/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psuExQYoP\xc3\xa8'

----------------------------------------------------------------------
Ran 359 tests in 4.451s

FAILED (failures=9, skipped=195)
make: *** [test] Error 1

[giampaolo]

There's already a Travis CI although I'm not sure what OSX version is running.
As for the issue at hand, your test output tells status() correctly signals a zombie process so I'm not sure how to proceed (and I don't have an OSX El Captain to tests against unfortunately).

[zachriggle]

Unfortunately, the issue is valid.

>>> import psutil
>>> psutil.__version__
'4.2.0'
>>> init = psutil.Process(1)
>>> init
<psutil.Process(pid=1, name='launchd') at 4422598736>
>>> init.status()
'running'
>>> init.is_running()
True
>>> init.cmdline()
---------------------------------------------------------------------------
ZombieProcess                             Traceback (most recent call last)
<ipython-input-10-b51ce1ac1fc9> in <module>()
----> 1 init.cmdline()

/Users/riggle/psutil/psutil/__init__.pyc in cmdline(self)
    615     def cmdline(self):
    616         """The command line this process has been called with."""
--> 617         return self._proc.cmdline()
    618
    619     def status(self):

/Users/riggle/psutil/psutil/_psosx.pyc in wrapper(self, *args, **kwargs)
    212                     raise NoSuchProcess(self.pid, self._name)
    213                 else:
--> 214                     raise ZombieProcess(self.pid, self._name, self._ppid)
    215             if err.errno in (errno.EPERM, errno.EACCES):
    216                 raise AccessDenied(self.pid, self._name)

ZombieProcess: psutil.ZombieProcess process still exists but it's a zombie (pid=1, name='launchd')
>>> %debug
> /Users/riggle/psutil/psutil/_psosx.py(214)wrapper()
    213                 else:
--> 214                     raise ZombieProcess(self.pid, self._name, self._ppid)
    215             if err.errno in (errno.EPERM, errno.EACCES):

ipdb> list
    209         except OSError as err:
    210             if err.errno == errno.ESRCH:
    211                 if not pid_exists(self.pid):
    212                     raise NoSuchProcess(self.pid, self._name)
    213                 else:
--> 214                     raise ZombieProcess(self.pid, self._name, self._ppid)
    215             if err.errno in (errno.EPERM, errno.EACCES):
    216                 raise AccessDenied(self.pid, self._name)
    217             raise
    218     return wrapper
    219
ipdb> p err
OSError(3, 'No such process')

[giampaolo]

This was Python 2 I suppose? Can you try Python 3? I wonder whether those apparently unicode related issues are 2.X only.

[zachriggle]

Nope, the test failures are still present during make test, and the Zombie process issue also repros.

$ python --version
Python 3.5.1
$ python -c 'import psutil; print(psutil.__version__)'
4.2.0
$ uname -a
Darwin riggle-macbookpro2 15.4.0 Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64
$ make test
======================================================================
ERROR: test_halfway_terminated_process (test_process.TestProcess)
----------------------------------------------------------------------
RuntimeError: proc_pidinfo(PROC_PIDLISTFDS) failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1277, in test_halfway_terminated_process
    ret = meth()
  File "/Users/riggle/psutil/psutil/__init__.py", line 1081, in open_files
    return self._proc.open_files()
  File "/Users/riggle/psutil/psutil/_psosx.py", line 208, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/riggle/psutil/psutil/_psosx.py", line 313, in open_files
    rawlist = cext.proc_open_files(self.pid)
SystemError: <class 'OSError'> returned a result with an error set

======================================================================
ERROR: test_zombie_process (test_process.TestProcess)
----------------------------------------------------------------------
RuntimeError: proc_pidinfo(PROC_PIDLISTFDS) failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1342, in test_zombie_process
    zproc.as_dict()
  File "/Users/riggle/psutil/psutil/__init__.py", line 473, in as_dict
    ret = attr()
  File "/Users/riggle/psutil/psutil/__init__.py", line 1081, in open_files
    return self._proc.open_files()
  File "/Users/riggle/psutil/psutil/_psosx.py", line 208, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/riggle/psutil/psutil/_psosx.py", line 313, in open_files
    rawlist = cext.proc_open_files(self.pid)
SystemError: <class 'OSError'> returned a result with an error set

======================================================================
FAIL: test_vmem_free (test_osx.TestSystemAPIs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
  File "/Users/riggle/psutil/psutil/tests/__init__.py", line 491, in wrapper
    return fun(*args, **kwargs)
  File "/Users/riggle/psutil/psutil/tests/test_osx.py", line 156, in test_vmem_free
    delta=MEMORY_TOLERANCE)
AssertionError: 3845955584 != 1574113280 within 512000 delta

======================================================================
FAIL: test_fetch_all (test_process.TestFetchAllProcesses)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1594, in test_fetch_all
    self.fail(''.join(failures))
AssertionError:
    ======================================================================
    FAIL: test_open_files (proc=psutil.Process(pid=1, name='launchd'))
    ----------------------------------------------------------------------
    RuntimeError: proc_pidinfo(PROC_PIDLISTFDS) failed

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1559, in test_fetch_all
        ret = attr(*args)
      File "/Users/riggle/psutil/psutil/__init__.py", line 1081, in open_files
        return self._proc.open_files()
      File "/Users/riggle/psutil/psutil/_psosx.py", line 208, in wrapper
        return fun(self, *args, **kwargs)
      File "/Users/riggle/psutil/psutil/_psosx.py", line 313, in open_files
        rawlist = cext.proc_open_files(self.pid)
    SystemError: <class 'OSError'> returned a result with an error set


======================================================================
FAIL: test_proc_cwd (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 2021, in test_proc_cwd
    self.assertEqual(encode_path(p.cwd()), funny_directory)
AssertionError: b'/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmp5gyeiu5j/%C0%80' != b'/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmp5gyeiu5j/\xc0\x80'

======================================================================
FAIL: test_proc_exe (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1983, in test_proc_exe
    self.assertEqual(encode_path(os.path.basename(p.exe())), b"\xc0\x80")
AssertionError: b'%C0%80' != b'\xc0\x80'

======================================================================
FAIL: test_proc_open_files (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 2036, in test_proc_open_files
    self.assertIn(funny_file, encode_path(path))
AssertionError: b'/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmp5gyeiu5j/\xc0\x80' not found in b'/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmp5gyeiu5j/%C0%80'

======================================================================
FAIL: test_prog_w_funky_name (test_process.TestProcess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 744, in test_prog_w_funky_name
    os.path.normcase(funky_path))
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu8reo_y0wfoo bar )' != '/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu8reo_y0wfoo bar )'
- /private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu8reo_y0wfoo bar )
? --------
+ /var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu8reo_y0wfoo bar )


======================================================================
FAIL: test_proc_cwd (test_process.TestUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1905, in test_proc_cwd
    self.assertEqual(p.cwd(), tdir)
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-è-x40mmfk_' != '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-è-x40mmfk_'
- /private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-è-x40mmfk_
?                                                                 ^^
+ /private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-è-x40mmfk_
?                                                                 ^


======================================================================
FAIL: test_proc_open_files (test_process.TestUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1919, in test_proc_open_files
    self.assertEqual(os.path.normcase(path), os.path.normcase(self.uexe))
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psux5cme4_xè' != '/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psux5cme4_xè'
- /private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psux5cme4_xè
? --------                                                            ^^
+ /var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psux5cme4_xè
?                                                             ^


----------------------------------------------------------------------
Ran 359 tests in 3.912s

FAILED (failures=8, errors=2, skipped=195)
...

[jomann09]

Hello Giampaolo, I am actually getting this error/issue with psutil 4.2.0 on Mac OS X 10.11 and running Python 2.7.10 which is the default for OS X 10.11. I have not had a chance to look at this closer but I will try to soon.