cpu-affinity test fail on Archlinux

[tijko]

Running Archlinux kernel 4.8.13-1 with the latest versions of Python 3.6.0 and psutil 5.1.0:

python test_linux.py
......

======================================================================
FAIL: test_linux.TestProcessAgainstStatus.test_cpu_affinity
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tijko/psutil/psutil/tests/test_linux.py", line 1278, in test_cpu_affinity
    self.proc.cpu_affinity(), list(range(min_, max_ + 1)))
AssertionError: Lists differ: [0, 1, 2, 3] != [0, 1, 2, 3, 4, 5, 6, 7]

Second list contains 4 additional elements.
First extra element 4:
4

- [0, 1, 2, 3]
+ [0, 1, 2, 3, 4, 5, 6, 7]

The test value for cpus_allowed is being read from /proc/$/status and where as I have the kernel configured with CONFIG_HOTPLUG_CPU=y set.

From cpuset.h:

If !CONFIG_HOTPLUG_CPU, present == possible, and active == online.

The cpu_possible_mask is fixed at boot time, as the set of CPU id's
that it is possible might ever be plugged in at anytime during the
life of that system boot. The cpu_present_mask is dynamic(*),
representing which CPUs are currently plugged in. And
cpu_online_mask is the dynamic subset of cpu_present_mask,
indicating those CPUs available for scheduling.

If HOTPLUG is enabled, then cpu_possible_mask is forced to have
all NR_CPUS bits set, otherwise it is just the set of CPUs that
ACPI reports present at boot.

It would seem that if cpu-hotplugging is enabled this value is the mask with all the bits flipped by default at boot-time, even if they're non-existent.

FWIW, if you call psutil.cpu_affinity(cpus_allowed) where cpus_allowed is a list of cpus from status. After this call, the correct value is listed in status, where it goes from say [0-7] to [0-3].

[giampaolo]

Mmm..... I'm not sure what you're suggesting as a fix.

[tijko]

I pushed a small change that attempts to set the affinity to the value listed under /proc/$/status. Even if the range is larger than the actual number of allowable mask is updated with the correct Cpus_allowed.

[giampaolo]

Can you show me what happens if you try to set a non eligible CPU?
Can you please try this:

 import psutil
 p = psutil.Process()
 p.cpu_affinity([6])

...and this:

import psutil, os
psutil._psplatform.cext.proc_cpu_affinity_set(os.getpid(), [6])

...and show me the exception?

[tijko]

Digging through the kernels source, the cpumask will accept a new mask so long as there is an intersection with the present cpus. So setting affinity to just one invalid will throw the OSError (EINVAL from sched_setaffinity):

>>> psutil.Process().cpu_affinity([6])
Traceback (most recent call last):
  File "/home/tijko/psutil/psutil/_pslinux.py", line 1527, in cpu_affinity_set
    cext.proc_cpu_affinity_set(self.pid, cpus)
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tijko/psutil/psutil/__init__.py", line 838, in cpu_affinity
    self._proc.cpu_affinity_set(list(set(cpus)))
  File "/home/tijko/psutil/psutil/_pslinux.py", line 1144, in wrapper
    return fun(self, *args, **kwargs)
  File "/home/tijko/psutil/psutil/_pslinux.py", line 1535, in cpu_affinity_set
    cpu, allcpus))
ValueError: invalid CPU number 6; choose between (0, 1, 2, 3)

and:

>>> psutil._psplatform.cext.proc_cpu_affinity_set(os.getpid(), [6])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument

Where as:

>>> psutil.Process().cpu_affinity([0, 6])
>>> psutil._psplatform.cext.proc_cpu_affinity_set(os.getpid(), [0, 6])

Both are fine.

My suggestion was to parse out /proc/$$/status twice. Once to attempt to set the Cpus-allowed from the first read. After the status is set with a value le the number of cpus that was actually allowable (e.g. the first read had the default of [0-7] set and after attempting to set this range you get [0-3] on the second)

I don't think this is great but it was the closest to how you currently have the test. The other option is you could parse out /sys/device/system/cpu and the online/ present files.

[giampaolo]

Considering that CPU affinity on Linux should be determined at runtime I would say this is not an issue with the test per se but something which requires a brand new functionality.
I'm going to change cpu_affinity so that if it gets passed an empty list as in cpu_affinity([]) the affinity will get set to all eligible CPUs.
On systems != Linux the eligibility will be calculated as list(range(psutil.cpu_count())).