os.cpu_count() returns wrong number of processors on specific systems

[yanirh]

BPO	33166
Nosy	@pfmoore, @pitrou, @giampaolo, @tjguk, @zware, @eryksun, @zooba, @csabella, @miss-islington
PRs	bpo-33166: Change os.cpu_count to return active (real) processors #15949 [3.8] bpo-33166: Change os.cpu_count to return active (real) processors (GH-15949) #15979
Files	Screenshot.png

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-11.15:20:09.367>
created_at = <Date 2018-03-28.09:44:42.722>
labels = ['type-bug', '3.8', '3.9', 'OS-windows']
title = 'os.cpu_count() returns wrong number of processors on specific systems'
updated_at = <Date 2019-09-11.15:56:16.043>
user = 'https://bugs.python.org/yanirh'

bugs.python.org fields:

activity = <Date 2019-09-11.15:56:16.043>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-11.15:20:09.367>
closer = 'zach.ware'
components = ['Windows']
creation = <Date 2018-03-28.09:44:42.722>
creator = 'yanirh'
dependencies = []
files = ['47504']
hgrepos = []
issue_num = 33166
keywords = ['patch']
message_count = 20.0
messages = ['314580', '314581', '314583', '314585', '314587', '314589', '314590', '314591', '314592', '314596', '314597', '314599', '314609', '314620', '314631', '314664', '334274', '351931', '351936', '351956']
nosy_count = 10.0
nosy_names = ['paul.moore', 'pitrou', 'giampaolo.rodola', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'cheryl.sabella', 'miss-islington', 'yanirh']
pr_nums = ['15949', '15979']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33166'
versions = ['Python 3.8', 'Python 3.9']

[yanirh]

wrong number of cpu's is reported on some specific platforms.

---

first platform:
server with X4 Intel® Xeon® E5-4620 (8 physical, 16 logical), running
a 64bit Windows Server 2012 R2 Standard.
results:
os.cpu_count() reports 64 units
psutil.cpu_count(logical=False) reports 32 units
psutil.cpu_count(logical=True) reports 64 units

multiprocessing using concurrent.futures able to fully utilize the server;

---

second platform:
server with X2 Intel® Xeon® Gold 6138 (20 physical, 40 logical), running a 64bit Windows Server 2016 Standard.
results:
os.cpu_count() reports 128 units
psutil.cpu_count(logical=False) reports 20 units
psutil.cpu_count(logical=True) reports 40 units

multiprocessing using concurrent.futures able to utilize only 1/4 of the server's power;

[yanirh]

wrong number of cpu's is reported on some specific platforms.

---

first platform:
server with X4 Intel Xeon E5-4620 (8 physical, 16 logical), running
a 64bit Windows Server 2012 R2 Standard.
results:
os.cpu_count() reports 64 units
psutil.cpu_count(logical=False) reports 32 units
psutil.cpu_count(logical=True) reports 64 units

multiprocessing using concurrent.futures able to fully utilize the server;

---

second platform:
server with X2 Intel Xeon Gold 6138 (20 physical, 40 logical), running a 64bit Windows Server 2016 Standard.
results:
os.cpu_count() reports 128 units
psutil.cpu_count(logical=False) reports 20 units
psutil.cpu_count(logical=True) reports 40 units

multiprocessing using concurrent.futures able to utilize only 1/4 of the server's power;

[pitrou]

os.cpu_count() reports 128 units
psutil.cpu_count(logical=False) reports 20 units
psutil.cpu_count(logical=True) reports 40 units

You mean os.cpu_count() reports *more* CPUs than exist on the machine? How can that happen?

[yanirh]

Yup.
Attaching a screenshot.

[giampaolo]

The difference between os.cpu_count() and psutil.cpu_count() is because one uses GetMaximumProcessorCount() and the other dwNumberOfProcessors.

This is tracked as a bug in psutil bug tracker but it's not fixed yet:
giampaolo/psutil#771

As for Python this is where it was discussed and changed:
https://bugs.python.org/issue30581
c67bae0

In summary: psutil is wrong and you should rely on os.cpu_count().

[yanirh]

Maybe i'm missing something, and would appreciate clarification.

Perhaps psutil is wrong, but it gives an answer that has something to do with the actual situation.

On platform 2, i have 2 Intel Xeon Gold 6138, each with 20 physical processors, 40 logicals.

you are saying i need to rely on os.cpu_count(), which outputs '128'. Can you elaborate on this?

Moreover, when attempting to parallelize on the processors, i reach 25% utilization, which suggests Python 'sees' only one processor group.

[giampaolo]

Oh! So both os.cpu_count() and psutil.cpu_count() are wrong? How do you determine the actual number of logical/physical CPUs on your machine?

[yanirh]

Yes. Both are wrong, and os.cpu_count() is completely off.
Regarding how to determine the number of physical/logical cores in my machine - well, not sure what you mean by that. I've attached a screenshot of Windows' system information. Also used 'cpu-z'.

It seems like aside from the os.cpu_count() issue, Python itself has some problem - it 'sees' only 1 CPU group. It is evident from the fact the when parallelizing, utilization level is only 25%.

[pitrou]

Let's not conflate different issues. The parallelization issue is distinct from the os.cpu_count() issue (and I'm skeptical Python is at fault there).

[yanirh]

Ok, no problem.
Just to be sure i'm doing the right thing - this thread will be dedicated to the os.cpu_count() issue, and i'll open a new issue on the parallelization problem.

makes sense?

[pitrou]

Just to be sure i'm doing the right thing - this thread will be dedicated to the os.cpu_count() issue, and i'll open a new issue on the parallelization problem.

Exactly.