os.cpu_count() returns wrong number of processors on system with > 64 logical processors

[robbuckley]

BPO	30581
Nosy	@pfmoore, @pitrou, @vstinner, @giampaolo, @tjguk, @zware, @zooba, @tzickel, @crwilcox, @robbuckley
PRs	bpo-30581: Windows: os.cpu_count() returns wrong number of processors #2934 bpo-30581: Backport to 3.6: Windows: os.cpu_count() returns wrong number of processors #3267
Files	ci.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-09-01.19:29:13.752>
created_at = <Date 2017-06-06.12:49:39.499>
labels = ['type-bug', '3.7', 'OS-windows']
title = 'os.cpu_count() returns wrong number of processors on system with > 64 logical processors'
updated_at = <Date 2017-12-01.09:45:38.001>
user = 'https://github.com/robbuckley'

bugs.python.org fields:

activity = <Date 2017-12-01.09:45:38.001>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2017-09-01.19:29:13.752>
closer = 'pitrou'
components = ['Windows']
creation = <Date 2017-06-06.12:49:39.499>
creator = 'robbuckley'
dependencies = []
files = ['46928']
hgrepos = []
issue_num = 30581
keywords = []
message_count = 18.0
messages = ['295254', '295255', '295258', '295365', '295377', '299462', '299476', '299477', '299480', '301017', '301018', '301033', '301146', '301147', '301150', '301151', '307374', '307376']
nosy_count = 10.0
nosy_names = ['paul.moore', 'pitrou', 'vstinner', 'giampaolo.rodola', 'tim.golden', 'zach.ware', 'steve.dower', 'tzickel', 'crwilcox', 'robbuckley']
pr_nums = ['2934', '3267']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30581'
versions = ['Python 3.6', 'Python 3.7']

[robbuckley]

os.cpu_count() seems to report incorrect values on windows systems with >64 logical processors

tried it on 2 similar systems, both running windows 7 / 10 with python 3.6.1 64bit (anaconda):

platform1 - 2x Xeon E5-2698v4. 20 cores/CPU = total 80 logical cpus with hyperthreading
platform2 - 2x Xeon E5-2697v3. 14 cores/CPU = total 56 logical cpus with hyperthreading

os.cpu_count() reports 40 cores on platform1 and 56 on platform2

I would expect 80 and 56 respectively.

I suppose this is because the windows api call used is not aware of processor groups, and reports only the number of processors in the current processor group ( eg GetSystemInfo vs GetMaximumProcessorCount )

[vstinner]

On Windows, os.cpu_count() is currently implemented with:

"GetSystemInfo(&sysinfo); return sysinfo.dwNumberOfProcessors;"

https://msdn.microsoft.com/en-us/library/windows/desktop/ms724958(v=vs.85).aspx

It seems to return the number of *logical* CPUs:

"""
dwNumberOfProcessors

The number of logical processors in the current group.

Note: For information about the physical processors shared by logical processors, call GetLogicalProcessorInformationEx with the RelationshipType parameter set to RelationProcessorPackage (3).
"""

It seems like you have two physical CPU packages. Maybe the function only returns infos from the first package?

[robbuckley]

yes, i believe its reporting the number of processors in the current group only, not across all groups.

attached output of windows sysinternals/coreinfo showing 2 processor groups

see giampaolo/psutil#771 for some further disucssion of this topic

the maintainer of psutil asked me to raise this bug, also had a quick check on #python IRC. Its my first bug on bugs.python.org so if you need more info just let me know

[crwilcox]

I am going to work on this if no one else has started.

[giampaolo]

Nobody has AFAIK.

[giampaolo]

MS documentation is not clear on what function should be used as there are many returning different values. Here it is being suggested to use GetLogicalProcessorInformationEx:
https://stackoverflow.com/questions/31209256/reliable-way-to-programmatically-get-the-number-of-cores-on-windows

[crwilcox]

I agree that the MS Docs for this are a bit confusing. I ended up reaching out to the guy who authored the GetMaximumProcessorCount function. I had also written an implementation that iterated over GetProcessorInformationEx and he advised against it.

One of the things that makes this interesting is that in 32 bit processes (wow64) your processor is simulated to fit in the confines of that old system. This method will only report 32 cores under 32 bit as that is all the program can access in 32 bit mode.

[giampaolo]

About GetMaximumProcessorCount, MS doc states that it returns the "maximum number of logical processors that a processor group or the system can have", so maybe it also includes "empty" CPU sockets.

GetActiveProcessorCount, on the other hand, returns "the number of active processors in a processor group or in the system", which adds even more confusion.

[crwilcox]

I was reviewing the docs for the os module and cpu_count should always return the number of cpus on the system, not the usable CPUs. GetMaximumProcessorCount returns a simulated count in WoW64. I have reached back out to the Windows API dev and will see if GetLogicalProcessorInformationEx will allow us to do this. He had thought that my solution that way had other limitations under WoW64.

[pitrou]

New changeset c67bae0 by Antoine Pitrou (Christopher Wilcox) in branch 'master':
bpo-30581: Windows: os.cpu_count() returns wrong number of processors (bpo-2934)
c67bae0

[pitrou]

Fixed. Someone might backport this to 3.6 if they want.