gh-109595: Add -Xcpu_count=<n> cmdline for container users #109667

corona10 · 2023-09-21T16:45:41Z

Issue: Support python -Xcpu_count=<n> feature for container environment. #109595

📚 Documentation preview 📚: https://cpython-previews--109667.org.readthedocs.build/

corona10 · 2023-09-21T16:46:39Z

corona10 · 2023-09-21T16:59:19Z

Please read the issue of why this flag is needed and see also actual use-case from #109595 (comment)

Even if the container system is important these days, there is no way to limit CPU resources for the container environment from Python program side.
JDK provides -XX:ActiveProcessorCount=<n> from JDK 21 instead of supporting cgroup, this patch will provide identical feature to Python user and super useful :)

Doc/library/os.rst

Doc/using/cmdline.rst

Include/cpython/initconfig.h

Doc/using/cmdline.rst

Python/initconfig.c

Co-authored-by: Victor Stinner <[email protected]>

vstinner · 2023-09-21T17:51:20Z

Programs/_testembed.c


    putenv("PYTHONINTMAXSTRDIGITS=6666");
    config.int_max_str_digits = 31337;
+    config.cpu_count = -1;


Can you please set a more interesting value like 1234?

vstinner

Can you please document the change in Doc/whatsnew/3.13.rst?

vstinner · 2023-09-21T18:01:42Z

Doc/using/cmdline.rst

+   * :samp:`-X cpu_count={n}` will override the number of CPU count from system.
+     If this option is provided, :func:`os.cpu_count` will return the overrided value.
+     And *n* must be greater equal then 1.


Suggested change

* :samp:`-X cpu_count={n}` will override the number of CPU count from system.

If this option is provided, :func:`os.cpu_count` will return the overrided value.

And *n* must be greater equal then 1.

* :samp:`-X cpu_count={n}` overrides the number of CPU count from system:

:func:`os.cpu_count` returns *n*. The value *n* must be greater than

or equal to 1.

Doc/using/cmdline.rst

Misc/NEWS.d/next/Core and Builtins/2023-09-22-01-44-53.gh-issue-109595.fVINgD.rst

Python/initconfig.c

vstinner · 2023-09-21T18:05:52Z

Python/initconfig.c

+static PyStatus
+config_init_cpu_count(PyConfig *config)
+{
+    int cpu_count = -1;


You can move the variable declaration inside the "if (sep)" block. Does it have to be initialized?

If you are afraid of undefined behavior, maybe config_wstr_to_int() should set *result to 0 on error.

Now I added PYTHONCPUCOUNT envvar, so the current structure will be needed.

If you want, you can move the variable declaration inside each "if (env)" block and duplicate the variable, to better show its scope. But well, that's just a personal preference. Feel free to ignore my coding style remark ;-)

Python/initconfig.c

Co-authored-by: Victor Stinner <[email protected]>

Python/sysmodule.c

Adds PYTHON_CPU_COUNT help text. Mentions multiprocessing.cpu_count. rewords a few statements.

vstinner · 2023-10-09T20:35:54Z

The majority wants PYTHON_CPU_COUNT env name: https://discuss.python.org/t/change-environment-variable-style/35180

corona10 · 2023-10-09T23:25:37Z

@gpshead I will merge this PR by tomorrow. Please let me know if there needs more change. :)
cc @vstinner

vstinner

LGTM. I see that you addressed the two @gpshead's comments. You can merge.

vstinner · 2023-10-10T10:25:56Z

Congrats.

gpshead · 2023-10-10T11:13:19Z

🎉 thanks everybody!

corona10 · 2023-10-10T21:20:22Z

Thanks @vstinner and @gpshead for all your efforts to consider every use case!!

vstinner · 2023-10-10T21:36:11Z

Follow-up: issue #110649 to add -X cpu_count=process.

…hon#109667) --------- Co-authored-by: Victor Stinner <[email protected]> Co-authored-by: Gregory P. Smith [Google LLC] <[email protected]>

morotti · 2025-12-11T11:10:36Z

Hello,

Just a ping to say that the function doesn't work and the doc is wrong.
https://docs.python.org/3/library/os.html#os.cpu_count

Changed in version 3.13: If -X cpu_count is given or PYTHON_CPU_COUNT is set, cpu_count() returns the overridden value n.

>>> sys.version_info
sys.version_info(major=3, minor=14, micro=0, releaselevel='final', serial=0)
>>> os.cpu_count()
48
>>> os.process_cpu_count()
47
>>> os.environ["PYTHON_CPU_COUNT"] = "1"
>>> os.cpu_count()
48
>>> os.process_cpu_count()
47

expected: PYTHON_CPU_COUNT should change the return value of os.cpu_count on python 3.13+ as explained by the doc.
what happens: PYTHON_CPU_COUNT has no effect.

It seems the value is only read once at interpreter startup.

FYI: This is a problem because python cannot detect the number cores correctly. It's not aware of the number of cpu exposed to containers/VM and try to use all CPU from the host machine, it's not aware of the number of CPU it should use in compute clusters that expect 1 core per job.
This requires developers/users to implement their own code to detect cpu, and patch the python code in the sitecustomize/interpreter code to fix applications they can't modify directly.

I was looking forward to setting PYTHON_CPU_COUNT on interpreter startup to finally fix the issue in a simple consistent way, but it doesn't work ^^

vstinner · 2025-12-11T11:28:26Z

expected: PYTHON_CPU_COUNT should change the return value of os.cpu_count on python 3.13+ as explained by the doc.
what happens: PYTHON_CPU_COUNT has no effect.
It seems the value is only read once at interpreter startup.

You're correct: the PYTHON_CPU_COUNT environment variable is only read once at startup, not during execution.

If you really have to change os.cpu_count() return value at runtime, you can mock the function:

$ python
>>> import os
>>> os.cpu_count()
12
>>> os.cpu_count=lambda:1
>>> os.cpu_count()
1

But it's better to set the environment variable before Python startup.

morotti · 2025-12-11T12:24:11Z

But it's better to set the environment variable before Python startup.

I agree but it's not possible in practice.

In spark/slurm compute clusters, I could configure the clusters to add an environment variable with the number of CPU to be used by the job
In kubernetes, there is no way to automatically configure kubernetes to add an environment variable matching the number of cpu limit on the pod. (this would need some logic anyway because cpu limit can be fractional)
plenty of other environments where it's not possible to set an environment variable before the process starts

So the fix for me is to fix the CPU detection in the sitecustomize.py and configure environment variables for all the known frameworks: OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, POLARS_MAX_THREADS, PYTHON_CPU_COUNT etc...
I was hoping to not have to patch os.cpu_count() anymore, but I guess no choice :D

For reference, this is a pretty critical issue with the last generation of server CPUs released in 2024 which have way more cores (up to 192c/384t per socket). It only take a few misbehaved apps/libraries each trying to create N subthreads/subprocesses to accidentally kill the server and other jobs.

64x64 on an older server is merely 4096 threads/processes. 512x512 on a recent server is 262144 threads/processes.

vstinner · 2025-12-11T12:28:41Z

This closed issue is not the right place to ask questions on how to use Python. I suggest you finding another place to ask your question such as the Python Help category of discuss.python.org.

morotti · 2025-12-11T13:26:19Z

This thread is the right place to discuss cpu detection in the interpreter, as it was modified by this ticket and needs further improvements.

do you mind if I send a patch to the doc, to mention that the variable is only read once at startup?
do you mind if we correct the code to actually respect the variable at runtime?
https://github.com/python/cpython/blob/3.14/Lib/os.py#L1180
do you mind if we correct these functions to return the correct number of CPU, taking into account cgroup limits.
there was another thread asking about that few years ago, which was closed by you.
since your last comment, the problem has been standardized with cgroup v2, it's just one file to read with some numbers. happy to discuss with this thread or the other.
os.cpu_count() should return a count assigned to a container or that the process is restricted to #80235

vstinner · 2025-12-11T13:49:37Z

do you mind if I send a patch to the doc, to mention that the variable is only read once at startup?

You can propose a PR, I don't know if it would be accepted. @corona10 added this feature.

do you mind if we correct the code to actually respect the variable at runtime?

I don't think that it's a good idea. It's common in Python to only read an environment variable once at startup.

do you mind if we correct these functions to return the correct number of CPU, taking into account cgroup limits.

So far, the consensus is to not read cgroup limits since there is not reliable way to get a number of CPUs. Again, this closed issue is not the right place to discuss it.

corona10 · 2025-12-11T14:28:37Z

You can propose a PR for doc. But rest of things does not look like good idea.

gpshead · 2025-12-11T17:22:18Z

morotti please open a thread on discuss.python.org ideally which could ultimately spawn new issues. closed PRs are not a meaningful place for discussions.

pythongh-109595: Add -Xcpu_count=<n> cmdline for container users

41be170

corona10 requested a review from vstinner September 21, 2023 16:45

bedevere-app bot added the awaiting review label Sep 21, 2023

bedevere-app bot mentioned this pull request Sep 21, 2023

Support python -Xcpu_count=<n> feature for container environment. #109595

Closed

corona10 added 2 commits September 22, 2023 01:47

Check style

f7a7428

Add help

7009bbe

vstinner reviewed Sep 21, 2023

View reviewed changes

corona10 and others added 8 commits September 22, 2023 02:23

Apply suggestions from code review

d1f91d8

Co-authored-by: Victor Stinner <[email protected]>

Apply suggestions from code review

8c92ed6

Co-authored-by: Victor Stinner <[email protected]>

Check style

c27bdfc

Address code review

b13e5ee

Address code review

45fce16

Address code review

49a48e4

Address code review

829a8e8

Fix test

cfb33a4

corona10 requested a review from vstinner September 21, 2023 17:48

Update NEWS.d

95a2173

vstinner reviewed Sep 21, 2023

View reviewed changes

corona10 added 6 commits September 22, 2023 02:56

Address code review

0394d1d

Update

f4a3f01

Update

60a28fe

nit

7e5595c

Update

8678012

Add PYTHONCPUCOUNT

cbc5484

vstinner reviewed Sep 21, 2023

View reviewed changes

corona10 and others added 3 commits September 22, 2023 03:13

Add PYTHONCPUCOUNT

4622b60

Apply suggestions from code review

50f0178

Co-authored-by: Victor Stinner <[email protected]>

Update Python/initconfig.c

5c3ac68

Co-authored-by: Victor Stinner <[email protected]>

corona10 requested a review from gpshead October 3, 2023 02:12

corona10 commented Oct 3, 2023

View reviewed changes

Python/sysmodule.c Show resolved Hide resolved

corona10 added 2 commits October 3, 2023 11:23

Address Greg's code review

de8bf53

nit

633914b

gpshead added type-feature A feature request or enhancement docs Documentation in the Doc dir 3.13 bugs and security fixes labels Oct 3, 2023

gpshead and others added 3 commits October 3, 2023 04:57

Include multiprocessing in docs, reword.

37fbdfe

Adds PYTHON_CPU_COUNT help text. Mentions multiprocessing.cpu_count. rewords a few statements.

Merge remote-tracking branch 'upstream/main' into pythongh-109595

a0cfb21

Merge remote-tracking branch 'upstream/main' into pythongh-109595

8daa3b7

vstinner approved these changes Oct 10, 2023

View reviewed changes

corona10 merged commit 0362cbf into python:main Oct 10, 2023

corona10 deleted the gh-109595 branch October 10, 2023 10:00

bedevere-app bot removed the awaiting core review label Oct 10, 2023

python locked as resolved and limited conversation to collaborators Dec 11, 2025

Uh oh!

gh-109595: Add -Xcpu_count=<n> cmdline for container users #109667

gh-109595: Add -Xcpu_count=<n> cmdline for container users #109667

Uh oh!

Conversation

corona10 commented Sep 21, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corona10 commented Sep 21, 2023

Uh oh!

corona10 commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vstinner Sep 21, 2023

Choose a reason for hiding this comment

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner Sep 21, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vstinner Sep 21, 2023

Choose a reason for hiding this comment

Uh oh!

corona10 Sep 21, 2023

Choose a reason for hiding this comment

Uh oh!

vstinner Sep 21, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vstinner commented Oct 9, 2023

Uh oh!

corona10 commented Oct 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner commented Oct 10, 2023

Uh oh!

gpshead commented Oct 10, 2023

Uh oh!

corona10 commented Oct 10, 2023

Uh oh!

vstinner commented Oct 10, 2023

Uh oh!

morotti commented Dec 11, 2025

Uh oh!

vstinner commented Dec 11, 2025

Uh oh!

morotti commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Dec 11, 2025

Uh oh!

morotti commented Dec 11, 2025

Uh oh!

vstinner commented Dec 11, 2025

Uh oh!

corona10 commented Dec 11, 2025

Uh oh!

gpshead commented Dec 11, 2025

Uh oh!

Reviewers

corona10 commented Sep 21, 2023 •

edited by github-actions bot

Loading

corona10 commented Sep 21, 2023 •

edited

Loading

corona10 commented Oct 9, 2023 •

edited

Loading

morotti commented Dec 11, 2025 •

edited

Loading