Running python test suite fails on macOS 10.14.4 with resource.RLIMIT_STACK error

[ned-deily]

BPO	36432
Nosy	@ronaldoussoren, @ned-deily, @methane, @berkerpeksag, @yan12125, @remilapeyre, @dimpase
Superseder	bpo-34602: python3 resource.setrlimit strange behaviour under macOS

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2019-04-29.19:53:25.019>
created_at = <Date 2019-03-26.07:09:11.590>
labels = ['OS-mac', '3.8', 'type-bug', '3.7']
title = 'Running python test suite fails on macOS 10.14.4 with resource.RLIMIT_STACK error'
updated_at = <Date 2019-07-02.07:57:03.169>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2019-07-02.07:57:03.169>
actor = 'ned.deily'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2019-04-29.19:53:25.019>
closer = 'ned.deily'
components = ['macOS']
creation = <Date 2019-03-26.07:09:11.590>
creator = 'ned.deily'
dependencies = []
files = []
hgrepos = []
issue_num = 36432
keywords = []
message_count = 9.0
messages = ['338858', '338862', '338865', '338867', '338941', '339307', '340369', '341116', '347120']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'methane', 'berker.peksag', 'yan12125', 'remi.lapeyre', 'dimpase']
pr_nums = []
priority = 'critical'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '34602'
type = 'behavior'
url = 'https://bugs.python.org/issue36432'
versions = ['Python 3.7', 'Python 3.8']

[ned-deily]

After upgrading my first macOS system to the newly released macOS 10.14.4 update, attempts to run the Python test suite via regrtest fail:

$ /usr/local/bin/python3.7 -m test -uall -j3 -w
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/__main__.py", line 2, in <module>
    main()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/main.py", line 640, in main
    Regrtest().main(tests=tests, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/main.py", line 586, in main
    self._main(tests, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/main.py", line 607, in _main
    setup_tests(self.ns)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/setup.py", line 77, in setup_tests
    resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))
ValueError: current limit exceeds maximum limit

The error is during regrtest initialization when it is trying to increase the process stack size to avoid previously seen problems when running tests. This code has been unchanged for a long time.

Stepping through the code in the REPL on 10.14.4:
>>> import resource
>>> resource.getrlimit(resource.RLIMIT_STACK)
(8388608, 67104768)
>>> soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
>>> newsoft = min(hard, max(soft, 1024*2048))
>>> newsoft
8388608
>>> resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: current limit exceeds maximum limit

The same code run on a macOS system still running 10.14.3 gives the same values from getrlimit but does not fail when calling setrlimit.

I will investigate further tomorrow; if this is a general problem with macOS 10.14.4, we'll need to provide a workaround and possibly open an incident report with Apple.

[methane]

My mac is also 10.14.4. I don't have older macOS now. Could someone test this?

$ ulimit -Sa
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4864
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1418
virtual memory          (kbytes, -v) unlimited

$ ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) unlimited
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 65532
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2128
virtual memory          (kbytes, -v) unlimited

[kakshay21]

I'm on macOS 10.14.3 and running those commands gives me this ->

~ ulimit -Sa
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8192
-c: core file size (blocks) 0
-v: address space (kbytes) unlimited
-l: locked-in-memory size (kbytes) unlimited
-u: processes 709
-n: file descriptors 4864
➜ ~ ulimit -Ha
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 65532
-c: core file size (blocks) unlimited
-v: address space (kbytes) unlimited
-l: locked-in-memory size (kbytes) unlimited
-u: processes 1064
-n: file descriptors unlimited

[methane]

I created simple program calling setrlimit and it succeeds.
I confirmed setrlimit argument is exactly same.
It's very curious why same Python code fails...

== c code
#include <sys/resource.h>
#include <stdio.h>

int
main(int argc, char *argv[])
{
    struct rlimit rl;
    int err;

    err = getrlimit(RLIMIT_STACK, &rl);
    if (err < 0) {
        perror("getrlimit");
        return err;
    }

    printf("%d, soft=%llu, hard=%llu\n", RLIMIT_STACK, rl.rlim_cur, rl.rlim_max);

    err = setrlimit(RLIMIT_STACK, &rl);
    if (err < 0) {
        perror("setrlimit");
        return err;
    }

    return 0;
}

== Python code
import resource
soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
print("limits=", soft, hard)
resource.setrlimit(resource.RLIMIT_STACK, (soft, hard))

== fails
Traceback (most recent call last):
  File "x.py", line 4, in <module>
    resource.setrlimit(resource.RLIMIT_STACK, (soft, hard))
ValueError: current limit exceeds maximum limit

[methane]

I think this issue is duplicate of https://bugs.python.org/issue34602

[remilapeyre]

I confirm that reverting 335ab5b work as a workaround, and does not seem to create regressions.

[berkerpeksag]

I can also confirm that reverting 335ab5b fixes the problem.

[ned-deily]

Thanks, Inada-san, for the pointer. Closing this as a duplicate of bpo-34602

[ned-deily]

See updated bpo-34602 discussion for why reverting 335ab5b causes other problems and for a different fix for this issue.