sigaddset() can fail on some signal numbers

[hroncok]

BPO	33329
Nosy	@pitrou, @larryhastings, @ned-deily, @njsmith, @ambv, @hroncok, @zatrazz
PRs	bpo-33329: Fix multiprocessing regression on newer glibcs #6575 [3.7] bpo-33329: Fix multiprocessing regression on newer glibcs (GH-6575) #6579 [3.6] bpo-33329: Fix multiprocessing regression on newer glibcs (GH-6575) #6580 [3.6] bpo-33329: Fix multiprocessing regression on newer glibcs (GH-6575) #6582 [3.5] bpo-33329: Fix multiprocessing regression on newer glibcs (GH-6… #12144 [3.4] bpo-33329: Fix multiprocessing regression on newer glibcs (GH-6… #12145
Files	upgraded_packages.txt: List of upgraded system packages

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pitrou'
closed_at = <Date 2019-03-04.00:02:50.291>
created_at = <Date 2018-04-22.08:07:51.422>
labels = ['3.7', '3.8', 'type-bug', 'tests', 'release-blocker']
title = 'sigaddset() can fail on some signal numbers'
updated_at = <Date 2019-05-10.18:19:26.738>
user = 'https://github.com/hroncok'

bugs.python.org fields:

activity = <Date 2019-05-10.18:19:26.738>
actor = 'ned.deily'
assignee = 'pitrou'
closed = True
closed_date = <Date 2019-03-04.00:02:50.291>
closer = 'larry'
components = ['Tests']
creation = <Date 2018-04-22.08:07:51.422>
creator = 'hroncok'
dependencies = []
files = ['47546']
hgrepos = []
issue_num = 33329
keywords = ['patch']
message_count = 39.0
messages = ['315590', '315594', '315595', '315596', '315597', '315598', '315599', '315600', '315601', '315602', '315604', '315605', '315609', '315610', '315611', '315612', '315613', '315614', '315615', '315616', '315617', '315618', '315626', '315654', '315659', '315660', '315661', '315662', '315667', '315672', '315674', '315675', '315676', '315677', '336968', '336969', '337053', '337055', '337056']
nosy_count = 8.0
nosy_names = ['pitrou', 'larry', 'ned.deily', 'njs', 'neologix', 'lukasz.langa', 'hroncok', 'azanella']
pr_nums = ['6575', '6579', '6580', '6582', '12144', '12145']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33329'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[hroncok]

Originally reported in https://bugs.python.org/issue31463

This started to bother us in Fedora rawhide for various Python versions, so chances are something changed on the system level.

# python3.7 -m test.regrtest test_multiprocessing_fork 
Run tests sequentially
0:00:00 load avg: 1.24 [1/1] test_multiprocessing_fork
/usr/lib64/python3.7/multiprocessing/semaphore_tracker.py:55: UserWarning: semaphore_tracker: process died unexpectedly, relaunching.  Some semaphores might leak.
  warnings.warn('semaphore_tracker: process died unexpectedly, '
Exception in thread Thread-26:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 139, in _serve
    signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG))
  File "/usr/lib64/python3.7/signal.py", line 60, in pthread_sigmask
    sigs_set = _signal.pthread_sigmask(how, mask)
ValueError: signal number 32 out of range

(hangs...)

^CProcess Process-184:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib64/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.7/test/_test_multiprocessing.py", line 3328, in child_access
    w = conn.recv()
  File "/usr/lib64/python3.7/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/lib64/python3.7/multiprocessing/connection.py", line 960, in rebuild_connection
    fd = df.detach()
  File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/usr/lib64/python3.7/multiprocessing/connection.py", line 498, in Client
    answer_challenge(c, authkey)
  File "/usr/lib64/python3.7/multiprocessing/connection.py", line 741, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/usr/lib64/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib64/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib64/python3.7/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_fork
  Before: <_weakrefset.WeakSet object at 0x7f2abd5f6d30>
  After:  <_weakrefset.WeakSet object at 0x7f2abd5f6128> 
Warning -- threading._dangling was modified by test_multiprocessing_fork
  Before: <_weakrefset.WeakSet object at 0x7f2abd5f66d8>
  After:  <_weakrefset.WeakSet object at 0x7f2abc835048> 

Test suite interrupted by signal SIGINT.
1 test omitted:
test_multiprocessing_fork

Total duration: 3 min 43 sec
Tests result: INTERRUPTED

--------------------------------

# python3.7 -m test.regrtest test_multiprocessing_forkserver
Run tests sequentially
0:00:00 load avg: 1.00 [1/1] test_multiprocessing_forkserver
Exception in thread Thread-26:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 139, in _serve
    signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG))
  File "/usr/lib64/python3.7/signal.py", line 60, in pthread_sigmask
    sigs_set = _signal.pthread_sigmask(how, mask)
ValueError: signal number 32 out of range

(hangs...)

^CProcess Process-184:
Traceback (most recent call last):
  ...
KeyboardInterrupt
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_forkserver
  Before: <_weakrefset.WeakSet object at 0x7f0323ecde10>
  After:  <_weakrefset.WeakSet object at 0x7f0323ecd208> 
Warning -- threading._dangling was modified by test_multiprocessing_forkserver
  Before: <_weakrefset.WeakSet object at 0x7f0323ecd7b8>
  After:  <_weakrefset.WeakSet object at 0x7f0323ecddd8> 

Test suite interrupted by signal SIGINT.
1 test omitted:
test_multiprocessing_forkserver

Total duration: 55 sec
Tests result: INTERRUPTED

--------------------------------

# python3.7 -m test.regrtest test_multiprocessing_spawn
Run tests sequentially
0:00:00 load avg: 1.49 [1/1] test_multiprocessing_spawn
Exception in thread Thread-26:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 139, in _serve
    signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG))
  File "/usr/lib64/python3.7/signal.py", line 60, in pthread_sigmask
    sigs_set = _signal.pthread_sigmask(how, mask)
ValueError: signal number 32 out of range

(hangs...)

^CProcess Process-184:
Traceback (most recent call last):
  ...
KeyboardInterrupt
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_spawn
  Before: <_weakrefset.WeakSet object at 0x7fb2134a9dd8>
  After:  <_weakrefset.WeakSet object at 0x7fb2134a91d0> 
Warning -- threading._dangling was modified by test_multiprocessing_spawn
  Before: <_weakrefset.WeakSet object at 0x7fb2134a9780>
  After:  <_weakrefset.WeakSet object at 0x7fb2134a9da0> 

Test suite interrupted by signal SIGINT.
1 test omitted:
test_multiprocessing_spawn

Total duration: 54 sec
Tests result: INTERRUPTED

--------------------------------

Happens with all 3 tests on 3.4 to 3.7.

To test in docker, one can do:

$ docker run -ti fedora:rawhide /bin/bash
# dnf update
# dnf install python37  # or python3-test for 3.6, or python35, python34

Note that without dnf update, the tests work for me (for now) so attaching a full package diff that "starts" this.

[pitrou]

Perhaps you can compile and run this C program, before and after the system changes, and post the output:

#include <errno.h>
#include <signal.h>
#include <stdio.h>

int main(int argc, char** argv)
{
    sigset_t set;
    int i, ret;

    printf("NSIG = %d\n", NSIG);
    for (i = 1; i < NSIG; i++) {
        errno = 0;
        ret = sigaddset(&set, i);
        printf("sigaddset(%d) returned %d, errno =%d\n", i, ret, errno);
    }
    return 0;
}

(on Ubuntu 16.04, I get NSIG = 65, and all signal numbers work fine)

[hroncok]

I'll try. In the meantime, I've checked and it's glibc update that makes the difference:

glibc-2.27.9000-7.fc29 -> glibc-2.27.9000-14.fc29

I'll see what was changed there and whether is was intentional, or bug in Fedora.

[pitrou]

I was able to compile the program above after the system upgrade using your Docker image, and the following stands out:

sigaddset(31) returned 0, errno = 0
sigaddset(32) returned -1, errno = 22
sigaddset(33) returned -1, errno = 22
sigaddset(34) returned 0, errno = 0

However, I do not know how to compile it *before* the system upgrade ("dnf install gcc" seems to install gcc).

[hroncok]

glibc-2.27.9000-14.fc29:

NSIG = 65
sigaddset(32) returned -1, errno =22
sigaddset(33) returned -1, errno =22

[hroncok]

"dnf install gcc" upgrades glibc, yes. OK, I'll dig into it a bit more, see why and where this change happened. Thanks for hints.

[hroncok]

dnf install https://kojipkgs.fedoraproject.org/packages/glibc/2.27.9000/7.fc29/x86_64/glibc-headers-2.27.9000-7.fc29.x86_64.rpm https://kojipkgs.fedoraproject.org/packages/glibc/2.27.9000/7.fc29/x86_64/glibc-devel-2.27.9000-7.fc29.x86_64.rpm
dnf install gcc

NSIG = 65
sigaddset(1) returned 0, errno =0
... all ok

[hroncok]

glibc-2.27.9000-13.fc29 ... all ok
glibc-2.27.9000-14.fc29 ... 32, 33 fail
glibc-2.27.9000-15.fc29 (latest built) ... 32, 33 fail

13 to 14 is this in upstream commits:

d39c0a459ef32a41daac4840859bf304d931adab to 583a27d525ae189bdfaa6784021b92a9a1dae12e

[pitrou]

Ok, this is due to:
https://sourceware.org/bugzilla/show_bug.cgi?id=22391

Short explanation here:
https://unix.stackexchange.com/a/155846

[pitrou]

Miro, could you check whether 2.7 is affected too?