Python 3.10: Under some trivial circunstances, GIL not released

[jcea]

BPO	44645
Nosy	@jcea, @ambv, @markshannon, @pablogsal, @miss-islington
PRs	bpo-44645: Check for interrupts on any potentially backwards edge. #27167 [3.10] bpo-44645: Check for interrupts on any potentially backwards edge. #27183 Revert "bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)" #27194 [3.10] Revert "bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)" (GH-27194) #27195 bpo-44645: Check for interrupts on any potentially backwards edge #27216 [3.10] bpo-44645: Check for interrupts on any potentially backwards edge (GH-27216) #27235

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-07-31.23:21:29.452>
created_at = <Date 2021-07-15.13:26:07.952>
labels = ['interpreter-core', '3.10', 'release-blocker']
title = 'Python 3.10: Under some trivial circunstances, GIL not released'
updated_at = <Date 2021-07-31.23:21:29.452>
user = 'https://github.com/jcea'

bugs.python.org fields:

activity = <Date 2021-07-31.23:21:29.452>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-07-31.23:21:29.452>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2021-07-15.13:26:07.952>
creator = 'jcea'
dependencies = []
files = []
hgrepos = []
issue_num = 44645
keywords = ['patch', '3.10regression']
message_count = 11.0
messages = ['397549', '397554', '397609', '397610', '397639', '397641', '397642', '397643', '397719', '397782', '397789']
nosy_count = 5.0
nosy_names = ['jcea', 'lukasz.langa', 'Mark.Shannon', 'pablogsal', 'miss-islington']
pr_nums = ['27167', '27183', '27194', '27195', '27216', '27235']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44645'
versions = ['Python 3.10']

[jcea]

After #18334, a "while condition: pass" in a thread will preclude releasing the GIL, hanging the program with CPU at 100%.

Trivial to reproduce:

"""
#!/usr/bin/env python3.10

import threading
import time

def worker():
    while cont:
        pass

def main():
    global cont
    cont = True
    t = threading.Thread(target=worker)
    t.start()
    time.sleep(1)
    cont = False
    t.join()

if __name__ == '__main__':
    main()
"""

[markshannon]

#18334 assumes that since all loops will contain a backwards edge, checking for interrupts on JUMP_ABSOLUTE should be sufficient.

However, #23743 changed the back edges in while statements. So worker compiles to:

2 0 LOAD_GLOBAL 0 (cont)
2 POP_JUMP_IF_FALSE 7 (to 14)

3 >> 4 NOP

2 6 LOAD_GLOBAL 0 (cont)
8 POP_JUMP_IF_TRUE 2 (to 4)
10 LOAD_CONST 0 (None)
12 RETURN_VALUE
>> 14 LOAD_CONST 0 (None)
16 RETURN_VALUE

which has no CALL or JUMP_ABSOLUTE instructions.

[markshannon]

New changeset 000e70a by Mark Shannon in branch 'main':
bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)
000e70a

[pablogsal]

New changeset 0e349ea by Mark Shannon in branch '3.10':
[3.10] bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167) (GH-27183)
0e349ea

[pablogsal]

PR 27167 hangs or fails. in the refleak buildbots, so I am going to proceed to revert

[pablogsal]

To reproduce the problem:

$./python -m test test_threading -R : -v

...

test_original_excepthook (test.test_threading.ExceptHookTests) ... ok
test_system_exit (test.test_threading.ExceptHookTests) ... ok
test_can_interrupt_tight_loops (test.test_threading.InterruptMainTests) ...

(HANGS here or some time fails like in:

======================================================================
FAIL: test_can_interrupt_tight_loops (test.test_threading.InterruptMainTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/buildbot/buildarea/pull_request.cstratak-RHEL7-ppc64le.refleak/build/Lib/test/support/threading_helper.py", line 62, in decorator
    return func(*args)
           ^^^^^^^^^^^
  File "/home/buildbot/buildarea/pull_request.cstratak-RHEL7-ppc64le.refleak/build/Lib/test/test_threading.py", line 1630, in test_can_interrupt_tight_loops
    self.assertNotEqual(iterations, 0)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0 == 0

---

https://buildbot.python.org/all/#/builders/411/builds/92

[ambv]

New changeset c90c591 by Pablo Galindo Salgado in branch 'main':
Revert "bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)" (bpo-27194)
c90c591

[ambv]

New changeset 42a5514 by Miss Islington (bot) in branch '3.10':
Revert "bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)" (GH-27194) (bpo-27195)
42a5514

[markshannon]

Can we at least include the fix, until we can come up with a better test?

That test fails on 3.9 as well as 3.10 and main.

With a 3.9 build and the test in bpo-27194:

./python -m test test_threading -R :
0:00:00 load avg: 0.95 Run tests sequentially
0:00:00 load avg: 0.95 [1/1] test_threading
beginning 9 repetitions
123456789
...test test_threading failed -- Traceback (most recent call last):
  File "/home/mark/repos/cpython/Lib/test/test_threading.py", line 1479, in test_can_interrupt_tight_loops
    self.assertNotEqual(iterations, 0)
AssertionError: 0 == 0

I don't know what the underlying issue is.

Can you observe a failure without the -R flag?

[markshannon]

New changeset d09c134 by Mark Shannon in branch 'main':
bpo-44645: Check for interrupts on any potentially backwards edge (GH-27216)
d09c134