test_exceptions.ExceptionTests.test_recursion_in_except_handler stack overflow on Windows debug builds

[Fidget-Spinner]

BPO	44348
Nosy	@pfmoore, @orsenthil, @vstinner, @tjguk, @markshannon, @zware, @zooba, @pablogsal, @isidentical, @Fidget-Spinner
PRs	bpo-44348: Revert "bpo-39573: Py_TYPE becomes a static inline function (GH-26493)" #26596 bpo-44348: Move trace-info to thread-state #26623 bpo-44348: Use debug builds for windows CI #26627 bpo-44348: BaseException deallocator uses trashcan #28190
Files	bench.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-07.13:43:45.052>
created_at = <Date 2021-06-08.09:47:57.526>
labels = ['OS-windows', 'type-crash', '3.11']
title = 'test_exceptions.ExceptionTests.test_recursion_in_except_handler stack overflow on Windows debug builds'
updated_at = <Date 2021-09-07.16:45:25.006>
user = 'https://github.com/Fidget-Spinner'

bugs.python.org fields:

activity = <Date 2021-09-07.16:45:25.006>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-07.13:43:45.052>
closer = 'vstinner'
components = ['Windows']
creation = <Date 2021-06-08.09:47:57.526>
creator = 'kj'
dependencies = []
files = ['50266']
hgrepos = []
issue_num = 44348
keywords = ['patch']
message_count = 17.0
messages = ['395316', '395320', '395322', '395340', '395392', '395451', '395514', '400992', '401150', '401161', '401185', '401194', '401195', '401269', '401271', '401305', '401307']
nosy_count = 10.0
nosy_names = ['paul.moore', 'orsenthil', 'vstinner', 'tim.golden', 'Mark.Shannon', 'zach.ware', 'steve.dower', 'pablogsal', 'BTaskaya', 'kj']
pr_nums = ['26596', '26623', '26627', '28190']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue44348'
versions = ['Python 3.11']

[Fidget-Spinner]

Hi all, I created this issue after discussion in https://bugs.python.org/issue39573#msg395206:

In bpo-39573 "[C API] Make PyObject an opaque structure in the limited C API" the commit f3fa63e ("Py_TYPE becomes a static inline function") may have caused test_recursion_in_except_handler to fail only on windows debug builds (according to git bisect anyways) due to higher stack consumption causing a stack overflow.

That test was added in 4e7a69b (Dec 2020), and was passing until last week. Currently another test (bpo-11105, test_ast test_recursion_direct) is masking the broken test, but it's also a stack overflow in windows debug. When I reverted commit f3fa63e, test_recursion_in_except_handler passes but test_recursion_direct still fails.

For test_recursion_direct, the overflow occurs after the Py_Enter/LeaveRecursiveCall guard, when calling some other function like _PyObject_LookupAttr.

I can think of a few possible ways to fix this (in no particular order):

1. Increase stack size for windows builds from 2MB (some valid points against this though, see msg395177).
2. Decrease the global recursion limit only for windows debug.
3. Custom recursion limits for each module depending on OS, which Batuhan has been working on for AST module at documentation minor issue : "Step back: WSGI" section from "HOWTO Use Python in the web" #70737.
4. Skip the tests on windows debug, since most people on windows use release builds anyways which are unaffected.

Thanks for your time! Which approach would you prefer?

[pablogsal]

There are many buildbot failures today and this is getting a bit out of hand so I'm starting with a revert of f3fa63e until we have a better plan.

[pablogsal]

New changeset 6d518bb by Pablo Galindo in branch 'main':
bpo-44348: Revert "bpo-39573: Py_TYPE becomes a static inline function (GH-26493)" (GH-26596)
6d518bb

[zooba]

Decrease the global recursion limit only for windows debug.

This is already what we do, so if someone has increased stack usage, they should also decrease the value here.

Hopefully the non-debug value doesn't have to be reduced. In that case, it's worth looking at the stack allocations to reduce it (e.g. I once fixed this by moving a rarely used buffer into a dynamic allocation rather than a local, which saved about 1KB/frame).

[vstinner]

See also bpo-44360 "test_compile segfault on AMD64 Ubuntu 3.x".