Crash when using multiple capturing groups in `re.Scanner`

[T90REAL]

Bug report

Bug description:

The re.Scanner class crashes with a segmentation fault (memory corruption) when any pattern in the lexicon contains named capturing groups.

import re

lexicon = [
    ('(?P<group1>a)(?P<group2>b)', None),  # Named capturing groups
]

scanner = re.Scanner(lexicon)
result, leftover = scanner.scan("ab")  # Segmentation fault occurs here

zsh: segmentation fault  python3 test.py

Scanner should process the input normally and return scan results without crashing, but Python interpreter crashes with segmentation fault, indicating memory corruption.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-140797: Forbid capturing groups in re.Scanner lexicon patterns #140944
• [3.14] gh-140797: Forbid capturing groups in re.Scanner lexicon patterns (GH-140944) #140982
• [3.13] gh-140797: Forbid capturing groups in re.Scanner lexicon patterns (GH-140944) #140983
• gh-140979: Fix off-by-one error in the RE code validator #140984
• [3.13] Revert "[3.13] gh-140797: Forbid capturing groups in re.Scanner lexicon patterns (GH-140944) (GH-140983)" #142231
• [3.14] Revert "gh-140797: Forbid capturing groups in re.Scanner lexicon patterns (GH-140944) (GH-140982)" #142309

[picnixz]

FTR, the traceback with faulthandler is:

Debug memory block at address p=0x7f5167149ea0: API 'm'
    16 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x7f5167149eb0 are not all FORBIDDENBYTE (0xfd):
        at tail+0: 0xf9 *** OUCH
        at tail+1: 0x3d *** OUCH
        at tail+2: 0x7a *** OUCH
        at tail+3: 0x66 *** OUCH
        at tail+4: 0x51 *** OUCH
        at tail+5: 0x7f *** OUCH
        at tail+6: 0x00 *** OUCH
        at tail+7: 0x00 *** OUCH
    Data at p: f8 3d 7a 66 51 7f 00 00 fa 3d 7a 66 51 7f 00 00

Memory block allocated at (most recent call first):
  File "/python/cpython/Lib/re/__init__.py", line 410

Fatal Python error: _PyMem_DebugRawFree: bad trailing pad byte
Python runtime state: initialized

Current thread 0x00007f5167222740 [python] (most recent call first):

[picnixz]

Looks like a deallocation issue in the scanner as the function completes correctly.

[picnixz]

So the segfault occurs in the destructor when trying to free the memory of state->mark (PyMem_Free((void*) state->mark);). Looks like either it was already freed and we have a double-free somewhere or something else is happening.

[picnixz]

Ok, I was able to patch this but only by messing a bit:

• state->mark is allocated as state->mark = (const void **)PyMem_New(const void *, pattern->groups * 2); So it depends on the number of patterns.
• The actual value here is pattern->groups == 1 when we're in the state that crash but I'm not sure it's correct here.
• When putting other kind of RE in the lexicon, it doesn't even work (I get invalid SRE code for '(a)(b)(c)' for instance).
• I get other crashes when having more capturing groups.

I think it's a more general issue here. My gut feeling is telling me that the crash is happening because we don't reset explicitly the memory that was allocated for the catching groups but I may be wrong (that is, in state_reset).

@serhiy-storchaka As you're more familiar with the state machine, can you investigate this please?