Repository navigation
re: limit the maximum capturing group to 1,073,741,823, reduce sizeof(match_context). #91412
Description
| BPO | 47256 |
|---|---|
| Nosy | @ezio-melotti, @serhiy-storchaka, @animalize |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2022-04-08.06:54:08.832>
labels = ['3.11', 'library', 'performance']
title = 're: limit the maximum capturing group to 1,073,741,823, reduce sizeof(match_context).'
updated_at = <Date 2022-04-08.07:02:49.829>
user = 'https://github.com/animalize'bugs.python.org fields:
activity = <Date 2022-04-08.07:02:49.829>
actor = 'malin'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-04-08.06:54:08.832>
creator = 'malin'
dependencies = []
files = []
hgrepos = []
issue_num = 47256
keywords = ['patch']
message_count = 1.0
messages = ['416960']
nosy_count = 4.0
nosy_names = ['ezio.melotti', 'mrabarnett', 'serhiy.storchaka', 'malin']
pr_nums = ['32411']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue47256'
versions = ['Python 3.11']
Activity
animalize commentedon Apr 8, 2022
These changes reduce sizeof(match_context):
sre uses stack and
match_contextstruct to simulate recursive call, smaller struct brings:- deeper recursive call
- less memory consume
- less memory realloc
Here is a test, if limit the stack size to 1 GiB, the max available value of n is:
1,073,741,823 capturing groups should enough for almost all users.
If limit it to 16,383 (2-byte integer), the context size may reduce more. But maybe some patterns generated by program will have more than this number of capturing groups.
1️⃣Performance:
Before
regex_dna: Mean +- std dev: 149 ms +- 1 ms
regex_effbot: Mean +- std dev: 2.22 ms +- 0.02 ms
regex_v8: Mean +- std dev: 22.3 ms +- 0.1 ms
my benchmark[1]: 13.9 sec +- 0.0 sec
Commit 1. limit the maximum capture group to 1,073,741,823
regex_dna: Mean +- std dev: 150 ms +- 1 ms
regex_effbot: Mean +- std dev: 2.16 ms +- 0.02 ms
regex_v8: Mean +- std dev: 22.3 ms +- 0.1 ms
my benchmark: 13.8 sec +- 0.0 sec
Commit 2. further reduce sizeof(SRE(match_context))
regex_dna: Mean +- std dev: 150 ms +- 1 ms
regex_effbot: Mean +- std dev: 2.16 ms +- 0.02 ms
regex_v8: Mean +- std dev: 22.2 ms +- 0.1 ms
my benchmark: 13.8 sec +- 0.1 sec
If further change the types of toplevel/jump from int to char, in 32-bit build sizeof(match_context) will be reduced from 36 to 32 (In 64-bit build still 56). But it's slower on 64-bit build, so I didn't adopt it:
regex_dna: Mean +- std dev: 150 ms +- 1 ms
regex_effbot: Mean +- std dev: 2.18 ms +- 0.01 ms
regex_v8: Mean +- std dev: 22.4 ms +- 0.1 ms
my benchmark: 14.1 sec +- 0.0 sec
2️⃣ The type of match_context.count is Py_ssize_t
- If change it to 4-byte integer, need to modify some engine code.
- If keep it as Py_ssize_t, SRE_MAXREPEAT may >= 4 GiB in future versions.
Currently SRE_MAXREPEAT can't >= 4 GiB.
So the type of match_context.count is unchanged.
[1] My re benchmark, it uses 16 patterns to process 100 MiB text data:
https://github.com/animalize/re_benchmarks