[doc] Clarify exactly what \w matches in UNICODE mode #69929
Description
| BPO | 25743 |
|---|---|
| Nosy | @ezio-melotti, @iritkatriel, @slateny |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2015-11-27.15:50:58.768>
labels = ['expert-regex', 'easy', '3.9', '3.10', '3.11', 'type-feature', 'docs']
title = '[doc] Clarify exactly what \\w matches in UNICODE mode'
updated_at = <Date 2022-02-28.07:39:46.535>
user = 'https://bugs.python.org/zwol'bugs.python.org fields:
activity = <Date 2022-02-28.07:39:46.535>
actor = 'slateny'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Regular Expressions']
creation = <Date 2015-11-27.15:50:58.768>
creator = 'zwol'
dependencies = []
files = []
hgrepos = []
issue_num = 25743
keywords = ['easy']
message_count = 5.0
messages = ['255463', '255464', '255465', '407440', '414180']
nosy_count = 7.0
nosy_names = ['ezio.melotti', 'mrabarnett', 'docs@python', 'zwol', 'Andi McClure', 'iritkatriel', 'slateny']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue25743'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
Activity
zwol commentedon Nov 27, 2015
The
remodule documentation does not do a good job of explaining exactly what\wmatches. Quoting https://docs.python.org/3.5/library/re.html :Empirically, this appears to mean "everything in Unicode general categories L* and N*, plus U+005F (underscore)". That is a perfectly sensible definition and the documentation should state it in those terms. "Unicode word characters" could mean any number of different things; note for instance that UTS#18 gives a very different definition.
(Further reading: https://gist.github.com/zackw/3077f387591376c7bf67 plus links therefrom).
AndiMcClure commentedon Nov 27, 2015
I would like to request also a clear explanation be given for the documentation in the 2.7 branch. From https://docs.python.org/2.7/library/re.html :
"\w ... If UNICODE is set, this will match the characters [0-9_] plus whatever is classified as alphanumeric in the Unicode character properties database"
This is ambiguous. Does it mean the "Alphabetic" property from UAX#44? Does it mean something else?
zwol commentedon Nov 27, 2015
FWIW, the actual behavior of \w matching "everything in Unicode general categories L* and N*, plus U+005F (underscore)" is consistent across all versions I can conveniently test (2.7, 3.4, 3.5).
In 2.7, there are four characters in general category Nl that \w doesn't match, but I believe that is just a bug, not an intentional difference of behavior.
iritkatriel commentedon Dec 1, 2021
It's too late for the 2.7 docs, but the current docs can still be updated.
[-]Clarify exactly what \w matches in UNICODE mode[/-][+][doc] Clarify exactly what \w matches in UNICODE mode[/+]slateny commentedon Feb 28, 2022
Would a change like this be accurate?
Matches Unicode word characters; this includes most alphanumeric characters as well as the underscore. In Unicode, alphanumeric characters are defined to be the general categories L + N (see https://unicode.org/reports/tr44/#General_Category_Values). If the :const:`ASCII` flag is used, only ``[a-zA-Z0-9_]`` is matched.
gh-69929: re docs: Add more specific definition of \w (#92015)
pythongh-69929: re docs: Add more specific definition of \w (pythonGH…
pythongh-69929: re docs: Add more specific definition of \w (pythonGH…
gh-69929: re docs: Add more specific definition of \w (GH-92015)
gh-69929: re docs: Add more specific definition of \w (GH-92015)
pythongh-69929: re docs: Add more specific definition of \w (python#9…
Merged main into regmachine branch (#50)