str.isidentifier() does not work with non-BMP non-canonicalized strings on Windows

[serhiy-storchaka]

BPO	40596
Nosy	@vstinner, @serhiy-storchaka
PRs	bpo-40596: Fix str.isidentifier() for non-canonicalized strings containing non-BMP characters on Windows. #20035 bpo-40596: Fix str.isidentifier() for non-canonicalized strings containing non-BMP characters on Windows. #20053

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-05-12.13:19:12.227>
created_at = <Date 2020-05-11.17:50:16.125>
labels = ['interpreter-core', 'type-bug', '3.9']
title = 'str.isidentifier() does not work with non-BMP non-canonicalized strings on Windows'
updated_at = <Date 2020-05-12.17:27:57.214>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2020-05-12.17:27:57.214>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-12.13:19:12.227>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2020-05-11.17:50:16.125>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 40596
keywords = ['patch']
message_count = 6.0
messages = ['368637', '368651', '368652', '368705', '368729', '368739']
nosy_count = 2.0
nosy_names = ['vstinner', 'serhiy.storchaka']
pr_nums = ['20035', '20053']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40596'
versions = ['Python 3.9']

[serhiy-storchaka]

>>> import _testcapi
>>> u = '\U0001d580\U0001d593\U0001d58e\U0001d588\U0001d594\U0001d589\U0001d58a'
>>> u.isidentifier()
True
>>> _testcapi.unicode_legacy_string(u).isidentifier()
False

[vstinner]

It's maybe time to speed up the deprecation of the legacy C API using Py_UNICODE...

[vstinner]

My previous change on this function:

commit f3e7ea5
Author: Victor Stinner <vstinner@python.org>
Date: Tue Feb 11 14:29:33 2020 +0100

bpo-39500: Document PyUnicode_IsIdentifier() function (GH-18397)

PyUnicode_IsIdentifier() does not call Py_FatalError() anymore if the
string is not ready.

[serhiy-storchaka]

I am not sure that changes in bpo-39500 was correct. It is easier to catch a bug if crash consistently when you pass a non-canonicalized strings then if silently return a wrong result for specific input on particular platform.

Alternatively, you could reimplement correct handling of surrogate pairs in PyUnicode_IsIdentifier().

[serhiy-storchaka]

New changeset 5650e76 by Serhiy Storchaka in branch 'master':
bpo-40596: Fix str.isidentifier() for non-canonicalized strings containing non-BMP characters on Windows. (GH-20053)
5650e76

[vstinner]

Thanks for the fix Serhiy!