str.isspace() for U+00A0 and U+202F differs from document

[Isa-rentacs]

BPO	36502
Nosy	@malemburg, @vstinner, @benjaminp, @ezio-melotti, @gnprice, @miss-islington, @Isa-rentacs
PRs	bpo-36502: Correct documentation of str.isspace. #15019 [3.8] bpo-36502: Correct documentation of str.isspace() (GH-15019) #15296 bpo-36502: Update link to UAX #44, the Unicode doc on the UCD. #15301 [3.7] bpo-36502: Correct documentation of str.isspace() (GH-15019) (GH-15296) #15332 [3.8] bpo-36502: Update link to UAX GH-44, the Unicode doc on the UCD. (GH-15301) #15806 [3.7] bpo-36502: Update link to UAX GH-44, the Unicode doc on the UCD. (GH-15301) #15807 [2.7] bpo-36502: Update link to UAX GH-44, the Unicode doc on the UCD (GH-15301) #15808

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-08-19.10:14:33.438>
created_at = <Date 2019-04-02.06:36:07.516>
labels = ['3.8', 'type-bug', '3.7', 'expert-unicode', 'docs']
title = 'str.isspace() for U+00A0 and U+202F differs from document'
updated_at = <Date 2019-09-09.18:41:16.900>
user = 'https://github.com/Isa-rentacs'

bugs.python.org fields:

activity = <Date 2019-09-09.18:41:16.900>
actor = 'miss-islington'
assignee = 'docs@python'
closed = True
closed_date = <Date 2019-08-19.10:14:33.438>
closer = 'vstinner'
components = ['Documentation', 'Unicode']
creation = <Date 2019-04-02.06:36:07.516>
creator = 'Jun'
dependencies = []
files = []
hgrepos = []
issue_num = 36502
keywords = ['patch']
message_count = 14.0
messages = ['339317', '339318', '339336', '339339', '348947', '349678', '349947', '349948', '349950', '349983', '351526', '351536', '351545', '351546']
nosy_count = 10.0
nosy_names = ['lemburg', 'vstinner', 'benjamin.peterson', 'ezio.melotti', 'SilentGhost', 'docs@python', 'Greg Price', 'miss-islington', 'Jun_', 'Jun']
pr_nums = ['15019', '15296', '15301', '15332', '15806', '15807', '15808']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36502'
versions = ['Python 3.7', 'Python 3.8']

[Isa-rentacs]

I was looking for a list of Unicode codepoints that str.isspace() returns true.

According to https://docs.python.org/3/library/stdtypes.html#str.isspace, it's
"Whitespace characters are those characters defined in the Unicode character database as “Other” or “Separator” and those with bidirectional property being one of “WS”, “B”, or “S”."

However, for U+202F(https://www.fileformat.info/info/unicode/char/202f/index.htm) which is a "Separator" and its bidirectional property is "CS", str.isspace() returns True while it shouldn't if we follow the definition above.

>>> "\u202f".isspace()
True

I'm not sure either the documentation should be updated or behavior should be updated, but at least those should be consistent.

[SilentGhost]

I think you have to read that "and" as "or". It's sufficient that '\u202f' is a separator for it to be considered a whitespace character.

[Jun]

Do you mean read the statement as follows?

Whitespace characters are characters that satisfy either one of:

1. Character type is "Other"
2. Character type is "Separator"
3. Characters with "WS", "B", or "S" bidirectional property

If that's the case, this is also not reflect the behavior as most of characters in "Other" are not whitespace characters and in fact str.isspace() returns False for those characters.

[SilentGhost]

According to comment for _PyUnicode_IsWhitespace it's supposed to include Zs category, plus documented BIDI properties. So, I'm not sure where "Other" came from.

[gnprice]

The actual behavior turns out to match that comment. See attached PR, which adds a test confirming that and also corrects the documentation.

(A related issue is bpo-18236 -- we should probably adjust the definition to match the one Unicode now provides. But meanwhile we'll want to correct the docs.)

[vstinner]

New changeset 6bccbe7 by Victor Stinner (Greg Price) in branch 'master':
bpo-36502: Correct documentation of str.isspace() (GH-15019)
6bccbe7

[vstinner]

New changeset 8c1c426 by Victor Stinner (Greg Price) in branch '3.8':
bpo-36502: Correct documentation of str.isspace() (GH-15019) (GH-15296)
8c1c426

[miss-islington]

New changeset 0fcdd8d by Miss Islington (bot) in branch '3.7':
bpo-36502: Correct documentation of str.isspace() (GH-15019) (GH-15296)
0fcdd8d