IDLE: remove \b from colorizer string prefix

[terryjreedy]

BPO	33204
Nosy	@tim-one, @terryjreedy, @taleinat, @serhiy-storchaka, @miss-islington
PRs	bpo-33204: IDLE: consistently color invalid string prefixes #6344 [3.7] bpo-33204: IDLE: consistently color invalid string prefixes (GH-6344) #6345 [3.6] bpo-33204: IDLE: consistently color invalid string prefixes (GH-6344) #6346 bpo-33204: IDLE - revise and extend colorizer test #6347 [3.7] bpo-33204: IDLE - revise and extend colorizer test. (GH-6347) #6348 [3.6] bpo-33204: IDLE - revise and extend colorizer test. (GH-6347) #6349

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2018-07-30.09:09:52.952>
created_at = <Date 2018-04-02.03:37:29.763>
labels = ['3.8', 'expert-IDLE', 'type-bug', '3.7']
title = 'IDLE: remove \\b from colorizer string prefix'
updated_at = <Date 2018-07-30.09:09:52.952>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2018-07-30.09:09:52.952>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2018-07-30.09:09:52.952>
closer = 'terry.reedy'
components = ['IDLE']
creation = <Date 2018-04-02.03:37:29.763>
creator = 'terry.reedy'
dependencies = []
files = []
hgrepos = []
issue_num = 33204
keywords = ['patch']
message_count = 13.0
messages = ['314793', '314794', '314796', '314797', '314802', '314803', '314804', '314805', '314810', '314811', '314812', '314813', '322659']
nosy_count = 5.0
nosy_names = ['tim.peters', 'terry.reedy', 'taleinat', 'serhiy.storchaka', 'miss-islington']
pr_nums = ['6344', '6345', '6346', '6347', '6348', '6349']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33204'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[terryjreedy]

Lib\idlelib\colorizer.py, line 25, is:
stringprefix = r"(?i:\br|u|f|fr|rf|b|br|rb)?"

The r prefix, but only the r prefix, must be preceded by a non-word character.

On pydev thread "IDLE colorizer", MRAB noted: "The \b will apply only to the first choice. Shouldn't it be more like:
stringprefix = r"(?:\b(?i:r|u|f|fr|rf|b|br|rb))?"

I think the \b should be removed instead. Currently uf'a' gets f'a' colored, while ur'a' only gets 'a' colored. (Tim Peters pointed out the difference with a different example.) I prefer coloring the maximal legal string rather than the minimum. I think the contrast between two chars legal by themselves, but differently colored when put together, makes the bug more obvious.

[terryjreedy]

The current test is part of the htest. The section
# All valid prefixes for unicode and byte strings should be colored
contains "ur'is invalid'". This should be moved to a new section with other invalid combinations.

# Invalid combinations of legal characters should be half colored.
ur'x', ru'x', uf'x', fu'x', UR'x'
(only one CAPS example is needed now).

[tim-one]

Sounds good (removing \b) to me, Terry!

[terryjreedy]

To see the visually verified test, run Lib/idlelib/colorizer as the main module, either from a command line or editor, and click the button.

[terryjreedy]

New changeset da58533 by Terry Jan Reedy in branch 'master':
bpo-33204: IDLE: consistently color invalid string prefixes (bpo-6344)
da58533

[miss-islington]

New changeset 6130675 by Miss Islington (bot) in branch '3.7':
bpo-33204: IDLE: consistently color invalid string prefixes (GH-6344)
6130675

[miss-islington]

New changeset 16cf84b by Miss Islington (bot) in branch '3.6':
bpo-33204: IDLE: consistently color invalid string prefixes (GH-6344)
16cf84b

[serhiy-storchaka]

Is ''or'' correctly colorized?

[terryjreedy]

Yes, still correct, as a keyword. Legal "x or''" is colored as "x" <normal> "or" <keyword> "''" <string>. Legal "if'':" is colored as "if" <keyword> "''" <string> ":" <normal>". New PR revises colorizer test and add the above.

[terryjreedy]

New changeset 55966f3 by Terry Jan Reedy in branch 'master':
bpo-33204: IDLE - revise and extend colorizer test. (GH-6347)
55966f3