IDLE needs syntax highlighting for async and await

[DavidEFrancoG]

BPO	29706
Nosy	@terryjreedy, @vadmium, @1st1, @miss-islington
PRs	bpo-29706: Test that IDLE colors async/await as keywords. #6846 [3.7] bpo-29706: Test that IDLE colors async/await as keywords. (GH-6846) #6867 [3.6] bpo-29706: Test that IDLE colors async/await as keywords. (GH-6846) #6868 [3.6] bpo-29706: IDLE now colors async and await as keywords in 3.6. #6879
Files	no syntax highlighting.png: sample screenshot of the current behavior colorize_async.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2018-05-16.01:38:52.201>
created_at = <Date 2017-03-03.03:43:57.355>
labels = ['easy', 'expert-IDLE', 'type-feature']
title = 'IDLE needs syntax highlighting for async and await'
updated_at = <Date 2018-05-16.01:38:52.200>
user = 'https://bugs.python.org/DavidEFrancoG'

bugs.python.org fields:

activity = <Date 2018-05-16.01:38:52.200>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2018-05-16.01:38:52.201>
closer = 'terry.reedy'
components = ['IDLE']
creation = <Date 2017-03-03.03:43:57.355>
creator = 'David E. Franco G.'
dependencies = []
files = ['46692', '46693']
hgrepos = []
issue_num = 29706
keywords = ['patch', 'easy']
message_count = 15.0
messages = ['288849', '288859', '288860', '288868', '288879', '288902', '288920', '288935', '288940', '288945', '316613', '316690', '316711', '316722', '316746']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'martin.panter', 'yselivanov', 'David E. Franco G.', 'miss-islington']
pr_nums = ['6846', '6867', '6868', '6879']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29706'
versions = ['Python 3.6']

[DavidEFrancoG]

Well, this is pretty self explanatory, when playing with this new features of async and await (https://docs.python.org/3.5/whatsnew/3.5.html#new-features) I found to me surprise that there is no syntax highlighting for it in the IDLE for py3.5 and also for py3.6

So I humbly ask for its addition. Thanks

[terryjreedy]

I presume you would like 'async' and 'await' highlighted as keywords. However, IDLE takes its definition of 'keyword' from keyword.kwlist. 'Async' and 'await' are currently not on that list as they are not yet keywords.

>> async = 1
>> await = 2

According to https://www.python.org/dev/peps/pep-0492/#transition-plan, the intention was (is?) to make them keywords in 3.7. As of Feb 11, that had not happened yet. If and when it does, this issue will be taken care of.

I may consider adding a special context sensitive case for 3.6, if the colorizer code makes it easily possible. Care is needed since it would be a mistake to mark them as keywords in the above statements.

The difficulty is that colorizer uses regexes and a bit of context sensitive code, while python is using a full grammar parse. I believe that either at the beginning of a line and 'async' followed by 'for' or 'with' or 'await' not followed by certain punctuation ('.', ',', or '=') should be treated as a keyword.

Yury, does the rule above look about right? 'Await' seems trickier than 'async'.

I think it may be possible to add regexes that are not literal words to the kwlist. If so, adding 'async +def ', 'async +for ', 'async +with ', and 'await +[^.,=]' and not worrying about 'beginning or line' or tabs (instead of spaces) between would be easy and probably good enough.

[1st1]

According to https://www.python.org/dev/peps/pep-0492/#transition-plan, the intention was (is?) to make them keywords in 3.7. As of Feb 11, that had not happened yet. If and when it does, this issue will be taken care of.

I plan to do that in the next couple of weeks.

I may consider adding a special context sensitive case for 3.6, if the colorizer code makes it easily possible. Care is needed since it would be a mistake to mark them as keywords in the above statements.

If possible I would make 'async' and 'await' to be always highlighted. Even if a user writes <3.7 code and uses 'async' as a variable, it would still be great to let the user know that there is something wrong with this variable.

[terryjreedy]

Always coloring 'async' and 'await' is trivial (augment expression and add test line) and acceptable to me.

I am a git neophyte and am waiting for workflow development to settle down before working on new setup for git. If you want to submit the change for 3.6.1, go ahead. (But leave issue open.) Otherwise, I will get it into 3.6.2 (and 3.7). Patch is against 3.6 idlelib.colorize, but 3.7 is identical.

The test is to run colorizer.py as main module (python -m idlelib.colorize, or F5 in IDLE editor). Click 'Test' button and check that 'async' and 'await' have same color as other keywords. This cannot affect anything in test suite; colorizer is only used by turtledemo, which is not tested, and which will not notice anyway.

[vadmium]

See also bpo-26264 about the “keyword” module

[DavidEFrancoG]

I presume you would like 'async' and 'await' highlighted as keywords.

Yes.
Some others tools like IPython and the Spider IDE that come with the Anaconda package already highlighted them as keywords, even if you can use them as normal variables.

The test is to run colorizer.py as main module (python -m >idlelib.colorize, or F5 in IDLE editor). Click 'Test' button and check >that 'async' and 'await' have same color as other keywords. This >cannot affect anything in test suite; colorizer is only used by >turtledemo, which is not tested, and which will not notice anyway.

super, what I have to do to put the patch so I can test it?

[terryjreedy]

David, the easiest thing to do would be to copy and paste the following
+ ['async', 'await']
into this line of colorizer.py (ColorDelegator.py in 3.5)
kw = r"\b" + any("KEYWORD", keyword.kwlist) + r"\b"
about line 17, to get
kw = r"\b" + any("KEYWORD", keyword.kwlist + ['async', 'await']) + r"\b"

[DavidEFrancoG]

Ok, Done.
that work in both 3.5 and 3.6
I also did the

python -m idlelib.ColorDelegator

and

python -m idlelib.colorizer

and they turn ok

I also open those modules in their respective idle and run them, ColorDelegator work ok but colorizer throw me this error

    Traceback (most recent call last):
      File "C:\Anaconda3\Lib\idlelib\colorizer.py", line 279, in <module>
        verbosity=2, exit=False)
      File "C:\Anaconda3\lib\unittest\main.py", line 63, in __init__
        self.module = __import__(module)
      File "C:\Anaconda3\lib\idlelib\idle_test\test_colorizer.py", line 8, in <module>
        from test.support import requires
    ImportError: bad magic number in 'test': b'\x03\xf3\r\n'
    >>>