IDLE: Have the shell mimic terminal handling of \r and \b control characters in outputs

[taleinat]

BPO	37827
Nosy	@rhettinger, @terryjreedy, @taleinat, @csabella
PRs	bpo-37827: IDLE shell handling of \r and \b control chars #15211

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2020-09-19.19:10:07.559>
created_at = <Date 2019-08-12.06:52:53.755>
labels = ['expert-IDLE', 'type-feature', '3.9']
title = 'IDLE: Have the shell mimic terminal handling of \\r and \\b control characters in outputs'
updated_at = <Date 2020-09-19.19:57:03.203>
user = 'https://github.com/taleinat'

bugs.python.org fields:

activity = <Date 2020-09-19.19:57:03.203>
actor = 'taleinat'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2020-09-19.19:10:07.559>
closer = 'taleinat'
components = ['IDLE']
creation = <Date 2019-08-12.06:52:53.755>
creator = 'taleinat'
dependencies = []
files = []
hgrepos = []
issue_num = 37827
keywords = ['patch']
message_count = 8.0
messages = ['349440', '349443', '350474', '350599', '350639', '356591', '377182', '377191']
nosy_count = 4.0
nosy_names = ['rhettinger', 'terry.reedy', 'taleinat', 'cheryl.sabella']
pr_nums = ['15211']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37827'
versions = ['Python 3.9']

[taleinat]

IDLE's shell doesn't currently handle \r and \b in any special way; they are written the the Tk Text widget which displays them in strange, system-dependent ways.

These are often used to show continuously updated progress, e.g. in text-based progress bars, without flooding the output, since they allow overwriting previously written output. If we implement handling for \r and \b, progress indicators such as these could finally work properly in IDLE's shell.

To make things worse, Tk's Text widget becomes increasingly slow when it wraps very long lines. IDLE's shell must wrap lines, and is therefore prone to such slowdowns. Attempting to show updating progress info using \r or \b results in such increasingly long lines of output, eventually slowing IDLE's shell down to a crawl. (The recent addition of squeezing of long outputs help for the case of single, very long outputs, but not with many short strings written on a single line.)

As a recent example, the basic Tensorflow tutorial shows such progress information for several of its stages. Due to the lack of handling for these control characters, it is practically unusable in the IDLE shell due to this. See issue bpo-37762 about this.

Since the shell aims to closely emulate using an interactive terminal session, and since showing progress is so common in interactive work, I propose to add special handling of these two control characters in outputs written to the shell window.

Related issues:

bpo-23220: Documents input/output effects of how IDLE runs user code (originally titled "IDLE does not display \b backspace correctly")

bpo-24572: IDLE Text Output With ASCII Control Codes Not Working (marked as duplicate of bpo-23220)

bpo-37762: IDLE very slow due a super long line output in chunks

StackOverflow: what is the difference between cmd and idle when using tqdm? (https://stackoverflow.com/questions/35895864/what-is-the-difference-between-cmd-and-idle-when-using-tqdm)

[taleinat]

See PR #59416 with a working implementation.

[gvanrossum]

I agree -- this was added to Emacs a long time ago and it makes a big difference for people (like myself) who do a lot of work in Emacs shell windows. I imagine it's the same for IDLE.

[terryjreedy]

In this post, I slightly expand the terminal mode proposal and define the context in which I will accept it. The context will be the subject of other issues. In the next post (tomorrow) I will try to define a reviewable specification for the terminal mode.
---

The proposal that IDLE's should *always* respond to \b and \r "properly
, like terminals" is the combined proposal of bpo-24572 and the original proposal of bpo-23220. The combined proposal was rejected in the latter, for reasons best given by Serhiy in msg246602. More is given below. bpo-23220 was then redefined as a doc issue.

I noted in bpo-23220 that one solution for people who wanted terminal-like behavior would be to add an option to execute code being edited to the system terminal. However, this a) throws away the benefits of running in Shell and b) would not apply to interactive exploration. While running in a terminal is needed for certain use cases, it is not needed for handling control of line output.

What Shell should do with control characters in output depends on the users goal. I propose that Shell should have 3 display modes that treat ascii control codes differently for different use cases.

1. Raw mode (the current mode). IDLE inserts all BMP chars into the tk Text. This is needed if one is using a font that has glyphs associated with control codes. On Windows 10, the 8514oem font (the MSDOS font) has symbols for all control codes except NUL and newline. This is a rare need; this mode should not be the default.

2. Development mode (my proposed new default). To the extent possible, developers should be able to see the characters their program produces without adding repr() to all their print and write statements (and later deleting them). Control chars would be replaced with unique glyphs, such as 'Control Pictures', \u2400-\u241f,
   ␀ ␁ ␂ ␃ ␄ ␅ ␆ ␇ ␈␉␊␋␌␍␎␏␐ ␑ ␒ ␓ ␔ ␕ ␖ ␗ ␘ ␙ ␚ ␛ ␜ ␞ ␟.
   or maybe circled letters, \u24d0-\u24df,\u24b6-\u24c5, ⓐ-ⓟ, Ⓐ-Ⓟ. Replacements would be tagged as such and get distinct highlight colors. The purpose of a visible tag is to differentiate replacements from characters actually output by the users program.

To fulfill the goal of Development mode, more is needed. Astral chars, instead of causing an exception, should be replaced by a string consisting of their \U escape and tagged as a replacement. This is needed anywhere astral chars can appear. It would also be nice to get the codepoint and name of any displayed character. If implemented, this should be added to the base editor class.

3. Terminal mode. Tk interprets \n and \t. Also interpret (to begin with) \a, \b, and \r. The motivation for the latter two has already been given. As said above, the specification is deferred to another post.

[taleinat]

Terry, thanks for the detailed writing of your thoughts on the matter and their context.

Serhiy's argument (in msg246602) is that different terminals interpret different control characters in different ways, and that we have no way of unifying their behavior. This is true, and I will add that we should also not aim to fully emulate any single terminal nor multiple types of terminals.

On the other hand, there is a common ground where the vast majority of terminals do, in fact, behave very similarly WRT control characters. For example, IDLE already does interpret '\n' in a special way (partially because the underlying Tk text widget does). The same goes for '\t'.

I argue that we should instead *slightly* expand the set of control characters which IDLE interprets, to include a few more which are universally treated in a consistent manner.

Looking at the list of ASCII control characters on Wikipedia[1], I don't think any beyond \a, \b, \n, \r and \t are universal enough and in common enough use to merit inclusion. (For reference, \a is for "bell".)

..[1]: https://en.wikipedia.org/wiki/C0_and_C1_control_codes

I find the suggestion to have more than a single "mode" for the IDLE shell contrary to the "simple and novice friendly" design principle that we are aiming for. I also think that it would bring little added benefit to the great majority of our users.

As for other control characters and astral characters, I very much agree that we could do better than to have them often "garbled" as done by the Tk text widget. I think this should be dealt with as a separate issue.

[taleinat]

Note that most of the issues with special characters described in Terry latest comment have been addressed by the fix for bpo-13153.

Can we continue discussing just properly rendering \r and \b, e.g. as done in PR #59416? Let's just decide whether we want this or not.

My main point in favor is that we already have special support for two similar control characters, \n and \t. IMO we should expand this to the other very standardized ones, \r and \b, and perhaps \a ("bell") as well.

[taleinat]

As the creator of this issue, I'm reverting the name back to the original, since I did not propose to add a new "mode" to the IDLE shell.

Such a proposal could be a separate issue.

I'm unfortunately going to mark close this as "rejected". If in the future there is interest in this again, it may be reopened.

[taleinat]

I will highlight the fact that IDLE's shell currently allows no way of showing a progress indicator, beyond extremely simple bar that only fill up from left to right, without any indication of the rightmost limit, a percentage indicator, or a rotating progress indicator. Specifically, all common progress indicator libraries do not work properly in IDLE.

This is a true detriment to interactive work.