IDLE: Document how Shell displays user code output

[AlSweigart]

BPO	23220
Nosy	@terryjreedy, @taleinat, @ned-deily, @4kir4, @vadmium, @serhiy-storchaka, @willingc, @miss-islington
PRs	bpo-23220: Explain how IDLE's Shell displays output #10356 [3.7] bpo-23220: Explain how IDLE's Shell displays output (GH-10356) #10365 [3.6] bpo-23220: Explain how IDLE's Shell displays output (GH-10356) #10366 [3.7] bpo-23220: Explain how IDLE's Shell displays output (GH-10356) #10368 [3.6] bpo-23220: Explain how IDLE's Shell displays output (GH-10356) #10369
Dependencies	bpo-33000: IDLE Doc: Text consumes unlimited RAM, consoles likely not

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2018-11-10.23:34:26.296>
created_at = <Date 2015-01-11.00:54:47.152>
labels = ['3.8', 'expert-IDLE', 'type-feature', '3.7', 'docs']
title = 'IDLE: Document how Shell displays user code output'
updated_at = <Date 2019-08-12.07:00:43.640>
user = 'https://bugs.python.org/AlSweigart'

bugs.python.org fields:

activity = <Date 2019-08-12.07:00:43.640>
actor = 'taleinat'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2018-11-10.23:34:26.296>
closer = 'terry.reedy'
components = ['Documentation', 'IDLE']
creation = <Date 2015-01-11.00:54:47.152>
creator = 'Al.Sweigart'
dependencies = ['33000']
files = []
hgrepos = []
issue_num = 23220
keywords = ['patch']
message_count = 28.0
messages = ['233828', '233830', '233831', '233832', '233838', '233862', '234162', '246596', '246602', '246614', '246626', '246629', '246633', '246673', '251825', '261528', '261529', '261531', '261738', '328856', '329370', '329373', '329374', '329391', '329650', '349392', '349438', '349442']
nosy_count = 11.0
nosy_names = ['terry.reedy', 'taleinat', 'ned.deily', 'THRlWiTi', 'docs@python', 'akira', 'martin.panter', 'Al.Sweigart', 'serhiy.storchaka', 'willingc', 'miss-islington']
pr_nums = ['10356', '10365', '10366', '10368', '10369']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23220'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[AlSweigart]

IDLE cannot display the \b backspace character correctly. From IDLE's interactive shell:

>>> print('hello\b\b\b\b\bHELLO')
hello�������◙◙◙◙◙HELLO

Whereas from cmd:

>>> print('hello\b\b\b\b\bHELLO')
HELLO

[willingc]

FWIW: On Mac OS X 10.9.5 using Python 3.4.2, IDLE's interactive shell (started from the IDLE.app icon):

>>print('hello\b\b\b\b\bHELLO')
helloHELLO

From the command line using python3 interactive shell:

>>print('hello\b\b\b\b\bHELLO')
HELLO

Both return a <class 'str'> for type('hello\b\b\b\b\bHELLO')
Interestingly, both behave the same when executing:

>>'hello\b\b\b\b\bHELLO'
'hello\x08\x08\x08\x08\x08HELLO'

I'm not sure that IDLE is used much on OS X since the Terminal is easily available. Since K12 education may use it, it would be nice to have consistency across the OSes.

[vadmium]

As far as I understand, Idle doesn’t interpret any terminal control codes apart from a plain \n for a new line. I know it doesn’t do a carriage return for \r either.

[AlSweigart]

For clarification, this happens on Windows 7.

[ned-deily]

It's helpful to keep in mind that IDLE is a Tk application written in Python and, as such, depends on Tk to do nearly everything associated with writing to displays and reading from keyboards and pointing devices. If you try outputting the same string using the Tk shell, wish, you'll see exactly the same behavior as you do in IDLE:

$ wish
% .text insert 1.0 "hello\b\b\b\b\bHELLO"
% pack .text

And you'll see the same Tk platform differences. With the native OS X Tk's, the backspace characters aren't displayed; with a Linux or OS X X11 Tk, empty box characters are displayed. That may also depend on which font is in use. In any case, the Tk text widget does not behave the same way in this regard as most terminal emulator windows do would do with backspace characters. So it's up to any Tk app to figure out how it wants to deal with them.

This issue has come up before for Tkinter in general: back in 2001, effbot suggested some code to search for and edit backspace runs in Tk text. Presumably something similar could be added to IDLE if there was general agreement that this was desirable (after verifying that it works on all of the Tk platforms that IDLE supports). See https://mail.python.org/pipermail/python-list/2001-December/085908.html

[willingc]

Ned, Thanks for the detailed example and confirming my gut instinct that Tk was the root cause of the differences seen between the IDLE's Python interactive shell (https://docs.python.org/3.4/library/idle.html) and the interactive interpreter invoked from the command line (https://docs.python.org/3.4/tutorial/interpreter.html#tut-invoking).

As an end user learning Python (such as the elementary education market), the current Standard Library documentation on IDLE guides me to the incorrect conclusion that the "Python shell window (aka interactive interpreter)" in IDLE would behave the same as invoking the interactive interpreter from the command line.

It seems reasonable to explicitly state in the Standard Library doc that:
"In rare cases, such as text handling with certain special characters (i.e. '\b' in a string), the IDLE's interactive Python shell may return a different response than the Python interactive interpreter invoked from the command line. This is due to IDLE's low level dependence on Tk (Tk itself is not part of Python; it is maintained at ActiveState. reference: first paragraph of https://docs.python.org/3.4/library/tkinter.html#module-tkinter)."

[terryjreedy]

Going back to msdos, there are graphic chars for all 256 bytes, including may single and double line box-drawing chars. Many In Idle, I see 5 solid white circles. In FireFox, there are 5 empty circles (on dark background, which are chr(9689). When I copy from FF back to Idle (3.4.2, Win7), there are 5 of each. I have no idea if the 9689s are on the site or added by FF.

Here is another difference.

>>> print('\x03')  # console
♥  # heart

>>> print('\x03')  # idle
�  # lower left single line corner in Idle, box on FF

Trying to match console-Idle(tk) print output for control chars even on Windows would be tough.
---

I have been planning to add a subsection of the doc that mentions known differences between console interpreter and Idle shell. The result of print() is one of them. Another print difference is that Idle displays many unicode chars that Windows replaces with boxes or ?s, depending on the codepage.

[terryjreedy]

I closed bpo-24572 as a duplicate of this. It is the same issue except for printing \r instead of \b. These issues are not about responses to keyboard actions when entering text into an Idle editor window. During entry, just about any cntl/alt/function/shift/key combination can be intercepted to and translated to arbitrary action. They are also not about REPL echo. For 3.4+ Win7, both console Python and Shell echo '\b\t\x08\x09' as '\x08\t\x08\t'. Carol's report suggest that the same is true on Mac also.

Both issues *are* about print(s, file=outfile), where s is the result of processing all args except 'file' and outfile defaults to sys.stdout. The print call is the same as outfile.write(s), so outfile (and sys.stdout) could be any object with a write method.

>>> print('hello\b\b\b\b\bHELLO')
hello�����HELLO
>>> import sys; sys.stdout.write('hello\b\b\b\b\bHELLO'+'\n')
hello�����HELLO
(I actually see the same circles as Al, but copy-paste does not seem to work as well for me.)

So both issues are about the effect of writing 'control chars', in particular \b and \r, to a file. Well, that depends on the file. Some possibilities are copy as is (StringIO), encode and copy (disk file, socket), ignore, display as one glyph, display as multiple chars, non-destructively backspace (like backspace on typewriters and printing terminals and left-arrow elsewhere), or destructively backspace (like backspace on as most (all?) screen keyboards). After non-destructive movement of the 'cursor' position, the possibilities for following graphical chars are overwrite (like typewriters), replace, and insert (the modes sometimes selected by the Insert key). Non-destructive backspace followed by overwrite (meaning 'HELLO' printed on top of 'hello') is the original meaning of 'backspace'.

Having said all this, I am sympathetic to the idea that there should be an option to have 'print(ascii_string)' in user code give the same result in the console and Idle. I think this would best be accomplished by a least-common-denominator SimpleTerm subclass of tkinter.Text put somewhere in the tkinter package. (This would be a new issue that should start on python-ideas list.) However, I would consider something Idle specific.

Does the following work the same on *nix and Mac as Windows?

>>> print('hello\rHELLO')
HELLO  # helloHELLO with Win7 Idle

Are there any control-chars (other than \n) that work might work in the consoles on all three systems and that should be included?

Carol, another difference between the Windows console and Idle is that tk and hence Idle support the entire BMP subset of unicode. This should also be mentioned in the doc.

[serhiy-storchaka]

Control characters are named control characters because they are control the output device. Different devices have different capabilities and reacts different on the same codes. Windows console, different sorts of Linux terminals and Tk text widget are different devices. Some prints funny characters, others not, some beeps, others not, some interprets particular flavor of ESC sequences, others not, some allows color and positioning, others not. Python can't unify the behavior of these devices without lost most of functionality as it can't unify the behavior of black-white matrix printer, graphical plotter and 24-bit color LCD monitor.

I would close this issue as not related to Python.

[terryjreedy]

Serhiy, thanks for the paraphrase of what I tried to say. This issue has already been changed to a doc issue to explain the input/output effects of Idle's way of running user code. I should have retitled before. Control codes are just one of the effects that have come up on the tracker, python-list, or stackoverflow. Misunderstanding had lead to user puzzlement and even undeserved insults directed as Python.

I propose to something like the following to the introductory section of the Idle doc, https://docs.python.org/3/library/idle.html#index-0, after the feature list.

---
When Idle runs user code from either the shell or an editor window, the result is nearly always be the same as running the same code directly with python in either interactive or batch mode. However, environmental differences can have visible effects. For instance, Idle import statements add numerous entries to sys.modules and a few module attributes to modules such as tkinter. The line `1import tkinter; tkinter.colorchoosernormally raises AttributeError but works when run with Idle because Idle has already imported tkinter.colorchooser. To detect whether code is running under Idle, runimport idlelib; idlelib.run`` within a try-except statement.

More important are the input/output differences. Python normally runs in a text console process with direct access to keyboard and screen. For code run with Idle, the keyboard and screen are controlled by the tk graphics subsystem and sys.stdin, sys.stdout, and sys.stderr are bound to objects that connect to the gui. (Note that print calls sys.stdout.write.) Here are some of the effects of keyboard and screen access being indirect.

• Operating system (OS) specific functions that directly access the keyboard may not work.
• The Idle shell works with complete statements rather than individual lines of code. One can edit and retrieve complete multiline statements instead of single lines.
• User code gets colorized. Normal output and error output to the shell get their own colors.
• Tk supports the Basic Multilingual Plane (BMP) subset of Unicode characters. This is worse than consoles that support supplementary planes (with appropriate fonts in use), but better than the Windows console, which only supports restricted subsets of the BMP, depending on the code page in use.
• Tk handling of ascii control chars depends on the OS and is usually different from the text console.
  ---