IDLE Doc: Text consumes unlimited RAM, consoles likely not

[jbrearley]

BPO	33000
Nosy	@terryjreedy, @serwy
PRs	bpo-33000: Document that IDLE's shell has no line limit. #10373 [3.7] bpo-33000: Document that IDLE's shell has no line limit. (GH-10373) #10374 [3.6] bpo-33000: Document that IDLE's shell has no line limit. (GH-10373) #10375
Files	Python_IDLEX.png

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2018-11-07.06:05:44.835>
created_at = <Date 2018-03-05.16:30:03.196>
labels = ['3.8', 'expert-IDLE', 'docs', '3.7', 'performance']
title = 'IDLE Doc: Text consumes unlimited RAM, consoles likely not'
updated_at = <Date 2018-11-07.06:05:44.808>
user = 'https://bugs.python.org/jbrearley'

bugs.python.org fields:

activity = <Date 2018-11-07.06:05:44.808>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2018-11-07.06:05:44.835>
closer = 'terry.reedy'
components = ['Documentation', 'IDLE']
creation = <Date 2018-03-05.16:30:03.196>
creator = 'jbrearley'
dependencies = []
files = ['47471']
hgrepos = []
issue_num = 33000
keywords = ['patch']
message_count = 15.0
messages = ['313263', '313272', '313327', '313348', '313364', '313394', '315806', '315810', '315821', '329094', '329111', '329146', '329403', '329404', '329405']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'roger.serwy', 'jbrearley', 'Big Stone']
pr_nums = ['10373', '10374', '10375']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue33000'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[jbrearley]

While running a tensorflow script in the IDLEX GUI that runs for 8 million steps and produce 2 lines stdout per step, my PC used all 16GB RAM and crashed the python process, not to mention messed up other apps, like Firefox & Norton AntiVirus. While the RAM was recovered, Firefox started responding, but Norton Antivirus didn’t, so the PC had to be rebooted.

The issue is easily reproduced with the short print loop that dumps 20K lines of stdout, at 171 characters / line on the IDLEX GUI window. When the script is run in the IDLEX GUI, the Windows Task Manager shows the python process start at 19MB RAM consumption, then grows to 569MB RAM consumption. If I run the script a second time in the same IDLEX GUI window, it grows to 1.1GB RAM consumption.

So 20K lines off output at 171 characters / line (“i: nnnnn” prefix + 2 * 80 byte string + newline) = 3.4M total characters stored in the scrollback buffer. The delta memory consumed was 569MB – 19MB = 550MB. The RAM consumed / character is 550MB / 3.4M = 161 bytes / character. This seems excessively inefficient.

I now understand how the tensorflow script would stop after 550K iterations and the 550K lines of stdout in the IDLEX GUI would consume all 16GB RAM on my PC.

BTW, when I run the same test script in the WinPython command prompt window, it only consumes 4MB RAM while it runs. However the scrollback buffer is limited to 10K lines, wrapped at the 80 character mark, so much less data saved.

I haven’t found any options in IDLEX GUI to limit the scrollback buffer size.

My request is to review the scrollback memory storage algorithms. If nothing can be done to improve them, then please add a circular buffer to limit the memory consumption.

# Print loop to test memory consumption in Python IDLEX GUI.
s1 = "0123456789"
s2 = s1+s1+s1+s1+s1+s1+s1+s1
for i in range(20000):
print("i:", i, s2, s2)

I am using Python 3.6.4 on Windows 7 PC, Intel i7-4770S, 3.1GHz, 16GB RAM.

[terryjreedy]

IDLEX is an independently developed and distributed set of IDLE eXtensions. We have nothing to do with it.

IDLE, as the name suggests and the doc spells out, is an Integrated Development and Learning Environment. It is not a production run environment. For interactive development, one can close the shell window and reclaim its memory while keeping an editor window open.

16 million lines of screen output way outside of IDLE's intended use. Why are you (mis)using it this way?

As you already discovered, an effectively infinite output stream can be sent to a console. Command Prompt has a similar behavior. If you want more saved, output to disk.

Or write your own gui for your simulations. IDLE (and hence IDLEX) uses the tkinter wrapper of the tcl/tk GUI framework. tcl/tk is also separately developed and distributed. We just use it. Its text widget does not come with any automatic size management. But you could write your own insert wrapper that would delete a line for each line added after you deem the widget 'full'.

When I ran your sample with 64 bit 3.7.0b2 on Win10, the memory reported by Task Manager increased from 30 to 195 Mb or 165 MB net, much less than you report. I once printed about 500,000 60 char lines. I had either 12 or 24 GB memory at the time.

I am inclined to close this issue as 'third party' (IDLEX and tcl/tk) but I will let you respond first.

[jbrearley]

Hi Terry: I am exploring the value of a language specific editor and runtime environment. Its definitely a large step up from Windows Notepad and Gnome Gedit. Perhaps some notes in the IDLEX documentation regarding the development versus production runtime usages would be in order?

Perhaps in your Win10 environment you get the memory back when you close the shell window. In Win7, you dont get the memory back until you close both windows.

[terryjreedy]

I did not close either window before looking at the Task Manager number.

Once your program is debugged, I cannot think of any advantage of running it through IDLE instead of directly with the Python interpreter.

Since you repeat 'IDLEX' I still do not know whether you are running IDLEX or IDLE itself.

[jbrearley]

Hi Terry: The icon on my Win 7 desktop points to: "C:\WinPython\IDLEX (Python GUI).exe".

This was part of the 430MB installer file WinPython-64bit-3.6.4.0Qt5b4.exe from https://sourceforge.net/projects/winpython.

I attached a screen shot of IDLEX window & help about.

[terryjreedy]

I don't know whether memory difference between us is due to IDLEX or Windows version or something else, but I don't really care. The IDLE doc has a section "IDLE-console differences". It should gain a paragraph explain the difference between memory-limited tk.Text-based Shell and limited-lines consoles. For Command Prompt, the buffer defaults to a pathetic 300 lines, which is not enough for the CPython test suite output. The max is 9999 for those who discover how to enlarge it. No limit is an advantage for most uses, but not for one who fills up memory.

I will also think about a sentence to add to the introductory section about development versus production runs.

[BigStone]

WinPython-64bit-3.6.4.0Qt5b4 doesn't in fact includes IDLEX: IDLEX is not compatible with Python-3.6+

The "IDLEX" icon of WinPython switches back to "normal" IDLE, when IDLEX is not detected.

As WinPython-3.5 is being deprecated, the icon should be renamed IDLE.

(see also https://bugs.python.org/issue17535 quest)

[terryjreedy]

Big Stone: thank you for the clarification that IDLEX, and its patching of idlelib, is not involved. WinPython is an independent augmented distribution. Your suggestion about its IDLEX icon should be sent to its authors.