IDLE can crash if file name contains non-BMP Unicode characters

[kamisky]

BPO	23672
Nosy	@terryjreedy, @serwy, @serhiy-storchaka
Files	issue23672(updated third).patch idlerun.png: IDLE's run window

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2015-08-17.01:39:25.132>
created_at = <Date 2015-03-15.08:51:42.479>
labels = ['expert-IDLE', 'type-bug']
title = 'IDLE can crash if file name contains non-BMP Unicode characters'
updated_at = <Date 2015-08-17.01:55:56.286>
user = 'https://bugs.python.org/kamisky'

bugs.python.org fields:

activity = <Date 2015-08-17.01:55:56.286>
actor = 'python-dev'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2015-08-17.01:39:25.132>
closer = 'terry.reedy'
components = ['IDLE']
creation = <Date 2015-03-15.08:51:42.479>
creator = 'kamisky'
dependencies = []
files = ['40098', '40143']
hgrepos = []
issue_num = 23672
keywords = ['patch']
message_count = 17.0
messages = ['238136', '238137', '238140', '238164', '238165', '247646', '247856', '247858', '247918', '248105', '248106', '248188', '248192', '248365', '248700', '248702', '248704']
nosy_count = 6.0
nosy_names = ['terry.reedy', 'roger.serwy', 'python-dev', 'serhiy.storchaka', 'kamisky', 'sanad']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23672'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']

[kamisky]

I could run the IDLE in the past time.But today,when I try and launch IDLE, the icon appears on the dock for a second and then disappears and the application doesn't run.Moreover,When I run IDLE3 in Terminal,it says:
bogon:~ Kamisky$ idle3
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/bin/idle3", line 5, in <module>
    main()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/idlelib/PyShell.py", line 1560, in main
    shell = flist.open_shell()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/idlelib/PyShell.py", line 315, in open_shell
    self.pyshell = PyShell(self)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/idlelib/PyShell.py", line 866, in __init__
    OutputWindow.__init__(self, flist, None, None)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/idlelib/OutputWindow.py", line 16, in __init__
    EditorWindow.__init__(self, *args)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/idlelib/EditorWindow.py", line 301, in __init__
    self.update_recent_files_list()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/idlelib/EditorWindow.py", line 927, in update_recent_files_list
    underline=0)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/tkinter/__init__.py", line 2719, in add_command
    self.add('command', cnf or kw)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/tkinter/__init__.py", line 2710, in add
    self._options(cnf, kw))
_tkinter.TclError: character U+1f393 is above the range (U+0000-U+FFFF) allowed by Tcl

Thanks.

[ned-deily]

Try removing IDLE's recently-used file list:

rm ~/.idlerc/recent-files.lst

[ned-deily]

At least on some platforms (e.g. OS X), it is easy to create files with legitimate names containing code points above the BMP limit (<= U+FFFF) currently imposed by Tcl/Tk. For IDLE 3, I suspect _filename_to_unicode() in EditorWindow could be modified to check for such cases to prevent problems if such file names end up in recent-files.lst. That might not be sufficient: there may be other problematic places. I also was able to crash a current IDLE 2.7 just opening a file with such a name.

[terryjreedy]

The full path of a file being edited also ends up in the title bar and the Window menu. I do not know whether the title bar is displayed by tk or the OS (Windows obviously displays the title of taskbar icons) but the Window list is definitely by tk. It seems to me that files need two names: the system name used to open (and re-open) a file (utf-8 bytes on *nix?) and a tk name (BMD unicode) for display in the various places.

The current limitation to BMP names is a limitation of the tcl/tk gui framework. I would like to add a workaround but do not consider its absence a bug. I am proposing on python-list the addition of some builtin means to replace non-BMP chars with \U000xxxxx escapes for display purposes. This would be generally useful for tkinter programming. The thread is "Add str.bmp() to only expand non-BMP chars, for tkinter use".

If this does not happen in 3.5, I would consider a patch to add a private function to EditorWindow.py to do the same. It would be far less efficient, but fast enough for short path names.

The EditorWindow.py line numbers are slight different from those in 3.4.3 (and 3.4.2, I believe), so I presume this is with 3.4.0 or 3.4.1. The result with 3.4.3 should be unchanged.

[terryjreedy]

Thinking more, there are two issue here. One is the fact that Idle stops when fed a filename with astral chars. This *is* a bug and should be fixed in all versions, even if that fix is to display a message box saying that Idle cannot work properly with such files.

The second is the one I addressed in the first message, the inability to work properly with files. Fixing that would obviate the need to display a message, but might be more work, especially on 2.7.

Kamisky, if you launch Idle and then try to open the file, do you see the name in the Open dialog? I presume that if you do, and select it, Idle would stop just as it did in your report.

[terryjreedy]

_filename_to_unicode returns unicode names as is. In 3.x, are filenames ever not unicode? They come from either a file dialog (via tk, hence unicode), or sys.argv. I do not know about the latter, which is possibly OS-dependent.

This function is used in three places in idlelib, all within EditorWindow:

1. def update_recent_files_list(self, new_file=None): ...
   ufile_name = self._filename_to_unicode(file_name)
2. def short_title(self): # reduce filename to basename
   return self._filename_to_unicode(filename)
3. def long_title(self):
   return self._filename_to_unicode(self.io.filename or "")

The _f2u output is not saved to disk or used to open files; it is display only. So replacing astral chars with either \Unnnnn escapes or the BMP box char should be fine.

1. The callback associated with each ufile_name encloses the original file_name, which is used to open the file. The original filename is also saved back to disk before the _f2u call.

2&3. The titles are display only. WindowList.py displays the long name for editor windows, but the callback is a wakeup function tied to the Window itself.

[sanad]

On the lines of changes proposed by Terry and others in the comments, this is the patch I'm submitting.

The function _filename_to_unicode() plays the major part in the plot. The function can get two types of filename ,'str' and 'bytes'. When its str, all the astral chars(characters that are outside BMP ) are replaced by the '�'(diamond question mark) character by performing a regex substitution in the return statement.

If the received filename is in the form of b'..' (bytes) or any other encoding ,it is first decoded(as was earlier) into str and before returning, all the out of range Unicode characters are replaced by the '�'(diamond question mark) character .

The effect on behavior is : the

1.IDLE is able to display correctly,the filename in the title-bar and in the file open dialog.
2. Any file with name that have astral chars in them are easily imported without any crash.

This is my first patch,please review it ,if any errors found I will correct them and upload again :)

[sanad]

Correction:

This patch fixes the problem of idle not opening when recent file list has filenames outside BMP and the crashing of idle on using filenames with astral characters.

The added benefit is that you can create a file with such chars and save it and access it again from the Recent Files List.

What this patch doesn't fix is :

1. The File Open Dialog displays the filename incorrectly.
2. The File Open Dialog doesn't allows the file with such filename to be opened.
3. When trying the to run the file with such name, it gives an Traceback to Tkinter. The following is the error message :

Exception in Tkinter callback
Traceback (most recent call last):
  File "/home/sanad/devpy/pessoc/cpython/Lib/tkinter/__init__.py", line 1549, in __call__
    return self.func(*args)
  File "/home/sanad/devpy/pessoc/cpython/Lib/idlelib/ScriptBinding.py", line 124, in run_module_event
    return self._run_module_event(event)
  File "/home/sanad/devpy/pessoc/cpython/Lib/idlelib/ScriptBinding.py", line 145, in _run_module_event
    interp.restart_subprocess(with_cwd=False, filename=code.co_filename)
  File "/home/sanad/devpy/pessoc/cpython/Lib/idlelib/PyShell.py", line 502, in restart_subprocess
    console.write("\n{0} {1} {0}".format(halfbar, tag))
  File "/home/sanad/devpy/pessoc/cpython/Lib/idlelib/PyShell.py", line 1294, in write
    'Non-BMP character not supported in Tk')
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 49-49: Non-BMP character not supported in Tk

Which I guess is another related issue.