IDLE 3.x can crash decoding recent file list

[naguilera]

BPO	10974
Nosy	@birkenfeld, @ned-deily
Files	issue10974.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2011-01-24.22:28:25.499>
created_at = <Date 2011-01-21.18:40:39.413>
labels = ['expert-IDLE', 'type-crash']
title = 'IDLE 3.x can crash decoding recent file list'
updated_at = <Date 2011-01-24.22:28:25.498>
user = 'https://bugs.python.org/naguilera'

bugs.python.org fields:

activity = <Date 2011-01-24.22:28:25.498>
actor = 'ned.deily'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2011-01-24.22:28:25.499>
closer = 'ned.deily'
components = ['IDLE']
creation = <Date 2011-01-21.18:40:39.413>
creator = 'naguilera'
dependencies = []
files = ['20500']
hgrepos = []
issue_num = 10974
keywords = ['patch']
message_count = 6.0
messages = ['126764', '126821', '126828', '126885', '126914', '126963']
nosy_count = 3.0
nosy_names = ['georg.brandl', 'ned.deily', 'naguilera']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue10974'
versions = ['Python 3.1', 'Python 3.2']

[naguilera]

Here is the report from "Console" (date/time removed):

[0x0-0x1a11a1].org.python.IDLE[5541]	Traceback (most recent call last):
[0x0-0x1a11a1].org.python.IDLE[5541]	  File "/Applications/Python 3.2/IDLE.app/Contents/Resources/idlemain.py", line 73, in <module>
[0x0-0x1a11a1].org.python.IDLE[5541]	    main()
[0x0-0x1a11a1].org.python.IDLE[5541]	  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/PyShell.py", line 1388, in main
[0x0-0x1a11a1].org.python.IDLE[5541]	    shell = flist.open_shell()
[0x0-0x1a11a1].org.python.IDLE[5541]	  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/PyShell.py", line 277, in open_shell
[0x0-0x1a11a1].org.python.IDLE[5541]	    self.pyshell = PyShell(self)
[0x0-0x1a11a1].org.python.IDLE[5541]	  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/PyShell.py", line 813, in __init__
[0x0-0x1a11a1].org.python.IDLE[5541]	    OutputWindow.__init__(self, flist, None, None)
[0x0-0x1a11a1].org.python.IDLE[5541]	  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/OutputWindow.py", line 16, in __init__
[0x0-0x1a11a1].org.python.IDLE[5541]	    EditorWindow.__init__(self, *args)
[0x0-0x1a11a1].org.python.IDLE[5541]	  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/EditorWindow.py", line 267, in __init__
[0x0-0x1a11a1].org.python.IDLE[5541]	    self.update_recent_files_list()
[0x0-0x1a11a1].org.python.IDLE[5541]	  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/EditorWindow.py", line 777, in update_recent_files_list
[0x0-0x1a11a1].org.python.IDLE[5541]	    rf_list = rf_list_file.readlines()
[0x0-0x1a11a1].org.python.IDLE[5541]	  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/encodings/ascii.py", line 26, in decode
[0x0-0x1a11a1].org.python.IDLE[5541]	    return codecs.ascii_decode(input, self.errors)[0]
[0x0-0x1a11a1].org.python.IDLE[5541]	UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 55: ordinal not in range(128)
com.apple.launchd.peruser.501[168]	([0x0-0x1a11a1].org.python.IDLE[5541]) Exited with exit code: 1

Thanks,

Néstor Aguilera

[ned-deily]

From the traceback, IDLE is crashing on a decode error when trying to process the list of recently used files which is used to populate its recent files menu. That list of files is saved in ~/.idlerc/recent-file.lst so an easy workaround is to delete the file before launching IDLE.app. Most likely, the list of files contains at least one path name with a non-ASCII character. IDLE appears to be dependent here on locale LANG settings. When launched by double-clicking, like all Mac GUI apps IDLE.app does not inherited environment variables set in shell profiles, as in a terminal session. The problem can be easily reproduced with bin/idle which is started from a shell environment:

$ echo $LANG
en_US.UTF-8
$ echo "/Users/nad/Documents/test_ä.py" > ~/.idlerc/recent-files.lst
$ /usr/local/bin/idle3.2
# with UTF-8 locale, idle starts normally with the umlauted file name in the recent files menu
$ unset LANG
$ /usr/local/bin/idle3.2
Traceback (most recent call last):
  File "/usr/local/bin/idle3.2", line 5, in <module>
    main()
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/PyShell.py", line 1388, in main
    shell = flist.open_shell()
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/PyShell.py", line 277, in open_shell
    self.pyshell = PyShell(self)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/PyShell.py", line 813, in __init__
    OutputWindow.__init__(self, flist, None, None)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/OutputWindow.py", line 16, in __init__
    EditorWindow.__init__(self, *args)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/EditorWindow.py", line 267, in __init__
    self.update_recent_files_list()
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/EditorWindow.py", line 777, in update_recent_files_list
    rf_list = rf_list_file.readlines()
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 26: ordinal not in range(128)

A similar exception occurs (without a crash) when saving a file to a non-ASCII file name and LANG is not properly set:

Exception in Tkinter callback
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/tkinter/__init__.py", line 1399, in __call__
    return self.func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/MultiCall.py", line 166, in handler
    r = l[i](event)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/IOBinding.py", line 328, in save
    self.save_as(event)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/IOBinding.py", line 350, in save_as
    self.updaterecentfileslist(filename)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/IOBinding.py", line 515, in updaterecentfileslist
    self.editwin.update_recent_files_list(filename)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/idlelib/EditorWindow.py", line 795, in update_recent_files_list
    rf_file.writelines(rf_list)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 16: ordinal not in range(128)

[naguilera]

Ned: thanks for the explanation.

Nestor

=======================================================================

On 22 Jan 2011, at 04:50, Ned Deily wrote:

Changes by Ned Deily <nad@acm.org>:

----------
Removed message: http://bugs.python.org/msg126820

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue10974\>

---

[naguilera]

On 22 Jan 2011, at 04:50, Ned Deily wrote:

[...]

A similar exception occurs (without a crash) when saving a file to a non-ASCII file name and LANG is not properly set:
[...]

Is there a way of telling tkinter to set the encoding to utf-8 (or whatever) before seeking recent-files.lst?

[ned-deily]

IDLE 3.x currently does not specify an explicit encoding when reading or writing the recent files list (~/.idlerc/recent-files.lst) and it defaults to "errors=strict". So IDLE 3.x is vulnerable to crashes if any of the recent files contain characters that can not be decoded using the LANG environment that IDLE is running under. That can happen if the user changes LANG or the recent-files.lst was updated by IDLE 2.x or, on OS X, when running as IDLE.app. The solution in the attached patch is to explicitly force the encoding of the file paths to utf_8 and to set "errors=replace".

Since this has the potential to crash IDLE, I propose it for inclusion in 3.2rc2.

[ned-deily]

Committed in py3k (r88174) for 3.2rc2, with release manager approval, and in 3.1 (r88177).