Importing readline produces erroneous output

[bkabrda]

BPO	19884
Nosy	@jcea, @pitrou, @vstinner, @benjaminp, @ned-deily, @davidmalcolm, @Wooble, @berkerpeksag, @vadmium, @vajrasky
Files	readline_disable_meta_key.patch meta-osx.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-08-27.07:16:55.839>
created_at = <Date 2013-12-04.12:06:16.426>
labels = ['extension-modules']
title = 'Importing readline produces erroneous output'
updated_at = <Date 2016-08-27.07:16:55.838>
user = 'https://bugs.python.org/bkabrda'

bugs.python.org fields:

activity = <Date 2016-08-27.07:16:55.838>
actor = 'martin.panter'
assignee = 'none'
closed = True
closed_date = <Date 2016-08-27.07:16:55.839>
closer = 'martin.panter'
components = ['Extension Modules']
creation = <Date 2013-12-04.12:06:16.426>
creator = 'bkabrda'
dependencies = []
files = ['35589', '43448']
hgrepos = []
issue_num = 19884
keywords = ['patch']
message_count = 36.0
messages = ['205217', '205221', '205228', '205293', '218275', '220336', '223823', '223824', '223864', '223865', '223873', '223879', '223889', '224012', '224484', '225008', '225593', '226551', '226558', '230345', '230622', '230624', '230676', '236724', '236734', '237154', '237155', '251656', '252096', '268780', '268781', '273587', '273596', '273686', '273757', '273764']
nosy_count = 21.0
nosy_names = ['jcea', 'pitrou', 'vstinner', 'benjamin.peterson', 'ned.deily', 'Arfrever', 'dmalcolm', 'geoffreyspear', 'python-dev', 'rpointel', 'berker.peksag', 'martin.panter', 'bkabrda', 'opoplawski', 'David.Edelsohn', 'vajrasky', 'Edd.Barrett', 'ischwabacher', 'vlee', 'Maxime Belanger', 'iyogeshjoshi']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue19884'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']

[bkabrda]

A simple reproducer:

python -c 'import readline' | xxd
0000000: 1b5b 3f31 3033 3468 .[?1034h

This was reported at [1] and originally at [2]. The readline maintainer suggests [3] using:

rl_variable_bind ("enable-meta-key", "off");

which was introduced in readline 6.1. Do you think it'd be safe to add the above line?

Thanks.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=880393
[2] https://bugzilla.redhat.com/show_bug.cgi?id=593799
[3] http://lists.gnu.org/archive/html/bug-readline/2011-04/msg00009.html

[pitrou]

I can't reproduce under Ubuntu 13.10. Is this Red Hat-specific?

(according to Dave, """This is a readline bug; it looks like it should not emit those characters when stdout is not a tty""")

[vajrasky]

Reproducible under Fedora 18.

$ ./python -c "import readline" | hexdump -C
00000000  1b 5b 3f 31 30 33 34 68                           |.[?1034h|
00000008

$ TERM=dumb ./python -c "import readline" | hexdump -C

[bkabrda]

I can also reproduce it on Arch Linux. It seems that the bad characters are only output if env variable TERM starts with "xterm".

[pitrou]

We seem to have gotten bit by this in bpo-21425.

Dave, is it possible for you to validate the proposed fix? I can't test this under Ubuntu.

Setting priority to "high" as it seems to actually annoy many people:

http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import
http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html
http://lists.gnu.org/archive/html/bug-readline/2007-08/msg00004.html

[vstinner]

Attached readline_disable_meta_key.patch: Implement the workaround suggested in (*), but only use the workaround if stdout is not a TTY (ex: output redirected), to limit the risk of regression.

(*) http://lists.gnu.org/archive/html/bug-readline/2011-04/msg00009.html

Extract of the patch:

+ if (!isatty(STDOUT_FILENO)) {
+ /* Issue bpo-19884: Don't try to enable any meta modifier key the terminal
+ claims to support when it is called. On many terminals (ex:
+ xterm-256color), the meta key is used to send eight-bit characters
+ (ANSI sequence "\033[1034h"). */
+ rl_variable_bind ("enable-meta-key", "off");
+ }

This issue becomes very annoying on my Fedora 20. The output of any Mercurial command now starts with "\033.[?1034h" (Mercurial uses Python 2.7). Example:

haypo@smithers$ hg root|hexdump -C
0000000 1b 5b 3f 31 30 33 34 68 2f 68 6f 6d 65 2f 68 61 |.[?1034h/home/ha|
00000010 79 70 6f 2f 70 72 6f 67 2f 70 79 74 68 6f 6e 2f |ypo/prog/python/|
00000020 64 65 66 61 75 6c 74 0a |default.|
00000028

Fedora 18 changed the default TERM environment variable to "xterm-256color":
http://fedoraproject.org/wiki/Features/256_Color_Terminals

Workaround in your application (to run on unpatched Python): set the TERM environment variable to "dummy", or unset this variable.

[python-dev]

New changeset 0177d8a4e82a by Victor Stinner in branch '2.7':
Issue bpo-19884: readline: Disable the meta modifier key if stdout is not a
http://hg.python.org/cpython/rev/0177d8a4e82a

New changeset 6303266beb80 by Victor Stinner in branch '3.4':
Issue bpo-19884: readline: Disable the meta modifier key if stdout is not a
http://hg.python.org/cpython/rev/6303266beb80

New changeset f85a968f9e01 by Victor Stinner in branch 'default':
(Merge 3.4) Issue bpo-19884: readline: Disable the meta modifier key if stdout is
http://hg.python.org/cpython/rev/f85a968f9e01

[vstinner]

I commited my patch.

[vstinner]

The test fails on AMD64 OpenIndiana 2.7:

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%202.7/builds/2354/steps/test/logs/stdio

test test_readline failed -- Traceback (most recent call last):
  File "/export/home/buildbot/64bits/2.7.cea-indiana-amd64/build/Lib/test/test_readline.py", line 52, in test_init
    self.assertEqual(stdout, b'')
AssertionError: '\x1b[?1034h' != ''

[ned-deily]

The changes are also causing segfaults when readline is linked with BSD libedit (the default on OS X) rather than GNU readline:

======================================================================
FAIL: test_init (test.test_readline.TestReadline)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/py/dev/3x/source/Lib/test/test_readline.py", line 52, in test_init
    TERM='xterm-256color')
  File "/py/dev/3x/source/Lib/test/script_helper.py", line 69, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/py/dev/3x/source/Lib/test/script_helper.py", line 55, in _assert_python
    "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is -11, stderr follows:
Fatal Python error: Segmentation fault

Current thread 0x00007fff75489310 (most recent call first):
File "<frozen importlib._bootstrap>", line 321 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1664 in load_module
File "<frozen importlib._bootstrap>", line 540 in _check_name_wrapper
File "<frozen importlib._bootstrap>", line 1110 in _load_backward_compatible
File "<frozen importlib._bootstrap>", line 1140 in _load_unlocked
File "<frozen importlib._bootstrap>", line 2175 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 2186 in _find_and_load
File "<string>", line 1 in <module>

[vstinner]

The changes are also causing segfaults when readline is linked with BSD libedit (the default on OS X) rather than GNU readline:

Oh wow. Do you have an idea of to fix the issue with libedit? Or make
the code condition, only use it with native readline?

[ned-deily]

Currently, readline.c uses #ifdef __APPLE__ to guard libedit-specific code (there is another open issue to generalize libedit support to other platforms).