readline: Wrong tab completion scope indices in Unicode terminals

[kunkku]

BPO	16182
Nosy	@malemburg, @loewis, @vstinner, @ezio-melotti, @vadmium, @serhiy-storchaka
Files	readline-wide-char-index.patch: Patch for adjusting the scope indices readline_locale.patch readline_locale.v2.patch readline_locale.v3.patch readline_locale.v4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-06-15.01:08:50.327>
created_at = <Date 2012-10-09.20:09:44.031>
labels = ['type-bug', 'library', 'expert-unicode']
title = 'readline: Wrong tab completion scope indices in Unicode terminals'
updated_at = <Date 2016-06-15.01:08:50.326>
user = 'https://bugs.python.org/kunkku'

bugs.python.org fields:

activity = <Date 2016-06-15.01:08:50.326>
actor = 'martin.panter'
assignee = 'none'
closed = True
closed_date = <Date 2016-06-15.01:08:50.327>
closer = 'martin.panter'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2012-10-09.20:09:44.031>
creator = 'kunkku'
dependencies = []
files = ['27508', '42952', '43068', '43098', '43359']
hgrepos = []
issue_num = 16182
keywords = ['patch']
message_count = 18.0
messages = ['172517', '223105', '266124', '266318', '266752', '266881', '267070', '268230', '268361', '268374', '268499', '268506', '268522', '268523', '268541', '268545', '268549', '268598']
nosy_count = 8.0
nosy_names = ['lemburg', 'loewis', 'vstinner', 'ezio.melotti', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'kunkku']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16182'
versions = ['Python 3.5', 'Python 3.6']

[kunkku]

Tab completion in the readline module does not seem to work well with Unicode terminals. The get_line_buffer function converts the line buffer to the str type (which are Unicode strings in Python 3), but the indices returned by get_begidx and get_endidx are not adjusted with respect to possible wide characters in the buffer, and hence are not very useful. The documentation is a bit vague on the index functions, but I think they should be relative to code points, regardless of the encoding used by the C library. The suggested correction is attached.

My second point of complaint is related to the use of PyUnicode_FromString in the module. The strings returned by the readline library use the current locale encoding, which is not necessarily UTF-8. I wonder if PyUnicode_DecodeLocale should be used instead for more portable code.

[BreamoreBoy]

@Kaarle please accept our apologies for the delay in getting back to you. Can one of our unicode gurus comment please.

[serhiy-storchaka]

Yes, the readline module is broken in Python 3. Underlying C library operates C strings and use locale-depended C functions to split it on Unicode characters. The Python wrapper always uses the UTF-8 encoding for converting between Python strings and C strings. It works only on UTF-8 locales. get_begidx() and get_endidx() don't correctly work at all for non-ASCII data. We should use locale encoding for converting.

Proposed patch makes the readline module to use locale depending coding functions instead of default UTF-8. It also corrects indices for get_begidx() and get_endidx().

[vadmium]

I’m a bit worried about flex_complete() covering up errors. If I add calls to PyErr_WriteUnraisable(), I can see both the s1 and s2 decodes failing. Input the following line:

>> "©"; import x;

and then move the cursor back one place, so it is directly after the “x”, but not after the semicolon (;). Then press Tab. Both errors are:

ValueError: embedded null byte

It looks like the reason is that PyUnicode_DecodeLocaleAndSize() requires that str[len] is a null character (the error message is misleading). It seems the len parameter is mainly there to verify that there are no embedded null characters, i.e. you cannot use it to give a truncated string.

It looks like Py_DecodeLocale() is used underneath; maybe it is simpler to call that directly. But it does not solve the string truncating problem.

A test case (perhaps using a pseudo-terminal) might also help pick this kind of thing up, if we can’t report errors any other way.

[vadmium]

Attached patch adds a possible test, and fixes the truncation problem I mentioned above.

I tried testing set_completer_delims() with a UTF-8 locale, but I suspect Gnu Readline does not support it. I called set_completer_delims("\xF6"), which encodes as C3 B6, but it seems to be breaking any UTF-8 sequence in half at a C3 byte. In other words, it is treating the delimiter list as a list of bytes, not code points. So I changed to an ASCII delimiter.

[vadmium]

V3 finishes what I started in v2:

• Changed unchecked PyBytes_AsString() → PyBytes_AS_STRING()
• Testing more functions for non-ASCII characters

I tried to test it with Editline on Linux (using my patch for bpo-13501). There seem to be many quirks with my version of Editline, some of which are not easy to work around:

• Initial CR swallowed when entering non-ASCII
• set_completion_display_matches_hook(), set_pre_input_hook(), set_completer_delims() all do nothing useful
• get_history_item() not updated straight after read_history_file()

I suspect Apple has patched their version of Editline, but if these quirks exist on Apple as well, it might be simplest to skip the test for Editline.

[serhiy-storchaka]

Thank you for your comments and updated patches, and especially for tests Martin. Added some comments on Rietveld.

[serhiy-storchaka]

Martin?

[vadmium]

I updated the patch to fix the error handling and memory leak. it also now skips the test in case the locale cannot encode the test data.

[serhiy-storchaka]

LGTM.

But the test fails with PYTHONIOENCODING=ascii.

$ PYTHONIOENCODING=ascii ./python -m test test_readline
Run tests sequentially
0:00:00 [1/1] test_readline
test test_readline failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_readline.py", line 194, in test_nonascii
    self.assertIn(b"result " + expected + b"\r\n", output)
AssertionError: b"result '[\\xefnserted]|t\\xebxt[after]'\r\n" not found in bytearray(b'^A^B^B^B^B^B^B^B\t\tx\t\r\n[\xc3\xafnserted]|t\xc3\xab[after]\x08\x08\x08\x08\x08\x08\x08text \'t\\xeb\'\r\nline \'[\\xefnserted]|t\\xeb[after]\'\r\nindexes 11 13\r\n\x07text \'t\\xeb\'\r\nline \'[\\xefnserted]|t\\xeb[after]\'\r\nindexes 11 13\r\nsubstitution \'t\\xeb\'\r\nmatches [\'t\\xebnt\', \'t\\xebxt\']\r\nx[after]\x08\x08\x08\x08\x08\x08\x08t[after]\x08\x08\x08\x08\x08\x08\x08\r\nTraceback (most recent call last):\r\n  File "<string>", line 39, in <module>\r\nUnicodeDecodeError: \'ascii\' codec can\'t decode byte 0xc3 in position 1: ordinal not in range(128)\r\n')

This is minor problem, since buildbots rarely configured with PYTHONIOENCODING=ascii.

[vadmium]

I get two other test suite failures if I set PYTHONIOENCODING, so I am not going to bother addressing this in test_readline :)

FAIL: test_forced_io_encoding (test.test_capi.EmbeddingTests)
FAIL: test_7 (test.test_pkg.TestPkg)

[python-dev]

New changeset 5122b3465a38 by Martin Panter in branch '3.5':
Issue bpo-16182: Fix readline begidx, endidx, and use locale encoding
https://hg.python.org/cpython/rev/5122b3465a38

New changeset 2ae2657d87a6 by Martin Panter in branch 'default':
Issue bpo-16182: Merge readline locale fix from 3.5
https://hg.python.org/cpython/rev/2ae2657d87a6