raw_input + readline: Ctrl+C during search breaks readline

[hartwork]

BPO	24266
Nosy	@Yhg1s, @terryjreedy, @pitrou, @bitdancer, @vadmium, @serhiy-storchaka, @tacaswell, @hartwork
Files	raw_input__workaround_demo.py raw_input__minimal.py readline-cancel.patch: Workaround in readline module readline-sigcleanup.patch: Use new Readline 7 function

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-03-22.21:47:14.534>
created_at = <Date 2015-05-22.14:58:03.813>
labels = ['interpreter-core', 'type-bug']
title = 'raw_input + readline: Ctrl+C during search breaks readline'
updated_at = <Date 2016-03-23.18:07:26.859>
user = 'https://github.com/hartwork'

bugs.python.org fields:

activity = <Date 2016-03-23.18:07:26.859>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2016-03-22.21:47:14.534>
closer = 'martin.panter'
components = ['Interpreter Core']
creation = <Date 2015-05-22.14:58:03.813>
creator = 'sping'
dependencies = []
files = ['39467', '39468', '39514', '40054']
hgrepos = []
issue_num = 24266
keywords = ['patch']
message_count = 17.0
messages = ['243828', '243844', '243846', '243864', '243975', '244146', '244680', '244682', '244683', '247571', '259311', '259313', '259314', '262167', '262176', '262213', '262294']
nosy_count = 10.0
nosy_names = ['twouters', 'terry.reedy', 'pitrou', 'Arfrever', 'r.david.murray', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'tcaswell', 'sping']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24266'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[hartwork]

Hi!

A college of mine ran into a bug with raw_input.
We have a shell derived from stdlib module "cmd" here but the bug shows
with plain raw_input, as well (see demo code).

For the symptoms: the shell is executing commands from history that the
user explicitly chose not to run. Since our shell does things like
deallocation of LVM volumes, that behavior is rather troublesome :)

How does it happen?

• When pressing Ctrl+R, an incremental reverse history search is started.
  During that search, you normalled press Escape/Ctrl+J/Ctrl+G to end
  search or Return to pick a result.

• When you press Ctrl+C though, the next call to the raw_input function
  believes to be in search mode still (while not showing "(reverse-i-search)"
  or indicating search some way).

• Now when (entering some text and) pressing return now, the string last
  shown during incremental search is being return from raw_input
  (and executed in context of the cmd module).

I have re-produced the issue with Python 2.7.3, 2.7.9, 3.2.3, 3.4.2.

For a workaround, one can handle KeyboardInterrupt and internal adjust
variable rl_readline_state of the C readline library.
I'm attaching (a minimal bug demo and) the ctypes based workaround that
works well over here.
(The workaround demo also shows that readline state is not fully reset
when Ctrl+C was pressed outside of search mode, since flag RL_STATE_DONE
is not set after.)

It would rock the house if this could be fixed in Python.

I'm looking forward to your reply.

Best,

Sebastian

[terryjreedy]

3.2 and 3.3 only get security fixes. I condensed the title to be visible in display boxes. It would help if you signed the PSF Contributor Agreement
https://www.python.org/psf/contrib/
https://www.python.org/psf/contrib/contrib-form/
and contributed a usable patch for test_readline under that agreement.

[serhiy-storchaka]

This behavior annoyed me long time, but I had no idea how to fix this.

[vadmium]

Also applies to the interactive interpreter mode.

[vadmium]

I suspect this is actually a bug with Gnu Readline. The same issue exists in GDB.

When a signal handler (such as SIGINT) raises an exception to abort the input, Python calls rl_free_line_state(). The documentation for that function, <https://cnswww.cns.cwru.edu/php/chet/readline/readline.html#IDX341\> says Readline’s own SIGINT handler (which Python doesn’t normally actually use) also calls it to abort the current line. It lists various states that are “freed”. Although the search state is not in the list, it seems like it should be.

[vadmium]

I reported this to Gnu Readline: <https://lists.gnu.org/archive/html/bug-readline/2015-05/msg00014.html\>. So far it sounds like there is no general solution. I am posting a patch implementing Sebastian’s workaround. But my personal philosophy is to leave Python alone if it is not doing anything wrong and this is indeed a Readline bug.

Chet also pointed out that there are other modes in addition to incremental search that are not cancelled. For instance, the “argument” mode where you press Alt+<number> etc.

[terryjreedy]

The patch is two lines, but I do not know our general policy for working around bugs in external modules. I do know that Idle has workarounds for tk bugs (or 'os-dependencies').

[bitdancer]

I believe that in general we do do workarounds if they don't make the code too complicated and upstream hasn't solved the problem (or we need to support older upstream versions).

[hartwork]

I guess supporting older upstream versions would be nice in this case.

[vadmium]

Readline 7.0 alpha version includes a new “feature” that should help:

i. rl_callback_sigcleanup: a new application function that can clean up and unset any state set by readline’s callback mode. Intended to be used after a signal.

Patch readline-sigcleanup.patch uses this function, depending on a compile-time version check. It fixes the bug when compiled against Readline 7.

[tacaswell]

I do not think that readline-cancel.patch is sufficient. The clean up function that readline uses internally (http://git.savannah.gnu.org/cgit/readline.git/tree/isearch.c#n720 ) also cleans up the context that used by isearch.

The functions to cleanup the context (and a pointer to the context) are exposed on rlprivate.h so python can get at them to support old versions of rl. Once started down that path for isearch mode, might as well vendor all of rl_callback_sigcleanup() which will end up being 50-100 LoC (some of the helper functions will also need to be vendored).

[vadmium]

I think it might be best to wait until we are sure that the new Readline 7 API is stable (Is already the case?), push something like readline-sigcleanup.patch, and don’t bother with hacks for Readline 6 (or Editline etc).