doctest pdb readline broken

[wosc]

BPO	5727
Nosy	@birkenfeld, @ncoghlan, @abalkin
Files	doctest-readline.patch readline.patch: svn diff for pdb.py and doctest.py (Python 2.6.4) pdbreadline.patch: svn diff of pdb, doctest and test_doctest pdb_doctest_readline.patch: svn diff of pdb, doctest and test_doctest

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-07-30.10:00:20.829>
created_at = <Date 2009-04-09.11:25:03.188>
labels = ['type-bug', 'library']
title = 'doctest pdb readline broken'
updated_at = <Date 2010-07-30.10:00:20.828>
user = 'https://bugs.python.org/wosc'

bugs.python.org fields:

activity = <Date 2010-07-30.10:00:20.828>
actor = 'georg.brandl'
assignee = 'none'
closed = True
closed_date = <Date 2010-07-30.10:00:20.829>
closer = 'georg.brandl'
components = ['Library (Lib)']
creation = <Date 2009-04-09.11:25:03.188>
creator = 'wosc'
dependencies = []
files = ['14193', '15681', '17109', '17204']
hgrepos = []
issue_num = 5727
keywords = ['patch']
message_count = 7.0
messages = ['85808', '88948', '92878', '96916', '104340', '104972', '112058']
nosy_count = 6.0
nosy_names = ['ping', 'georg.brandl', 'ncoghlan', 'belopolsky', 'sriram', 'wosc']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5727'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

[wosc]

When pdb is called from inside a doctest under python2.5, the readline
keys do not work anymore -- like they did just fine in 2.4.

Steps to reproduce: 
1. Create two files, foo.txt and foo.py, like so:
$ cat > foo.txt
>>> import pdb; pdb.set_trace()
$ cat > foo.py
import doctest; doctest.testfile('foo.txt')
2. run it
$ python2.5 foo.py
3. If I now press Ctrl-P, I get "^P" printed on the prompt, instead of
going back in the readline history. Likewise, the arrow keys print
escape sequences instead of moving the cursor.

[wosc]

I've tracked down the reason by diffing pdb.py and cmd.py between 2.4
and 2.5:
It turns out that pdb.Pdb in 2.5 changes the way it handles input
depending on whether an explicit output was provided, more precisely, it
disables readline in that case. I don't understand what's going on here,
but there is a simple, non-intrusive fix on the doctest side, see the
attached patch.

Unfortunately, I can't imagine how to write a test to check this behaviour.

[sriram]

Hi,

How about changing pdb's behavior, that it disables readline only if the
passed stream is not the stdout stream?

Also when looking at doctest module, I found that bdb's trace_dispatch
was overridden to set the debugger's output stream to something other
than stdout. I presume, that since prior to 2.5, pdb never took the
stdin and stdout arguments and that overridden method was needed to make
pdb use a stream other than stdout, but now that 2.5 allows us to pass
the output stream in init method of pdb, that function is no longer
needed.

Thanks

[sriram]

Hi,

This is the first bug am working in python, kindly excuse my mistakes,
if any.

As far as I can understand, the pdb disabled readline when an explicit
stdin or stdout is passed, to allow remote debugging.

I found this in Python 2.5.4 Release log.
"""
Patch bpo-721464: pdb.Pdb instances can now be given explicit stdin and
stdout arguments, making it possible to redirect input and output
for remote debugging.
"""

Now in doctest.py since we pass the stdout argument (which is always
sys.stdout) to pdb.py, readline is always disabled when pdb is invoked
from doctest.py.

One fix I can think of is to have pdb disable use of readline, not if
any output stream is passed but only if a output stream other than
sys.stdout is passed. I infact believe, this will preserve the
functionality of pdb.py that existed before release of version 2.5.4

The above fix would still not solve the problem because before we pass
the output stream to pdb.py, we override sys.stdout in doctest to
doctest's spoofout. So in pdb.py, sys.stdout will not point to real
sys.stdout. This can be fixed by overriding sys.stdout after we
initialize the debugger.

Please find the patch for doctest and pdb as an attachment.

Thanks
Sriram

[sriram]

Hi,

I believe this behaviour can be tested if we can prove that Cmd's cmdloop uses raw_input to get the data as against self.stdin.readline().

To test it, ideally I would have liked to override sys.stdin with a fake input stream and pass the list of args (like it's done in test_doctest.py). I can then pass the character codes for Up and Down Arrows in our fake stream and check if the raw_input can use readline and move through the fake input stream but that won't be possible because the C implementation of raw_input uses the readline functionality if only both stdin and stdout are from a terminal.

So alternatively, if we can just ensure that doctest's pdb (_OutputREdirectingPdb) has use_rawinput as 1, we can be assured that readline will be used.

I have attached the svn diff with trunk. Please review and comment

Thanks
Sriram

[sriram]

Hi,

On second thoughts, it made more sense to validate pdb directly instead of validating doctest's debugger.

I have also used few inputs, I got from irc chat at #python-dev room in writing the test case. Thanks to them.

I have attached the svn diff.

Please review them

Thanks
Sriram

[birkenfeld]

Fixed in r83271. I think the first patch's approach is better, since it does not affect pdb, only doctest's adaption of pdb. Consequently, I couldn't use the test case; thanks anyway for that!