curses.unget_wch and test_curses fail when linked with ncurses 5.7 and earlier

[jcbollinger]

BPO	15037
Nosy	@loewis, @birkenfeld, @vstinner, @ned-deily, @Wooble, @serhiy-storchaka, @applio
PRs	bpo-15037: Add a workaround for getkey() in curses for ncurses 5.7 and earlier. #3826 [3.6] bpo-15037: Add a workaround for getkey() in curses for ncurses 5.7 and earlier. (GH-3826) #4218

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-11-01.17:23:58.215>
created_at = <Date 2012-06-08.16:08:32.022>
labels = ['3.7', 'type-bug', 'tests']
title = 'curses.unget_wch and test_curses fail when linked with ncurses 5.7 and earlier'
updated_at = <Date 2017-11-01.17:23:58.214>
user = 'https://bugs.python.org/jcbollinger'

bugs.python.org fields:

activity = <Date 2017-11-01.17:23:58.214>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2017-11-01.17:23:58.215>
closer = 'serhiy.storchaka'
components = ['Tests']
creation = <Date 2012-06-08.16:08:32.022>
creator = 'jcbollinger'
dependencies = []
files = []
hgrepos = []
issue_num = 15037
keywords = ['patch']
message_count = 20.0
messages = ['162532', '163320', '163330', '163334', '163336', '163361', '164097', '166027', '167105', '167563', '167564', '168555', '168586', '168977', '169025', '214583', '303318', '303342', '305383', '305384']
nosy_count = 9.0
nosy_names = ['loewis', 'georg.brandl', 'vstinner', 'ned.deily', 'geoffreyspear', 'python-dev', 'serhiy.storchaka', 'jcbollinger', 'davin']
pr_nums = ['3826', '4218']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue15037'
versions = ['Python 3.6', 'Python 3.7']

[jcbollinger]

I encountered this test failure while attempting to verify a patch for a separate issue, and I found that it occurs with the unmodified source on the default branch:

LD_LIBRARY_PATH="$PWD" ./python -bb -Wd -m test -r -w -uall -v test_curses
== CPython 3.3.0a4+ (default:4aeb5b9b62d7, Jun 8 2012, 10:23:35) [GCC 4.4.6
20110731 (Red Hat 4.4.6-3)]
==   Linux-2.6.32-220.4.1.el6.x86_64-x86_64-with-centos-6.2-Final
little-endian
==   /home/jbolling/cpython/build/test_python_26873
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0,
dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0,
verbose=0, bytes_warning=2, quiet=0, hash_randomization=1)
Using random seed 3072318
[1/1] test_curses
test test_curses crashed -- Traceback (most recent call last):
  File "/home/jbolling/cpython/Lib/test/regrtest.py", line 1237, in
runtest_inner
    test_runner()
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 338, in
test_main
    main(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 324, in main
    test_unget_wch(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 283, in
test_unget_wch
    read = chr(read)
OverflowError: signed integer is greater than maximum

1 test failed:
    test_curses
Re-running failed tests in verbose mode
Re-running test 'test_curses' in verbose mode
test test_curses crashed -- Traceback (most recent call last):
  File "/home/jbolling/cpython/Lib/test/regrtest.py", line 1237, in
runtest_inner
    test_runner()
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 338, in
test_main
    main(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 324, in main
    test_unget_wch(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 283, in
test_unget_wch
    read = chr(read)
OverflowError: signed integer is greater than maximum

[123272 refs]

Python was built and the tests run on CentOS 6.2 / x86_64, using the platform's standard tool chain, configured with "--enable-shared --with-threads --with-pydebug".

All other tests pass for me, although test_builtin failed when run as part of the whole suite but passed when run by itself.

For what it's worth, it looks like that particular message is emitted in exactly one place: Python/getargs.c:661 (function convertsimple()), which in this case I guess is being called indirectly from Python/bltinmodule.c:526 (function builtin_chr()). It's not obvious to me why that would be failing.

[ned-deily]

I just ran into this problem on another platform. I believe the problem is due to a sign-extension bug in the ncurses library unget_wch function (see link below). It was apparently fixed in nurses 5.8; I've tested with the current ncurses 5.9 and test_ncurses now runs without error.

Are you using a version of libncurses/libncursew older than 5.8? If so, can you try building with 5.8 or 5.9?

Another question is how to handle this for the 3.3.0 release. It looks like not all platforms have the latest ncurses and, if not, the unget_wch function and test_curses are likely to fail. Unfortunately, there doesn't seem to be an attribute available in the curses module that gives the version of the underlying curses/ncurses library. At the least, the problem should probably be documented somewhere.

http://invisible-island.net/ncurses/NEWS.html#t20091010

[jcbollinger]

The system on which I encountered the test failure uses ncurses 5.7, so that's consistent with the theory that the test is tickling an ncurses bug.

I'll have a look at testing with ncurses 5.8, but it is not available from RedHat or CentOS (and it never will be for the current and past versions of those systems), so that's not a good solution for most users.

On the other hand, it's not clear to me how serious is the bug revealed by the test failures, nor whether there is any viable workaround on the Python side.

[jcbollinger]

Clarification: "so that's not a good solution for most users" ... of RedHat-family distros, version 6.2 and earlier.

In fact, it looks like RedHat is sticking with its current version of ncurses for RHEL 6.3, too, so no help is coming from that direction any time soon.

[jcbollinger]

Ok, I confirm that the test passes after the system's ncurses library is upgraded to ncurses 5.8, and fails again when ncurses is downgraded back to version 5.7.

[ned-deily]

Thanks for the testing.

Georg, haypo: I think a call should be made on what, if anything, to do about this prior to 3.3.0-final. It seems that there are still OS distributions out there with older versions of ncurses. Is documenting this bug sufficient? If so, where?

[birkenfeld]

I agree it should be documented, probably wherever unget_wch is documented.

If there is a way to detect the ncurses version, the test should be skipped on < 5.8 as well.

[birkenfeld]

Documentation issue, can be resolved after b2.