curses.ascii.isblank() function is broken. It confuses backspace (BS 0x08) with tab (0x09)

[kevinpt]

BPO	9770
Nosy	@4kir4, @serhiy-storchaka
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	curses_ascii_isblank_issue9770.patch: fix isblank, add tests for character classification functions from curses.ascii module curses-ascii-negative.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-12-28.08:08:42.885>
created_at = <Date 2010-09-04.00:01:48.509>
labels = ['3.7', 'type-bug', 'library']
title = 'curses.ascii.isblank() function is broken. It confuses backspace (BS 0x08) with tab (0x09)'
updated_at = <Date 2017-03-31.16:36:08.553>
user = 'https://bugs.python.org/kevinpt'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:08.553>
actor = 'dstufft'
assignee = 'none'
closed = True
closed_date = <Date 2016-12-28.08:08:42.885>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2010-09-04.00:01:48.509>
creator = 'kevinpt'
dependencies = []
files = ['35693', '45938']
hgrepos = []
issue_num = 9770
keywords = ['patch']
message_count = 6.0
messages = ['115544', '220852', '221008', '223833', '283484', '284161']
nosy_count = 4.0
nosy_names = ['akira', 'kevinpt', 'python-dev', 'serhiy.storchaka']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9770'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[kevinpt]

The isblank() function defined in curses.ascii is incorrect and doesn't match the output from C's isblank() from ctype.h

Incorrect definition:
def isblank(c): return _ctoi(c) in (8,32)

Should be:
def isblank(c): return _ctoi(c) in (9,32)

This most likely affects all versions of Python, not just 2.7.

[BreamoreBoy]

The problem and fix are simple but who is best placed to take a look at this?

[4kir4]

I've fixed isblank to accept tab instead of backspace and added tests
for character classification functions from curses.ascii module that
have corresponding analogs in ctype.h. They've uncovered issues in
isblank, iscntrl, and ispunct functions.

Open questions:

• is it a security bug (backspace is treated as tab in isblank())?
  If it is then 3.1, 3.2, 3.3 branches should also be updated
  [devguide]. If not then only 2.7, 3.4, and default branches should
  be changed.

  [devguide]: http://hg.python.org/devguide/file/9794412fa62d/devcycle.rst#l105

• iscntrl() mistakenly returns false for 0x7f but c11 defines it as
  a control character. Should iscntrl behavior (and its docs) be
  changed to confirm? Should another issue be opened?

• ispunct() mistakenly returns true for control characters such as
  '\n'. The documentation says (paraphrasing) 'any printable except
  space and alnum'. string.printable includes '\n' but 'printing
  character' in C11 does not include the newline. Moreover
  curses.ascii.isprint follows C behavior and excludes control
  characters. Should another issue be opened to return false from
  ispunct() for control characters such as '\n'?

• ispunct() mistakenly returns true for non-ascii characters such as
  0xff

• negative integer values: C functions are defined for EOF macros
  (some negative value) and the behavior is undefined for any other
  negative integer value. What should be curses.ascii.is* predicates
  behavior? Should Python guarantee that False is returned?

• curses.ascii.isspace/.isblank doesn't raise TypeError for bytes,
  None on Python 3

• should constants from string module be used? What is more
  fundamental: string.digits or curses.ascii.isdigit?

• no tests for: isascii, isctrl, ismeta (they are not defined in
  ctype.h). It is unclear what the behaviour should be e.g., isascii
  mistakenly returns True for negative ints, ismeta returns True for
  any non-ascii character including Unicode letters. It is not clear
  how isctrl is different from iscntrl.

[4kir4]

I've made the title more explicit: "curses.isblank function doesn't match
ctype.h" -> "curses.ascii.isblank() function is broken. It confuses
backspace (BS 0x08) with tab (0x09)"

If a core developer could review the open questions from the
previous message msg221008 then I could prepare a proper patch for the
issue.

[serhiy-storchaka]

Most issues was fixed in bpo-27079. Except handling negative integers. Following patch fixes the latter issue.

[python-dev]

New changeset cba619a7bf6a by Serhiy Storchaka in branch '3.5':
Issue bpo-9770: curses.ascii predicates now work correctly with negative integers.
https://hg.python.org/cpython/rev/cba619a7bf6a

New changeset 84ca252ac346 by Serhiy Storchaka in branch '2.7':
Issue bpo-9770: curses.ascii predicates now work correctly with negative integers.
https://hg.python.org/cpython/rev/84ca252ac346

New changeset eb81f2d2a42b by Serhiy Storchaka in branch '3.6':
Issue bpo-9770: curses.ascii predicates now work correctly with negative integers.
https://hg.python.org/cpython/rev/eb81f2d2a42b

New changeset 1c0b72996e60 by Serhiy Storchaka in branch 'default':
Issue bpo-9770: curses.ascii predicates now work correctly with negative integers.
https://hg.python.org/cpython/rev/1c0b72996e60