integer overflow in _json.encode_basestring_ascii

[pkt]

BPO	23369
Nosy	@serhiy-storchaka
Files	poc_ascii_escape.py test_encode_basestring_ascii_overflow.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-02-02.22:47:38.834>
created_at = <Date 2015-02-01.13:59:35.874>
labels = ['type-crash']
title = 'integer overflow in _json.encode_basestring_ascii'
updated_at = <Date 2016-08-13.23:48:41.102>
user = 'https://bugs.python.org/pkt'

bugs.python.org fields:

activity = <Date 2016-08-13.23:48:41.102>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2015-02-02.22:47:38.834>
closer = 'python-dev'
components = []
creation = <Date 2015-02-01.13:59:35.874>
creator = 'pkt'
dependencies = []
files = ['37968', '37978']
hgrepos = []
issue_num = 23369
keywords = ['patch']
message_count = 6.0
messages = ['235177', '235213', '235255', '235298', '272156', '272623']
nosy_count = 4.0
nosy_names = ['Arfrever', 'python-dev', 'serhiy.storchaka', 'tehybel_']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue23369'
versions = ['Python 3.3', 'Python 3.4', 'Python 3.5']

[pkt]

# static PyObject *
# ascii_escape_unicode(PyObject *pystr)
# {
# ...
#
# input_chars = PyUnicode_GET_LENGTH(pystr);
# input = PyUnicode_DATA(pystr);
# kind = PyUnicode_KIND(pystr);
#
# /* Compute the output size */
# for (i = 0, output_size = 2; i < input_chars; i++) {
# Py_UCS4 c = PyUnicode_READ(kind, input, i);
# if (S_CHAR(c))
# output_size++;
# else {
# switch(c) {
# ...
# default:
# 1 output_size += c >= 0x10000 ? 12 : 6;
# ...
#
# 2 rval = PyUnicode_New(output_size, 127);
#
# 1. if c is \uFFFF then output_size += 6. There are no overflow checks on this
# variable, so we can overflow it with a sufficiently long (2**32/6+1 chars)
# string
# 2. rval buffer is too small to hold the result
#
# Crash:
# ------
#
# Breakpoint 3, ascii_escape_unicode (pystr='...') at /home/p/Python-3.4.1/Modules/_json.c:198
# 198 rval = PyUnicode_New(output_size, 127);
# (gdb) print output_size
# $9 = 4
# (gdb) c
# Continuing.
#
# Program received signal SIGSEGV, Segmentation fault.
# 0x4057888f in ascii_escape_unichar (c=65535,
# output=0x40572358 "...",
# chars=19624) at /home/p/Python-3.4.1/Modules/_json.c:155
# 155 output[chars++] = Py_hexdigits[(c >> 8) & 0xf];
#
# OS info
# -------

# 
# % ./python -V
# Python 3.4.1
#  
# % uname -a
# Linux ubuntu 3.8.0-29-generic #42~precise1-Ubuntu SMP Wed Aug 14 15:31:16 UTC 2013 i686 i686 i386 GNU/Linux
#  
 
from _json import encode_basestring_ascii as enc
s="\uffff"*int((2**32)/6+1)
enc(s)

[python-dev]

New changeset 8699b3085db3 by Benjamin Peterson in branch '3.3':
fix possible overflow in encode_basestring_ascii (closes bpo-23369)
https://hg.python.org/cpython/rev/8699b3085db3

New changeset 4f47509d7417 by Benjamin Peterson in branch '3.4':
merge 3.3 (bpo-23369)
https://hg.python.org/cpython/rev/4f47509d7417

New changeset 02aeca4974ac by Benjamin Peterson in branch 'default':
merge 3.4 (bpo-23369)
https://hg.python.org/cpython/rev/02aeca4974ac

[serhiy-storchaka]

"\uffff"*((2**32)//6 + 1) is calculated at compile time. This requires much memory and can cause swapping. May be this was a cause of failing tests on some buildbots:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/2623/steps/test/logs/stdio
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.4/builds/749/steps/test/logs/stdio

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/test/__main__.py", line 3, in <module>
    regrtest.main_in_temp_cwd()
  File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/test/regrtest.py", line 1564, in main_in_temp_cwd
    main()
  File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/test/regrtest.py", line 738, in main
    raise Exception("Child error on {}: {}".format(test, result[1]))
Exception: Child error on test_json: Exit code -9
*** [buildbottest] Error code 1

At least my computer hanged on first run of this test.

To prevent computing this string constant at compile time you can use a variable. And '\x00' can be used instead of '\uffff', it needs less memory.

[python-dev]

New changeset 5c730d30ffbc by Benjamin Peterson in branch '3.3':
reduce memory usage of test (closes bpo-23369)
https://hg.python.org/cpython/rev/5c730d30ffbc

[tehybel]

I noticed that this is only fixed for python 3.3 and 3.4, not for 2.7. Is that intentional? If so, why?

[python-dev]

New changeset 6fa0ebfdc136 by Benjamin Peterson in branch '2.7':
fix possible overflow in encode_basestring_ascii (bpo-23369)
https://hg.python.org/cpython/rev/6fa0ebfdc136