integer overflow in computing byte's object representation

[pkt]

BPO	22519
Nosy	@vstinner, @tiran
Files	poc_repr_bytes.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-24.19:06:12.557>
created_at = <Date 2014-09-29.21:03:05.579>
labels = ['type-bug']
title = "integer overflow in computing byte's object representation"
updated_at = <Date 2016-09-24.19:06:12.555>
user = 'https://bugs.python.org/pkt'

bugs.python.org fields:

activity = <Date 2016-09-24.19:06:12.555>
actor = 'christian.heimes'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-24.19:06:12.557>
closer = 'christian.heimes'
components = []
creation = <Date 2014-09-29.21:03:05.579>
creator = 'pkt'
dependencies = []
files = ['36755']
hgrepos = []
issue_num = 22519
keywords = []
message_count = 4.0
messages = ['227838', '227856', '227909', '277326']
nosy_count = 5.0
nosy_names = ['vstinner', 'christian.heimes', 'Arfrever', 'python-dev', 'pkt']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22519'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4', 'Python 3.5']

[pkt]

# PyBytes_Repr(PyObject *obj, int smartquotes)
# {
# PyBytesObject* op = (PyBytesObject*) obj;
# 1 Py_ssize_t i, length = Py_SIZE(op);
# size_t newsize, squotes, dquotes;
# ...
#
# /* Compute size of output string */
# newsize = 3; /* b'' */
# s = (unsigned char*)op->ob_sval;
# for (i = 0; i < length; i++) {
# ...
# default:
# if (s[i] < ' ' || s[i] >= 0x7f)
# 2 newsize += 4; /* \xHH */
# else
# newsize++;
# }
# }
# ...
# 3 if (newsize > (PY_SSIZE_T_MAX - sizeof(PyUnicodeObject) - 1)) {
# PyErr_SetString(PyExc_OverflowError,
# "bytes object is too large to make repr");
# return NULL;
# }
# 4 v = PyUnicode_New(newsize, 127);
# ...
# *p++ = 'b', *p++ = quote;
# for (i = 0; i < length; i++) {
# ...
# 5 *p++ = c;
# }
# *p++ = quote;
# 6 assert(_PyUnicode_CheckConsistency(v, 1));
# return v;
# }
#
# 1. length=2^30+1=1073741825
# 2. newsize=length*4+3=7 (overflow)
# 3. check is inefficient, because newsize=7
# 4. allocated buffer is too small
# 5. buffer overwrite
# 6. this assert will likely fail, since there is a good chance the allocated
# buffer is just before the huge one, so the huge one will overwrite itself.

[python-dev]

New changeset d9cd11eda152 by Benjamin Peterson in branch '2.7':
fix overflow checking in PyString_Repr (closes bpo-22519)
https://hg.python.org/cpython/rev/d9cd11eda152

New changeset f5c662a7f7e6 by Benjamin Peterson in branch '3.3':
fix overflow checking in PyBytes_Repr (closes bpo-22519)
https://hg.python.org/cpython/rev/f5c662a7f7e6

New changeset ed31cdf11ac2 by Benjamin Peterson in branch '3.4':
merge 3.3 (closes bpo-22519)
https://hg.python.org/cpython/rev/ed31cdf11ac2

New changeset 0ddc5fc5f395 by Benjamin Peterson in branch 'default':
merge 3.4 (bpo-22519)
https://hg.python.org/cpython/rev/0ddc5fc5f395

[vstinner]

It would be nice to have a "bigmem" test checking that repr(b'\x00'*(2**30+1)) doesn't crash anymore.

[tiran]

The code doesn't crash any more. It took me more than 5 GB of resident memory and about 90 CPU seconds to reproduce the circumstances of the overflow.