scan_eol() Buffer Over-read

[JohnLeitch]

BPO	24989
Nosy	@vstinner, @larryhastings, @vadmium, @serhiy-storchaka
Files	scan_eol_Buffer_Over-read.patch: patch scan_eol_Buffer_Over-read.py: repro scan_eol_Buffer_Over-read_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-09-04.05:13:49.956>
created_at = <Date 2015-09-02.23:40:26.639>
labels = ['type-security', 'extension-modules', 'expert-IO', 'release-blocker']
title = 'scan_eol() Buffer Over-read'
updated_at = <Date 2015-09-08.18:34:24.218>
user = 'https://bugs.python.org/JohnLeitch'

bugs.python.org fields:

activity = <Date 2015-09-08.18:34:24.218>
actor = 'Arfrever'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-09-04.05:13:49.956>
closer = 'larry'
components = ['Extension Modules', 'IO']
creation = <Date 2015-09-02.23:40:26.639>
creator = 'JohnLeitch'
dependencies = []
files = ['40326', '40327', '40330']
hgrepos = []
issue_num = 24989
keywords = ['patch']
message_count = 11.0
messages = ['249584', '249600', '249602', '249606', '249610', '249617', '249699', '249711', '249717', '249718', '249738']
nosy_count = 8.0
nosy_names = ['vstinner', 'larry', 'Arfrever', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'JohnLeitch', 'brycedarling']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue24989'
versions = ['Python 3.5', 'Python 3.6']

[JohnLeitch]

Python 3.5 suffers from a vulnerability caused by the behavior of the scan_eol() function. When called, the function gets a line from the buffer of a BytesIO object by searching for a newline character starting at the position in the buffer.

However, if the position is set to a value that is larger than the buffer, this logic will result in a call to memchr that reads off the end of the buffer:

/* Move to the end of the line, up to the end of the string, s. */
start = PyBytes_AS_STRING(self->buf) + self->pos;
maxlen = self->string_size - self->pos;
if (len < 0 || len > maxlen)
    len = maxlen;

    if (len) {
        n = memchr(start, '\n', len);

In some applications, it may be possible to exploit this behavior to disclose the contents of adjacent memory. The buffer over-read can be observed by running the following script:

import tempfile
a = tempfile.SpooledTemporaryFile()
a.seek(0b1)
a.readlines()

Which, depending on the arrangement of memory, may produce an exception such as this:

0:000> g
(698.188): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=fff8a14c ebx=0a0a0a0a ecx=00000000 edx=05bb1000 esi=061211b0 edi=89090909
eip=61c6caf2 esp=010af8dc ebp=010af914 iopl=0 nv up ei ng nz ac po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010292
python35!memchr+0x62:
61c6caf2 8b0a mov ecx,dword ptr [edx] ds:002b:05bb1000=????????
0:000> k1
ChildEBP RetAddr
010af8e 61b640f1 python35!memchr+0x62 [f:\dd\vctools\crt_bld\SELF_X86\crt\src\INTEL\memchr.asm @ 125]

To fix this issue, it is recommended that scan_eol() be updated to check that the position is not greater than or equal to the size of the buffer. A proposed patch is attached.

[vadmium]

Simpler test case, which might find a place somewhere like /Lib/test/test_memoryio.py:

>>> from io import BytesIO
>>> b = BytesIO()
>>> b.seek(1)
1
>>> b.readlines()  # Should return an empty list
Segmentation fault (core dumped)
[Exit 139]

The patch looks like the right approach. Just watch out for the indentation (tabs vs spaces).

However I don’t see why we need the GET_BYTES_SIZE() macro or the (size_t) casting. Would it be okay comparing PyBytes_GET_SIZE() directly to self->pos? They are both meant to be Py_ssize_t.

The bug looks like it was introduced by Serhiy’s changes in bpo-15381.

[JohnLeitch]

We based our fix on the check in write_bytes:

    if (endpos > (size_t)PyBytes_GET_SIZE(self->buf)) {
        if (resize_buffer(self, endpos) < 0)
            return -1;
    }

I see now that our casting was extraneous. As for the macro, it was suspected that similar issues may be present and we wanted to write reusable code, but this also seems unnecessary now that it's known the cast is unneeded.

Early tomorrow I'll take some time to create a revised patch.

[serhiy-storchaka]

It should be self->string_size, not PyBytes_GET_SIZE(self->buf).

Thank you for your report and your patch John. Here is revised patch with tests based on Martin's test.

Larry, perhaps this bug is grave enough to be fixed in RC3.

[vstinner]

scan_eol_Buffer_Over-read_2.patch looks good to me.

Larry, perhaps this bug is grave enough to be fixed in RC3.

Since it looks like a regression in Python 3.5, yes, it's a severe bug. Please send a pull request to Larry for it.

[larryhastings]

Yes, please create a pull request for this patch. Thanks!

And just to confirm: I just applied patch 2 to CPython, then undid the change to bytesio.c. The new test fails, and sometimes Python will segmentation fault. If I then apply the patch to bytesio.c it passes. Nice work!

[python-dev]

New changeset a5858c30db7c by Serhiy Storchaka in branch '3.5':
Issue bpo-24989: Fixed buffer overread in BytesIO.readline() if a position is
https://hg.python.org/cpython/rev/a5858c30db7c

New changeset 215800fb955d by Serhiy Storchaka in branch 'default':
Issue bpo-24989: Fixed buffer overread in BytesIO.readline() if a position is
https://hg.python.org/cpython/rev/215800fb955d

[serhiy-storchaka]

https://bitbucket.org/larry/cpython350/pull-requests/13/issue-24989/diff

[larryhastings]

Pull request accepted. Please forward-merge. Thanks!