Possible integer overflow in iov_setup()

[serhiy-storchaka]

BPO	33871
Nosy	@ronaldoussoren, @ned-deily, @serhiy-storchaka, @pablogsal, @miss-islington
PRs	bpo-33871: Fix os.sendfile(), os.writev(), os.readv(), etc. #7931 [3.7] bpo-33871: Fix os.sendfile(), os.writev(), os.readv(), etc. (GH-7931) #8583 [3.6] bpo-33871: Fix os.sendfile(), os.writev(), os.readv(), etc. (GH-7931) #8584

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-22.18:31:42.575>
created_at = <Date 2018-06-15.18:14:36.988>
labels = ['3.8', '3.7', 'library', 'type-crash']
title = 'Possible integer overflow in iov_setup()'
updated_at = <Date 2018-09-22.18:31:42.574>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-09-22.18:31:42.574>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-22.18:31:42.575>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2018-06-15.18:14:36.988>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 33871
keywords = ['patch']
message_count = 8.0
messages = ['319636', '319641', '319645', '319757', '319879', '322739', '322746', '322749']
nosy_count = 5.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'serhiy.storchaka', 'pablogsal', 'miss-islington']
pr_nums = ['7931', '8583', '8584']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue33871'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[serhiy-storchaka]

The iov_setup() helper in posixmodule.c returns the total size of all buffers. But there is possible an integer overflow because the sequence of buffers can contain the same buffer repeated multiple times.

On 32-bit platform:

>>> import os
>>> f = open('/tmp/temp', 'wb')
>>> os.writev(f.fileno(), [b'x' * 2**16] * 2**15)
-1

Since the overflowed sum is negative, os_writev_impl() returns -1 as a signal of error, but since the exception is not set, -1 is returned as the result of os.writev(). If the overflowed sum is not negative, the sequence of buffers is passed to OS and an OSError is raised:

>>> os.writev(f.fileno(), [b'x' * 2**16] * 2**16)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument

I have not tested (because have not installed corresponding 32-bit OSes, and it is harder to reproduce on 64-bit), but seems this can even cause a crash in os.sendfile() on FreeBSD, DragonFly BSD and Mac OS.

This sum is used only in os.sendfile() on Mac OS. In all other cases it is enough to return just an error flag. I can't find the documentation for os.sendfile() on Mac OS for checking if this value actually is needed.

[pablogsal]

I can reproduce the issue on a i686 GNU/Linux Debian system:

>>> import os
>>> f = open('/tmp/temp', 'wb')
>>> l = os.writev(f.fileno(), [b'x' * 2**16] * 2**15)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: error return without exception set

[pablogsal]

Docs for sendfile in macOS:

https://www.unix.com/man-page/osx/2/sendfile/

[ned-deily]

Test case:

import os
fo = open('/tmp/temp', 'wb')
fi = open('/tmp/temp', 'rb')
os.sendfile(fo.fileno(), fi.fileno(), 0, 0, headers=[b'x' * 2**16] * 2**15)

--
run against current master HEAD (2f9cbaa, as of 2018-06-15)
--with-pydebug
current macOS 10.13.5

64-bit Python

$ ./bin/python3.8 ~/Desktop/test_s.py
Traceback (most recent call last):
 File "/Users/nad/Desktop/test_s.py", line 4, in <module>
   os.sendfile(fo.fileno(), fi.fileno(), 0, 0, headers=[b'x' * 2**16] * 2**15)
OSError: [Errno 38] Socket operation on non-socket
sys:1: ResourceWarning: unclosed file <_io.BufferedWriter name='/tmp/temp'>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/temp'>

32-bit Python

$ ./bin/python3.8-32 ~/Desktop/test_s.py
Fatal Python error: a function returned NULL without setting an error
SystemError: <built-in function sendfile> returned NULL without setting an error

Current thread 0xa983a1c0 (most recent call first):
File "/Users/nad/Desktop/test_s.py", line 4 in <module>
Abort trap: 6

[serhiy-storchaka]

Thank you Pablo and Ned!

Seems there is other bug on Mac OS, not related to integer overflow. I am working on it.

[serhiy-storchaka]

New changeset 9d57273 by Serhiy Storchaka in branch 'master':
bpo-33871: Fix os.sendfile(), os.writev(), os.readv(), etc. (GH-7931)
9d57273

[miss-islington]

New changeset 3e4b688 by Miss Islington (bot) in branch '3.7':
bpo-33871: Fix os.sendfile(), os.writev(), os.readv(), etc. (GH-7931)
3e4b688

[serhiy-storchaka]

New changeset ada5d99 by Serhiy Storchaka in branch '3.6':
[3.6] bpo-33871: Fix os.sendfile(), os.writev(), os.readv(), etc. (GH-7931) (GH-8584)
ada5d99