FileIO not 64-bit safe under Windows

[pitrou]

BPO	9611
Nosy	@loewis, @amauryfa, @pitrou, @vstinner, @tjguk, @briancurtin
Files	read_write_32bits.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-07-05.09:49:52.592>
created_at = <Date 2010-08-15.17:25:41.177>
labels = ['extension-modules', 'type-bug', 'expert-IO', 'OS-windows']
title = 'FileIO not 64-bit safe under Windows'
updated_at = <Date 2011-07-05.09:51:53.940>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2011-07-05.09:51:53.940>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2011-07-05.09:49:52.592>
closer = 'vstinner'
components = ['Extension Modules', 'Windows', 'IO']
creation = <Date 2010-08-15.17:25:41.177>
creator = 'pitrou'
dependencies = []
files = ['19899']
hgrepos = []
issue_num = 9611
keywords = ['patch']
message_count = 28.0
messages = ['113974', '113980', '113985', '113988', '113990', '113992', '113995', '113997', '120388', '120389', '120394', '120430', '120431', '120434', '120443', '120444', '120445', '120446', '120448', '123041', '123042', '125257', '125313', '125961', '139836', '139838', '139840', '139841']
nosy_count = 8.0
nosy_names = ['loewis', 'amaury.forgeotdarc', 'pitrou', 'vstinner', 'tim.golden', 'brian.curtin', 'mspacek', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9611'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

[pitrou]

Modules/_io/fileio.c assumes that read() and write() allow a Py_ssize_t length, but under Windows the length is an int, limiting chunk size to 2GB.

It should be easy enough to read or write in multiple chunks, although testing might be difficult.

[pitrou]

os.write() (in posixmodule.c) is also affected. os.read(), however, is limited to 32-bit inputs.

[loewis]

Using chunked writes is tricky. If the first write succeeds, and the second one fails, how do you report the result?

[pitrou]

The write() man page says:

   The number of bytes written may be less than count if, for example, there is  insufficient  space  on
   the underlying physical medium, or the RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)),
   or the call was interrupted by a signal handler after having written less  than  count  bytes.   (See
   also pipe(7).)

So, we could return the number of bytes successfully written, and let the next call fail.

Another possibility is to only write 2GB-1 and let the caller retry.
Most people use buffered I/O, and the buffered layer automatically retries.

[loewis]

Most people use buffered I/O, and the buffered layer automatically retries.

I see. I think this is already slightly problematic: if you send an
interrupt, it won't oblige. IMO, any such loop ought to be
interruptable.

[pitrou]

> Most people use buffered I/O, and the buffered layer automatically retries.

I see. I think this is already slightly problematic: if you send an
interrupt, it won't oblige. IMO, any such loop ought to be
interruptable.

Well, the loop stops when an error status is returned by the raw IO
layer. At that point, the buffered IO layer re-raises the error after a
bit of internal cleanup.

[loewis]

Well, the loop stops when an error status is returned by the raw IO
layer. At that point, the buffered IO layer re-raises the error after a
bit of internal cleanup.

Assume the following case:

1. writing starts, and writes some data
2. Ctrl-C is pressed, raises a signal, and interrupts the current
   system call (EINTR)
3. having already written data, the signal is discarded, and the
   number of successfully written bytes is returned.
4. the loop retries to write the rest. Not receiving any signal
   anymore, the subsequent write operations wait for completion.

End consequence: the signal is discarded without any effect.

[pitrou]

Le dimanche 15 août 2010 à 18:53 +0000, Martin v. Löwis a écrit :

Martin v. Löwis <martin@v.loewis.de> added the comment:

> Well, the loop stops when an error status is returned by the raw IO
> layer. At that point, the buffered IO layer re-raises the error after a
> bit of internal cleanup.

Assume the following case:

1. writing starts, and writes some data
2. Ctrl-C is pressed, raises a signal, and interrupts the current
   system call (EINTR)
3. having already written data, the signal is discarded, and the
   number of successfully written bytes is returned.
4. the loop retries to write the rest. Not receiving any signal
   anymore, the subsequent write operations wait for completion.

Ok, I guess the loop should run PyErr_CheckSignals() somewhere.

Simulate such a situation in an unit test will be a bit tricky.
Perhaps we can use os.pipe() and depend on the fact that writes greater
than the pipe buffer size will be blocking.

[mspacek]

We've got a near duplicate bpo-9015. This thread seems more considered though :) Note that writing more than 2**32-1 bytes at once results in a hung process with 100% CPU in 64-bit Windows, which has to be killed with Task Manager. So I think that qualifies as a crash. This is apparently due to fwrite limitations in msvc, even in win64.

[amauryfa]

Fortunately, the lower-level write() has no such bug, at least when used in binary mode as FileIO does: it's almost a direct call to WriteFile().
This issue is more considered because it's not a bug in the Microsoft CRT, but in the Python implementation of the IO stack.

About the issue, I'd suggest to just clamp the length to 2**32-1, and don't bother retrying; leave this to the buffered layer.

[pitrou]

About the issue, I'd suggest to just clamp the length to 2**32-1, and
don't bother retrying; leave this to the buffered layer.

Yes, I think it's reasonable.
(or perhaps clamp to something page-aligned, such as 2**32-4096).
Also, the signal issue which was raised by Martin above has since been fixed in r84239.