Add a new socket.sendfile() method

[giampaolo]

BPO	17552
Nosy	@gvanrossum, @pitrou, @giampaolo, @tiran, @asvetlov, @4kir4, @1st1, @MojoVampire
Files	socket-sendfile1.patch: initial draft socket-sendfile2.patch: update file offset add 'offset' parameter socket-sendfile3.patch: docs, selectors module, always seek() file, use memoryview(), ssl tests bench.py socket-sendfile4.patch: return no. of bytes sent instead of (ok, exc); "sleep" on BlockingIOError socket-sendfile5.patch: fix some issues on windows socket-sendfile6.patch: get rid of "blocksize" and "use_fallback" args, add "count" arg, use fstat() socket-sendfile7.patch: adds send_blocksize argument

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/giampaolo'
closed_at = <Date 2014-06-11.02:04:53.134>
created_at = <Date 2013-03-26.19:27:53.402>
labels = ['type-feature', 'library']
title = 'Add a new socket.sendfile() method'
updated_at = <Date 2014-06-11.02:04:53.054>
user = 'https://github.com/giampaolo'

bugs.python.org fields:

activity = <Date 2014-06-11.02:04:53.054>
actor = 'giampaolo.rodola'
assignee = 'giampaolo.rodola'
closed = True
closed_date = <Date 2014-06-11.02:04:53.134>
closer = 'giampaolo.rodola'
components = ['Library (Lib)']
creation = <Date 2013-03-26.19:27:53.402>
creator = 'giampaolo.rodola'
dependencies = []
files = ['29583', '29734', '34986', '34994', '34997', '35004', '35039', '35507']
hgrepos = []
issue_num = 17552
keywords = ['3.2regression']
message_count = 35.0
messages = ['185292', '185294', '185295', '185303', '186305', '216915', '216968', '216973', '216975', '216979', '217015', '217058', '217080', '217081', '217082', '217090', '217099', '217104', '217120', '217121', '217128', '217143', '217144', '217156', '217165', '219926', '219927', '219928', '219929', '220173', '220191', '220195', '220202', '220220', '220226']
nosy_count = 12.0
nosy_names = ['gvanrossum', 'pitrou', 'giampaolo.rodola', 'christian.heimes', 'josiah.carlson', 'asvetlov', 'neologix', 'akira', 'rosslagerwall', 'python-dev', 'yselivanov', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17552'
versions = ['Python 3.5']

[giampaolo]

This is based on progress made in bpo-13564 and aims to provide a new sendfile() method for the socket class taking advantage of high-performance "zero-copy" os.sendfile() available on most POSIX platforms.
A wrapper on top of os.sendfile() appears to be convenient because getting everything right is a bit tricky. Also we can avoid code duplication in case we want to add sendfile() support to ftplib (bpo-13564) and httplib.

=== THE API ===

Attached is a draft patch proposing an API which:

• uses os.sendfile() whenever available and usable (a mmap-like file is required)
• if not falls back on using plain socket.sendall()
• returns a tuple of two elements which indicates whether sendfile() was used and an exception instance in case it wasn't on account of an internal error, if any
• does not support non-blocking sockets (will raise ValueError)

=== ALTERNATE API ===

1. In case the transfer is interrupted halfway because of an error the user has no clue on how many bytes were sent (and file.tell() won't tell =)). As such it probably makes sense to provide a custom socket.SendfileError exception encapsulating the original exception and the bytes sent as arguments.

2. For the same reason the returned tuple should probably include the number of bytes transmitted.

=== WINDOWS SUPPORT ===

Further development may include Windows support by using TransmitFile (http://msdn.microsoft.com/en-us/library/windows/desktop/ms740565(v=vs.85).aspx). Once we agree on an API I might start looking into Windows code (which appears to be quite tricky btw).

Any feedback is very welcome.

[pitrou]

A couple of comments:

• I don't understand the point of the second member in the tuple
• the timeout logic should be fixed so that the total operation time doesn't exceed the timeout, rather than each iteration (in other words, a deadline should be computed at the start and then the poll / select calls be issued in accordance with that deadline)
• non-blocking sockets could be supported by returning partial writes, and letting the caller pass an offset argument
• as a vocabulary nit, I don't really understand what "mmap-like file" means

[neologix]

• I don't understand the "running out of FDs" thing. select() is limited to FDs less than FD_SETSIZE, but I don't see how you could get EMFILE (select() will return a ValueError)

• is there any platform with sendfile() which doesn't support poll()?

• I still don't see the point of calling sendfile() with a block size. Everybody just uses st_size, and that's how it's supposed to be done.

[giampaolo]

I don't understand the point of the second member in the tuple

The 'exception' member can be useful to know the reason why sendfile() failed and send() was used as fallback.

the timeout logic should be fixed so that the total operation
time doesn't exceed the timeout, rather than each iteration

Do you mean that if the user sets a timeout=2 and the whole transfer takes longer than that you expect a timeout exception?
That's pretty unusual (e.g. ftplib does not behave like that).

non-blocking sockets could be supported by returning partial writes,
and letting the caller pass an offset argument

I see little value in supporting non-blocking sockets because someone willing to do that usually wants to use sendfile() directly (no wrappers involved) as they are forced to handle errors and take count of transmitted bytes in their own code anyway.
Here's an example of how sendfile is supposed to be used with non-blocking sockets:
https://code.google.com/p/pyftpdlib/source/browse/tags/release-0.7.0/pyftpdlib/ftpserver.py#1035
An extra layer such as the one proposed in my patch is completely useless and also slow considering the amount of instructions before the actual sendfile() call.
It is however a good idea to provide an 'offset' argument.

as a vocabulary nit, I don't really understand what "mmap-like file" means

Will turn that into "regular files".

I don't see how you could get EMFILE (select() will return a ValueError)

Right. Will fix that.

is there any platform with sendfile() which doesn't support poll()?

No idea. I made a quick tour on snakebite and apparently all of the provided platforms have poll().

I still don't see the point of calling sendfile() with a block size

I will make some actual benchmarks and fix that later if makes sense.
Assuming providing a high blocksize (say 1MB) makes no difference in terms of CPU usage it's better to keep that for consistency with sendall() in case we fallback on using it.

[giampaolo]

New patch in attachment includes a new 'offset' parameter, new tests and also update file offset on return or in case of error so that file.tell() can be used to tell how many bytes were transmitted at any time.
This way we'll avoid using a custom exception.

In summary, the API looks like this.

Transfer ok:

>>> file = open('somefile', 'rb')
>>> s = socket.socket()
>>> sock.sendfile(file)
(True, None)
>>> file.tell()
20000000
>>>

...and in case sendfile() could not be used internally because file was not a regular file:

>>> file = io.BytesIO(b'x' * 1*1024*1024)
>>> sock.sendfile(file)
(False, UnsupportedOperation('fileno',))
>>> file.tell()
20000000
>>>

I still haven't looked into TransmitFile on Windows as I have to figure out how to compile Python 3.4 on Windows.

[giampaolo]

New patch in attachment. Changes:

• docs

• replaced select() / poll() with the new selectors module

• file position is always updated both on return and on error; this means file.tell() is the designated way to know how many bytes were sent

• replaced sendall() with send() so that we can count the number of bytes transmitted (related and rejected proposal: https://mail.python.org/pipermail/python-ideas/2014-April/027689.html)

• send() now uses memoryview() for better performances to re-transmit data which was not sent by the first send() call

• pre-emptively raise exception if file is not opened in binary mode

• tests for ssl module

I've tried to work on Windows TransmitFile support but I got stuck as I'm not sure how to convert a file object into a HANDLE in C. I suppose Windows support can also be added later as a separate ticket and in the meantime I'd like to push this forward.

Open questions:

• Is the current return value desirable (do we really care if os.sendfile() was used internally?)? Should the returned tuple also include the number transmitted bytes?

• default blocksize: Charles-François was suggesting to remove the blocksize argument; FWIW I've made some quick benchmarks by using "time" cmdline utility with different blocksizes and I didn't notice substantial difference. I still think a blocksize parameter is necessary in case we fallback on using send() and also for consistency with ftplib's storbinary() method which will be involved later (bpo-13564).

[giampaolo]

Attached is a simple benchmark script transmitting a 100MB file.
On my Linux box sendfile() is almost twice as fast as send():

send()
real     0.0613s
user     0.0100s
sys      0.0900s
total    0.1000s

sendfile()
real     0.0318s
user     0.0000s
sys      0.0500s
total    0.0500s

[MojoVampire]

For TransmitFile support, the Windows function to turn an integer file descriptor into a WinAPI file HANDLE should be _get_osfhandle: http://msdn.microsoft.com/en-us/library/ks2530z6.aspx

[4kir4]

Should socket.sendfile() always return number of bytes sent because file.tell() may be changed by something else that uses the same file descriptor?

What happens if the file grows?

Instead of returning (was_os_sendfile_used, os_sendfile_error), you could specify no_fallback=False that could be set to True to assert that the fallback is not used (with no_fallback=True the os_sendfile_error is raised instead of using socket.send() as a fallback).

If possible; always include number of bytes sent in any error that is raised.

[giampaolo]

Instead of returning [...] you could specify no_fallback=False that
could be set to True to assert that the fallback is not used
[...] and return the number of bytes sent.

Good idea, thanks, that is much better indeed. Updated patch is in attachment.

[...] file.tell() may be changed by something else that uses
the same file descriptor? What happens if the file grows?

I would say that is a use case we should explicitly not support as it probably implies you're doing something you're not supposed to.

If possible always include the number of bytes sent in any error that is raised.

That's similar to my recent (rejected) proposal for socket.sendall():
https://mail.python.org/pipermail/python-ideas/2014-April/027689.html
IMO the patch as it stands is fine as you can determine the number of bytes which were sent either by using the function return value or file.tell() (in case of error).
Also, updating the file offset on exit makes the sendfile() implementation behave exactly like send().

[giampaolo]

Yet another patch fixing some problems on Windows.
Hopefully this should be the last one as for what concerns the POSIX systems. As such I would kindly ask for a review and/or further suggestions.