[http.client] HTTPConnection.request not support "chunked" Transfer-Encoding to send data

[harobed]

BPO	12319
Nosy	@orsenthil, @pitrou, @vstinner, @akheron, @vadmium, @eryksun, @demianbrecht, @matrixise, @RKrahl
Files	chunkedhttp.py: A custom module implementing upload with chunked transfer encoding for urllib issue12319.patch issue12319_1.patch: Removing unused imports issue12319_2.patch issue12319_3.patch issue12319_4.patch issue12319_5.patch issue12319_6.patch chunkedhttp-2.py: Illustration of the situation after applying issue12319_6.patch from user's point of view issue12319_7.patch issue12319_8.patch issue12319_9.patch issue12319_10.patch issue12319_11.patch issue12319_12.patch chunked-file.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vadmium'
closed_at = <Date 2016-08-27.04:12:55.150>
created_at = <Date 2011-06-12.10:47:13.084>
labels = ['type-feature', 'library']
title = '[http.client] HTTPConnection.request not support "chunked" Transfer-Encoding to send data'
updated_at = <Date 2016-08-27.04:12:55.149>
user = 'https://bugs.python.org/harobed'

bugs.python.org fields:

activity = <Date 2016-08-27.04:12:55.149>
actor = 'martin.panter'
assignee = 'martin.panter'
closed = True
closed_date = <Date 2016-08-27.04:12:55.150>
closer = 'martin.panter'
components = ['Library (Lib)']
creation = <Date 2011-06-12.10:47:13.084>
creator = 'harobed'
dependencies = []
files = ['36493', '38351', '38352', '38528', '38670', '38772', '38797', '39447', '39466', '43398', '43579', '43973', '44057', '44076', '44133', '44217']
hgrepos = []
issue_num = 12319
keywords = ['patch', 'needs review']
message_count = 63.0
messages = ['138203', '138242', '138258', '138296', '138357', '138691', '171268', '226012', '226018', '226024', '236024', '236037', '237309', '237312', '237313', '237427', '237509', '238314', '239119', '239151', '239769', '239771', '239793', '243518', '243601', '243730', '243731', '243747', '243751', '243762', '243800', '243823', '243837', '243878', '268611', '268715', '268894', '269430', '269497', '271771', '271815', '272121', '272164', '272239', '272241', '272322', '272354', '272394', '272395', '272413', '272438', '272442', '272937', '273532', '273533', '273534', '273536', '273538', '273547', '273548', '273628', '273751', '273758']
nosy_count = 13.0
nosy_names = ['orsenthil', 'pitrou', 'vstinner', 'harobed', 'python-dev', 'petri.lehtinen', 'martin.panter', 'piotr.dobrogost', 'eryksun', 'demian.brecht', 'matrixise', 'whitemice', 'Rotkraut']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12319'
versions = ['Python 3.6']

[harobed]

Hi,

HTTPConnection.putrequest not support "chunked" Transfer-Encodings to send data.

Exemple, I can't do PUT request with chunk transfert.

Regards,
Stephane

[akheron]

What's the use case? Do you have an iterable that yields data whose size is unknown?

AFAIK, most web servers don't even support chunked uploads.

(Removing Python 2.7 from versions as this is clearly a feature request.)

[harobed]

I use http.client in WebDAV client.

Mac OS X Finder WebDAV client perform all his request in "chunk" mode : PUT and GET.

Here, I use http.client to simulate Mac OS X Finder WebDAV client.

Regards,
Stephane

[akheron]

harobed wrote:

I use http.client in WebDAV client.

Mac OS X Finder WebDAV client perform all his request in "chunk" mode : PUT and GET.

Here, I use http.client to simulate Mac OS X Finder WebDAV client.

Now I'm confused. Per the HTTP specification, GET requests don't have
a body, so "Transfer-Encoding: chunked" doesn't apply to them.

Are you sure you don't confuse with the response that the server
sends? In responses, "Transfer-Encoding: chunked" is very common.

[harobed]

Now I'm confused. Per the HTTP specification, GET requests don't have
a body, so "Transfer-Encoding: chunked" doesn't apply to them.

Are you sure you don't confuse with the response that the server
sends? In responses, "Transfer-Encoding: chunked" is very common.

Sorry, yes GET requests have "Transfer-Encoding: chunked" in server response.
PUT requests can send body data in transfer-encoding chunked mode.

[orsenthil]

We had support for chunked transfer encoding for POST method recently, which is exposed via urllib2 wrapper function. PUT is not exposed via urllib2 and users should use httplib. This feature of chunked transfer can be added to PUT by taking the body of the message as iterable.

[pitrou]

We had support for chunked transfer encoding for POST method recently,
which is exposed via urllib2 wrapper function.

I couldn't find what you're talking about.
If I look at AbstractHTTPHandler.do_request_, it actually mandates a Content-Length header for POST data (hence no chunked encoding).

[RKrahl]

I'd like to support the request. I have a use case where I definitely need this feature: I maintain a Python client for a scientific metadata catalogue, see 1 for details. The client also features the upload of the data files. The files may come in as a data stream from another process, so my client takes a file like object as input. The files may be large (several GB), so buffering them is not an option, they must get streamed to the server as they come in. Therefore, there is have no way to know the Content-length of the upload beforehand.

I implemented chunked transfer encoding in a custom module that monkey patches the library, see the attached file. This works fine, but of course it's an awkward hack as I must meddle deeply into the internals of urllib and http.client to get this working. This module is tested to work with Python 2.7, 3.1, 3.2, 3.3, and 3.4 (for Python 3 you need to pass it through 2to3 first). I really would like to see this feature in the standard library in order to get rid of this hack in my package. I would happy to transform my module into a patch to the library if such a patch would have a chance to get accepted.

[piotrdobrogost]

@rotkraut

The truth is http in stdlib is dead.
Your best option is to use 3rd party libs like requests or urllib3.
Authors of these libs plan to get rid of httplib entirely; see "Moving away from httplib" (urllib3/urllib3#58)

[RKrahl]

Thanks for the notice! As far as I read from the link you cite, getting rid of the current httplib in urllib3 is planned but far from being done. Furthermore, I don't like packages with too many 3rd party dependencies. Since my package is working fine with the current standard lib, even though using an ugly hack in one place, I'd consider switching to urllib3 as soon as the latter makes it into the standard lib.

I still believe that adding chunked transfer encoding to http.client and urllib in the current standard lib would only require a rather small change that can easily be done such that the lib remains fully compatible with existing code. Still waiting for feedback if such a patch is welcome.

[vadmium]

One interesting question is how to convey data to the chunked encoder. There are two sets of options in my mind, a pull interface:

• iterable: seems to be the popular way amoung commenters here
• file reader object: encoder calls into stream’s read() method

and a push interface:

• chunked encoder is a file writer object: user calls encoder’s write() and close() methods. This would suit APIs like saxutils.XMLGenerator and TextIOWrapper.
• chunked encoder has a “feed parser” interface, codecs.IncrementalEncoder interface, or something else.

The advantage of the push interface is that you could fairly easily feed data from an iterable or file reader into it simply by just doing shutil.copyfileobj() or equivalent. But to adapt the pull interface to a push interface would require “asyncio” support or a separate thread or something to invert the flow of control. So I think building the encoder with a push interface would be best. Rolf’s ChunkedHTTPConnectionMixin class appears to only support the pull interface (iterables and stream readers).

I would welcome support for chunked uploading in Python’s “http.client” module, especially with push or stream writer support. I don’t think overwriting _send_request should be necessary; just call putrequest(), putheader() etc manually, and then call send() for each chunk. Perhaps there is scope for sharing the code with the “http.server” module (for encoding chunked responses).

[RKrahl]

The design goal for my implementation was compatibility. My version can be used as a drop in replacement for the existing urllib's HTTPHandler. The only thing that need to be changed in the calling code is that it must call build_opener() to select the chunked handler in the place of the default HTTPHandler. After this, the calling code can use the returned opener in the very same way as usual.

I guess, switching to a push interface would require major changes in the calling code.

In principle, you could certainly support both schemes at the same time: you could change the internal design to a push interface and than wrap this by a pull interface for the compatibility with existing code. But I'm not sure whether this would be worth the effort. If, as Piotr suggests, the current urllib is going to be replaced by urllib3, then I guess, its questionable if it makes sense to add major design changes that are incompatible with existing code to the current standard lib.