distutils upload/register should use CRLF in HTTP requests

[BrianJones]

BPO	10510
Nosy	@orsenthil, @tarekziade, @merwok, @bitdancer, @sigmavirus24
Files	compliant_distutils.patch: Issue patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/merwok'
closed_at = <Date 2014-09-27.20:59:57.489>
created_at = <Date 2010-11-23.05:11:52.451>
labels = ['easy', 'type-bug', 'library']
title = 'distutils upload/register should use CRLF in HTTP requests'
updated_at = <Date 2014-11-21.23:35:14.210>
user = 'https://bugs.python.org/BrianJones'

bugs.python.org fields:

activity = <Date 2014-11-21.23:35:14.210>
actor = 'eric.araujo'
assignee = 'eric.araujo'
closed = True
closed_date = <Date 2014-09-27.20:59:57.489>
closer = 'r.david.murray'
components = ['Distutils']
creation = <Date 2010-11-23.05:11:52.451>
creator = 'Brian.Jones'
dependencies = []
files = ['35067']
hgrepos = []
issue_num = 10510
keywords = ['patch', 'easy']
message_count = 25.0
messages = ['122192', '122210', '122220', '122223', '122440', '122455', '122457', '122458', '122459', '122460', '124093', '139069', '139106', '139109', '139139', '146215', '211826', '211828', '217336', '218822', '219157', '221100', '227728', '227760', '231502']
nosy_count = 10.0
nosy_names = ['orsenthil', 'jerub', 'tarek', 'eric.araujo', 'r.david.murray', 'mitchellh', 'Brian.Jones', 'python-dev', 'icordasc', 'Matthew.Iversen']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10510'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[BrianJones]

In trying to write a PyPI service, I of course need to support the registration and upload features of setup.py, which uses distutils for those actions. One thing making this a bit more difficult than need be is the fact that distutils.command.register and distutils.command.upload both use bare line feeds as a line terminator in the HTTP message headers. While there are probably tolerant servers out there, I think this behavior should be considered a bug in light of the HTTP spec, which reads:

"The line terminator for message-header fields is the sequence CRLF. However, we recommend that applications, when parsing such headers, recognize a single LF as a line terminator and ignore the leading CR."

The second sentence can be interpreted in multiple ways that either agree with or contradict the first sentence, but the first sentence is at least perfectly clear, and I'd like to see "\r\n" added to the beginning of (specifically in this case) the Content-Disposition headers assembled by distutils.command.upload and distutils.command.register.

The change involves taking identical lines in each file that currently look like this:

       title = '\nContent-Disposition: form-data; name="%s"' % key

...and making them look like this:

       title = '\r\nContent-Disposition: form-data; name="%s"' % key

The result is that servers which do not support a lax adherence to the protocol will still be able to support users of distutils in Python 2.6, 2.7, and 3.1 (and maybe others).

[merwok]

Thank you for the report. This is how I understand the part of the spec you quoted: Strictly valid HTTP uses CRLF, but servers and clients should be liberal in what they accept and deal with LF too. Senthil, do you agree with that?

There are a small number of PyPI-like servers out there which seem to deal fine with what distutils produces. We have strong backward compatibility constraints in distutils, so we try to make the less changes possible to fix bugs. Because we’ve haven’t had this reported by those other servers before, I’m tempted to reject this report. The good part is that since you’re still writing the server, you can change its code to follow Postel’s Law and be liberal in what you accept. Does that sound acceptable to you?

Best regards

[BrianJones]

In truth, I don't personally know if the other PyPI server implementations also have to work around this issue. Other comments on that are welcome.

As for my own implementation, I've implemented a workaround to this, but I'm working around and reimplementing what is otherwise a feature built into the server (and every other one).

I believe the suggested change makes the world a better place, and having administered several different web servers, I can't imagine ill effects from making the change. Web servers support *at least* the standard. Any ill effects we might imagine are, at least for my part, pure speculation. I, on the other hand, have found a real-life problem now :)

I guess in the end I think that servers are more likely to err on the side of strict HTTP than make allowances for every "SHOULD" in the protocol spec, and leaving things as-is relies on a "SHOULD", which I submit is incorrect behavior, and therefore a bug.

It's not like a fix is going to magically improve my life, as I'll have to support older buggy versions anyway, but making the change now can help grease the wheels for future implementations who won't have to hit this hurdle.

My $.02.

[orsenthil]

I think, it is better that distutils.command.register and
distutils.command.upload use CRLF as the line terminator for header
values.
It just helps in many cases, we can safely side by this case, but not
relying LR or CR only may not be always possible.

[merwok]

I am convinced. Do you want to work on a patch? Guidelines are found at http://www.python.org/dev/patches/

[BrianJones]

Sure. I'll create a patch in the next few days and submit it. Thanks for the link to the guidelines. :)

[BrianJones]

So... have I missed a memo, or is it currently impossible to test the current svn version of distutils in the current svn version of Python?

The tests for (at least) register and upload are written using Python 2.x syntax and modules. How are new features and fixes for distutils being tested? Is it valid to use a 2.x version of Python to test the trunk version of distutils?

Sorry if these questions have really obvious answers. First time patcher :?

[tarekziade]

If you can make your server work with the current implementation, I'd rather not change this in distutils but in distutils2.

Distutils is frozen and we make only bug fixes. By bug fix I mean anything that is a bug. A non-strict implementation of the EOLs is not a bug, and I suggest that you write your patch against distutils2.

Eric, thanks for all the hard work in triaging the bugs ! Let's make sure we do the bare minimal maintenance in distutils, and suggest to people to contribute in distutils2.

Brian, for distutils1, the register and upload command would be considered buggy if they don't work for PyPI, which is not the case.

Thanks!

[BrianJones]

If it's not a bug in distutils1, I imagine it will not be a bug in distutils2, since that will also presumably work with PyPI, and PyPI will be the single solitary supported implementation of the service?

I also don't see distutils2 in this list http://svn.python.org/projects/ which means either distutils2 isn't part of the standard library, or it doesn't follow the documented patch submission process, or perhaps both? Insight welcome here.

[tarekziade]

If it's not a bug in distutils1, I imagine it will not be a bug in distutils2

Yes that would be a feature request. we would be happy to add. e.g. make the commands works with server X ou server Y.

I also don't see distutils2 in this list http://svn.python.org/projects/ which means either distutils2 isn't part of the standard library, or it doesn't follow the documented patch submission process, or perhaps both? Insight welcome here.

Distutils2 is developed at http://hg.python.org/distutils2 and will be included when ready in the stdlib (circa 3.3), as a distutils replacer. It's in fact the distutils trunk, with lots of changes (some are backward incompatible). It will be installable from 2.4 onward.

It does not follow the same process. Which I have described in python-dev last week: http://mail.python.org/pipermail/python-dev/2010-November/105772.html

[merwok]

Updated guidelines: http://wiki.python.org/moin/Distutils/FixingBugs

is it currently impossible to test the current svn version of
distutils in the current svn version of Python?
It is. Branches that get bugfixes are 3.2, 3.1 and 2.7; each one has its interpreter code and its distutils lib and tests. I’m not sure what prompted your question: was it something on the patches guideline page?