WebSocket schemes in urllib.parse

[oberstet]

BPO	13244
Nosy	@orsenthil, @ezio-melotti, @merwok, @bitdancer, @vadmium
Files	issue13244.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = <Date 2014-04-14.22:42:51.932>
created_at = <Date 2011-10-22.11:07:42.848>
labels = ['easy', 'type-feature', 'library']
title = 'WebSocket schemes in urllib.parse'
updated_at = <Date 2014-04-14.22:42:51.931>
user = 'https://bugs.python.org/oberstet'

bugs.python.org fields:

activity = <Date 2014-04-14.22:42:51.931>
actor = 'orsenthil'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2014-04-14.22:42:51.932>
closer = 'orsenthil'
components = ['Library (Lib)']
creation = <Date 2011-10-22.11:07:42.848>
creator = 'oberstet'
dependencies = []
files = ['23497']
hgrepos = []
issue_num = 13244
keywords = ['patch', 'easy']
message_count = 25.0
messages = ['146167', '146188', '146189', '146190', '146192', '146193', '146194', '146195', '146197', '146202', '146203', '146205', '146304', '146377', '146401', '146474', '146477', '146482', '146489', '146491', '146559', '185121', '204160', '204191', '216243']
nosy_count = 7.0
nosy_names = ['orsenthil', 'ezio.melotti', 'eric.araujo', 'r.david.murray', 'nailor', 'martin.panter', 'oberstet']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13244'
versions = ['Python 3.4']

[oberstet]

The urlparse module currently does not support the new "ws" and "wss" schemes used for the WebSocket protocol.

As a workaround, we currently use the following code (which is a hack of course):

import urlparse
wsschemes = ["ws", "wss"]
urlparse.uses_relative.extend(wsschemes)
urlparse.uses_netloc.extend(wsschemes)
urlparse.uses_params.extend(wsschemes)
urlparse.uses_query.extend(wsschemes)
urlparse.uses_fragment.extend(wsschemes)

===

A WebSocket URL has scheme "ws" or "wss", MUST have a network location and MAY have a resource part with path and query components, but MUST NOT have a fragment component.

[nailor]

I added a patch that adds support for WebSocket URL protocol.

However, a few pointers (and questions):

• The patch is now implemented according to Draft 17[1] of WebSocket protocol

• Draft 17 does not support fragments, it states that the fragments should be treated as a part of the URL, so I didn't add the ws and wss support in uses_fragments. However, the Draft 17 states also, that those should always be encoded. Should there be a special case for ws/wss URLs that have non-encoded number-signs in them, for example should we raise some sort of an exception?

[1] http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-17

[oberstet]

fragment identifiers:

the spec says:

"Fragment identifiers are meaningless in the context of WebSocket
URIs, and MUST NOT be used on these URIs. The character "#" in URIs
MUST be escaped as %23 if used as part of the query component."

[see last line of my initial comment]

I nevertheless added the ws/wss schemes to urlparse.uses_fragment so that I can detect them being used and throw.

Does urllib throw when an URL contains a fragment identifier, but the scheme of the URL is not in urlparse.uses_fragment?

If so, thats fine and of course better than putting the burden of checking on the user.

==

Further, when "#" is to be used in a WS URL, it MUST be encoded, and if so, it's interpreted as part of the query component.

So in summary, I think the best would be:

urllib throws upon non-encoded "#", and interpret it as part of the query component when encoded.

[oberstet]

Well, thinking about it, %23 can also appear in a percent encoded path component.

I don't get the conditional "..if used as part of the query component" in the spec.

[nailor]

Actually, if I get it right, it means that following url is valid:

ws://example.com/something#somewhere/

and the # should be considered as being a part of the path. The spec does not say a thing should the # in path component be encoded, so I think it's safe to assume it should can be unencoded. However, following url

ws://example.com/something?query=foo#bar

Is not considered to be valid, as the # is in the query part and is not escaped. So the valid would be:

ws://example.com/something?query=foo%23bar

I think the motivation behind this is to reduce possible conflicts with browsers that might take the #-part as a fragment when it should be part of the query parameters. However, the confusion is still possible with # in path part.

My take on this would be to omit fragments and just parse the url as is without fragments. Encoding could be left to user, even in the case # is in query part.

[oberstet]

I see how you interpret that sentence in the spec, but I would have read it differently:

invalid:

1. ws://example.com/something#somewhere
2. ws://example.com/something#somewhere/
3. ws://example.com/something#somewhere/foo
4. ws://example.com/something?query=foo#bar

valid:

5. ws://example.com/something%23somewhere
6. ws://example.com/something%23somewhere/
7. ws://example.com/something%23somewhere/foo
8. ws://example.com/something?query=foo%23bar

You would take 2. and 3. as valid, but 1. and 4. as invalid, right?

But you are right, the spec does not talk about # in path.

If above is a valid summary of the question, I'd better take that to the Hybi list to get feedback before rushing into anything with urllib ..

[nailor]

I'd take only 4. as invalid, as the WebSocket (to my interpretation) do not have fragments, so it is assumed to be a part of the path in that case.

But yeah, a confirmation from HyBi would be great. Can you link to the discussion from here, if you ask them (in case it's possible)?

[oberstet]

I'll ask (to be sure) and link.

However, after rereading the Hybi 17 section, it says

"""
path = <path-abempty, defined in [RFC3986], Section 3.3>
"""

And http://tools.ietf.org/html/rfc3986 says:

"""
The path is terminated by the first question mark ("?") or number sign ("#") character, or by the end of the URI.
"""

So my reading would be: non-escaped # can never be part of path for a WebSocket URL by reference of RFC3986.

[oberstet]

here the links to the question on the Hybi list:

http://www.ietf.org/mail-archive/web/hybi/current/msg09257.html

and

http://www.ietf.org/mail-archive/web/hybi/current/msg09258.html
http://www.ietf.org/mail-archive/web/hybi/current/msg09243.html

==

I'll track those and come back when there is a conclusion ..

[merwok]

Actually, if I get it right, it means that following url is valid:
ws://example.com/something#somewhere/
I don’t think so. The URI syntax RFC is generic, so a scheme spec cannot redefine the parsing to mean that in your above example, there is no fragment and the path is /something#somewhere/ . I believe the spec means that a # in any component must be %-escaped.

Does urllib throw when an URL contains a fragment identifier, but the scheme of the URL is not
in urlparse.uses_fragment?
Could you try it in a shell and tell us? BTW, please don’t use “throw” in a code or doc patch: exceptions are raised, throw is related but different generator method.

[oberstet]

sorry for "throw" .. somewhat bad habit (stemming from wandering between languages).

uses_fragment extended:

[autobahn@autobahnhub ~/Autobahn]$ python
Python 2.7.1 (r271:86832, Dec 13 2010, 15:52:15)
[GCC 4.2.1 20070719  [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.
>>> import urlparse
>>> wsschemes = ["ws", "wss"]
>>> urlparse.uses_relative.extend(wsschemes)
>>> urlparse.uses_netloc.extend(wsschemes)
>>> urlparse.uses_params.extend(wsschemes)
>>> urlparse.uses_query.extend(wsschemes)
>>> urlparse.uses_fragment.extend(wsschemes)
>>> urlparse.urlparse("ws://example.com/something#somewhere/")
ParseResult(scheme='ws', netloc='example.com', path='/something', params='', query='', fragment='somewhere/')
>>> urlparse.urlparse("ws://example.com/something#somewhere")
ParseResult(scheme='ws', netloc='example.com', path='/something', params='', query='', fragment='somewhere')
>>>

=> fragment extracted

uses_fragment not extended:

[autobahn@autobahnhub ~/Autobahn]$ python
Python 2.7.1 (r271:86832, Dec 13 2010, 15:52:15)
[GCC 4.2.1 20070719  [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.
>>> import urlparse
>>> wsschemes = ["ws", "wss"]
>>> urlparse.uses_relative.extend(wsschemes)
>>> urlparse.uses_netloc.extend(wsschemes)
>>> urlparse.uses_params.extend(wsschemes)
>>> urlparse.uses_query.extend(wsschemes)
>>> urlparse.urlparse("ws://example.com/something#somewhere/")
ParseResult(scheme='ws', netloc='example.com', path='/something#somewhere/', params='', query='', fragment='')
>>> urlparse.urlparse("ws://example.com/something#somewhere")
ParseResult(scheme='ws', netloc='example.com', path='/something#somewhere', params='', query='', fragment='')
>>>

=> no fragment extracted, but interpreted as part of path component
=> no exception raised

The answer on Hybi outstanding, but I would interpret Hybi-17: # must always be escaped, both in path and query components. Fragment components are not allowed. Thus, unescaped # can never appear in WS URL. Further, it must not be ignored, but the WS handshake failed.

If this should indeed be the correct reading of the WS spec, then I think urlparse should raise an exception upon unescaped # within URLs from ws/wss schemes.

[merwok]

>>> urlparse.urlparse("ws://example.com/something#somewhere")
ParseResult(scheme='ws', netloc='example.com', path='/something#somewhere', params='', query='', fragment='')

This makes me sad. I thought we had fixed urllib months ago to follow the damn rules that have been in a bunch of RFCs for years.