urlparse fails if the path is numeric

[BjrnLindqvist]

BPO	27657
Nosy	@orsenthil, @benjaminp, @ned-deily, @leifwalsh, @bitdancer, @ambv, @mgorny, @vadmium, @timgraham, @miss-islington, @Roguelazer
PRs	bpo-27657: Fix urlparse() with numeric paths #661 [3.7] bpo-27657: Fix urlparse() with numeric paths (GH-661) #16837 [3.8] bpo-27657: Fix urlparse() with numeric paths (GH-661) #16839 [3.8] bpo-27657: Revert "Fix urlparse() with numeric paths (GH-661)  #18525 [3.7] bpo-27657: Revert - Fix urlparse() with numeric paths (GH-661)" #18526

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = <Date 2020-06-08.17:16:41.595>
created_at = <Date 2016-07-30.19:57:17.789>
labels = ['type-bug', 'library', '3.9']
title = 'urlparse fails if the path is numeric'
updated_at = <Date 2021-07-02.00:01:49.848>
user = 'https://bugs.python.org/BjrnLindqvist'

bugs.python.org fields:

activity = <Date 2021-07-02.00:01:49.848>
actor = 'leif.walsh'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2020-06-08.17:16:41.595>
closer = 'orsenthil'
components = ['Library (Lib)']
creation = <Date 2016-07-30.19:57:17.789>
creator = 'Bj\xc3\xb6rn.Lindqvist'
dependencies = []
files = []
hgrepos = []
issue_num = 27657
keywords = ['3.7regression']
message_count = 31.0
messages = ['271702', '271703', '271719', '271738', '271739', '271823', '271824', '289557', '354889', '354894', '354903', '355320', '359273', '359277', '360196', '361815', '362103', '362107', '362632', '362675', '362742', '370048', '370120', '371015', '371024', '371040', '371042', '371043', '371044', '371046', '396833']
nosy_count = 13.0
nosy_names = ['orsenthil', 'benjamin.peterson', 'ned.deily', 'leif.walsh', 'r.david.murray', 'lukasz.langa', 'mgorny', 'martin.panter', 'Bj\xc3\xb6rn.Lindqvist', 'Tim.Graham', 'miss-islington', 'roguelazer', 'Chris Dent']
pr_nums = ['661', '16837', '16839', '18525', '18526']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27657'
versions = ['Python 3.9']

[BjrnLindqvist]

This affects both Python 2 and 3. This is as expected:

>>> urlparse('abc:123.html')
ParseResult(scheme='abc', netloc='', path='123.html', params='', query='', fragment='')
>>> urlparse('123.html:abc')
ParseResult(scheme='123.html', netloc='', path='abc', params='', query='', fragment='')
>>> urlparse('abc:123/')
ParseResult(scheme='abc', netloc='', path='123/', params='', query='', fragment='')

This is NOT:

>>> urlparse('abc:123')
ParseResult(scheme='', netloc='', path='abc:123', params='', query='', fragment='')

Expected is path='123' and scheme='abc'. At least according to my reading of the rfc (https://tools.ietf.org/html/rfc1808.html) that is what should happen.

[bitdancer]

See bpo-14072. It may be time to look at this again, but we may still be constrained by backward compatibility.

[vadmium]

The main backward compatibility consideration would be bpo-754016, but don’t agree with the changes made, and would support reverting them. The original bug reporter wanted urlparse("1.2.3.4:80", "http") to be treated as the URL http://1.2.3.4:80, but the IP address was being parsed as a scheme, so the default “http” scheme was ignored.

The original fix (r83701) affected any URL that had a digit 0–9 immediately after the “scheme:” prefix. In such URLs, the scheme component was no longer parsed. A test case for “path:80” was added, and a demonstration of not parsing any scheme from www.cwi.nl:80/%7Eguido/Python.html was added in the documentation.

Later, the logic was altered to test if the URL looked like an integer (revision 495d12196487, bpo-11467). This restored proper parsing of clsid:85bbd92o-42a0-1o69-a2e4-08002b30309d and mailto:1337@example.org, although another URL given, javascript:123, remains misparsed. The documentation was subsequently adjusted in bpo-16932 to just demonstrate www.cwi.nl/%7Eguido/Python.html being parsed as a path.

The logic was watered down to its current form by revision 9f6b7576c08c, bpo-14072. Now it tests for a non-digit anywhere after the scheme, so that tel:+31641044153 is again parsed properly. But it was pointed out that tel:1234 remains misparsed.

What’s the next step in the watering-down process? All the attempts so far break valid URLs in favour of special-casing inputs that are not valid URLs.

[bitdancer]

I hate to say it, but this may require a python-dev discussion. We probably ought to be parsing valid urls correctly as our top priority, but if that breaks our parsing of "reasonable" non-valid URLs (that existing code is depending on), it's going to be a backward compatibility problem.

[bitdancer]

On second thought, what are the chances that special casing something that looks like an IP address in the scheme position would maintain backward compatibility?

[vadmium]

Depends on how you define “looks like an IP address”. Does the www.cwi.nl:80 case look like an IP address? What about “path:80” or “localhost:80”? If there is any code relying on the bug, it may just as easily involve host name as a numeric IP address.

[bitdancer]

Ah, good point, I misread the scope of the problem.

[timgraham]

Based on discussion in bpo-16932, I agree that reverting the parsing decisions from bpo-754016 (as Martin suggested in msg271719) seems appropriate. I created a pull request that does that.

[orsenthil]

New changeset 5a88d50 by Senthil Kumaran (Tim Graham) in branch 'master':
bpo-27657: Fix urlparse() with numeric paths (#661)
5a88d50

[miss-islington]

New changeset 82b5f6b by Miss Islington (bot) in branch '3.7':
bpo-27657: Fix urlparse() with numeric paths (GH-661)
82b5f6b

[orsenthil]

New changeset 0f3187c by Senthil Kumaran in branch '3.8':
[3.8] bpo-27657: Fix urlparse() with numeric paths (GH-661) (bpo-16839)
0f3187c