code removal from urllib.parse.urlsplit()

[atodorov]

BPO	22891
Nosy	@gpshead, @orsenthil, @vadmium, @timgraham, @atodorov, @miss-islington
PRs	bpo-27657: Fix urlparse() with numeric paths #661 [3.7] bpo-27657: Fix urlparse() with numeric paths (GH-661) #16837 [3.8] bpo-27657: Fix urlparse() with numeric paths (GH-661) #16839
Dependencies	bpo-27657: urlparse fails if the path is numeric
Files	urlparse_refactor.patch: patch which removes unnecessary lines from urlparse()

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-19.01:37:03.557>
created_at = <Date 2014-11-17.10:42:24.782>
labels = ['type-feature', 'library']
title = 'code removal from urllib.parse.urlsplit()'
updated_at = <Date 2021-05-19.01:37:03.554>
user = 'https://github.com/atodorov'

bugs.python.org fields:

activity = <Date 2021-05-19.01:37:03.554>
actor = 'gregory.p.smith'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-19.01:37:03.557>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2014-11-17.10:42:24.782>
creator = 'Alexander.Todorov'
dependencies = ['27657']
files = ['37212']
hgrepos = []
issue_num = 22891
keywords = ['patch']
message_count = 10.0
messages = ['231278', '231283', '231309', '238254', '271712', '289586', '354890', '354895', '354904', '393915']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'orsenthil', 'martin.panter', 'Tim.Graham', 'Alexander.Todorov', 'miss-islington']
pr_nums = ['661', '16837', '16839']
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22891'
versions = ['Python 3.5']

[atodorov]

In the urllib.parse (or urlparse on Python 2.X) module there is this function:

157 def urlsplit(url, scheme='', allow_fragments=True):
158     """Parse a URL into 5 components:
159     <scheme>://<netloc>/<path>?<query>#<fragment>
160     Return a 5-tuple: (scheme, netloc, path, query, fragment).
161     Note that we don't break the components up in smaller bits
162     (e.g. netloc is a single string) and we don't expand % escapes."""
163     allow_fragments = bool(allow_fragments)
164     key = url, scheme, allow_fragments, type(url), type(scheme)
165     cached = _parse_cache.get(key, None)
166     if cached:
167         return cached
168     if len(_parse_cache) >= MAX_CACHE_SIZE: # avoid runaway growth
169         clear_cache()
170     netloc = query = fragment = ''
171     i = url.find(':')
172     if i > 0:
173         if url[:i] == 'http': # optimize the common case
174             scheme = url[:i].lower()
175             url = url[i+1:]
176             if url[:2] == '//':
177                 netloc, url = _splitnetloc(url, 2)
178             if allow_fragments and '#' in url:
179                 url, fragment = url.split('#', 1)
180             if '?' in url:
181                 url, query = url.split('?', 1)
182             v = SplitResult(scheme, netloc, url, query, fragment)
183             _parse_cache[key] = v
184             return v
185         for c in url[:i]:
186             if c not in scheme_chars:
187                 break
188         else:
189             scheme, url = url[:i].lower(), url[i+1:]
190 
191     if url[:2] == '//':
192         netloc, url = _splitnetloc(url, 2)
193     if allow_fragments and '#' in url:
194         url, fragment = url.split('#', 1)
195     if '?' in url:
196         url, query = url.split('?', 1)
197     v = SplitResult(scheme, netloc, url, query, fragment)
198     _parse_cache[key] = v
199     return v

There is an issue here (or a few of them) as follows:

• if url[:1] is already lowercase (equals "http") (line 173) then .lower() on line 174 is reduntant:
  174 scheme = url[:i].lower() # <--- no need for .lower() b/c value is "http"

• OTOH line 173 could refactor the condition and match URLs where the scheme is uppercase. For example
  173 if url[:i].lower() == 'http': # optimize the common case

• The code as is returns the same results (as far as I've tested it) for both:
  urlsplit("http://github.com/atodorov/repo.git?param=value#myfragment")
  urlsplit("HTTP://github.com/atodorov/repo.git?param=value#myfragment")
  urlsplit("HTtP://github.com/atodorov/repo.git?param=value#myfragment")

but the last 2 invocations also go through lines 185-199

• Lines 174-184 are essentially the same as lines 189-199. The only optimization I can see is avoiding the for loop around lines 185-187 which checks for valid characters in the URL scheme and executes only a few loops b/c scheme names are quite short usually.

My personal vote goes for removal of lines 173-184.

Version-Release number of selected component (if applicable):

This is present in both Python 3 and Python 2 on all versions I have access to:

python3-libs-3.4.1-16.fc21.x86_64.rpm
python-libs-2.7.8-5.fc21.x86_64.rpm
python-libs-2.7.5-16.el7.x86_64.rpm
python-libs-2.6.6-52.el6.x86_64

Versions are from Fedora Rawhide and RHEL. Also the same code is present in the Mercurial repository.

Bug first reported as
https://bugzilla.redhat.com/show_bug.cgi?id=1160603

and now filing here for upstream consideration.

[orsenthil]

Thanks for filing this. Does test cases pass after removal of those lines? (I will be surprised) and if the tests fail, then after analyzing it, this could only be considered a new change (thus change to made in latest python) in order not to break compatibility.

[atodorov]

Does test cases pass after removal of those lines? (I will be surprised)

Yes they do. Also see my detailed explanation above, there's no reason for test cases to fail.

[vadmium]

The patch seems sensible. The only behaviour change I can forsee would be the odd case of http:1234 no longer being parsed like this:

>>> urlsplit("http:1234")
SplitResult(scheme='http', netloc='', path='1234', query='', fragment='')

Instead it would be parsed the same as HTTP:1234 (or tel:1234!):

>>> urlsplit("HTTP:1234")
SplitResult(scheme='', netloc='', path='HTTP:1234', query='', fragment='')

If optimizing for “http:” really is important, it might still be done without the code duplication. Other options might be factoring a subroutine, using str.strip(), set.issubset(), or regular expressions.

[vadmium]

bpo-27657 has been opened about the quirk with numeric URLs

[timgraham]

I sent a pull request for this issue's dependency (bpo-27657) and added a second commit there with this patch. A test added in the first commit demonstrates that removing this code no longer results in the behavior change described in msg238254.

[orsenthil]

New changeset 5a88d50 by Senthil Kumaran (Tim Graham) in branch 'master':
bpo-27657: Fix urlparse() with numeric paths (#661)
5a88d50

[miss-islington]

New changeset 82b5f6b by Miss Islington (bot) in branch '3.7':
bpo-27657: Fix urlparse() with numeric paths (GH-661)
82b5f6b

[orsenthil]

New changeset 0f3187c by Senthil Kumaran in branch '3.8':
[3.8] bpo-27657: Fix urlparse() with numeric paths (GH-661) (bpo-16839)
0f3187c

[gpshead]

There are still a lot of cleanliness and even logical issues lurking within urllib.parse. But this bug is dated and other changes have happened since. I'm closing it as out of date. Open new issues for specific action items.