http.client.HTTPConnection.putrequest encode error

[MiZou]

BPO	17214
Nosy	@orsenthil, @tiran, @ezio-melotti, @berkerpeksag, @vadmium, @vajrasky
Files	patch_to_urllib_handle_non_ascii_char_in_url.txt issue17214.patch issue17214.redirect.v2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-05-16.09:44:51.646>
created_at = <Date 2013-02-16.10:52:39.782>
labels = ['type-bug', 'library']
title = 'http.client.HTTPConnection.putrequest encode  error'
updated_at = <Date 2016-05-16.09:44:51.644>
user = 'https://bugs.python.org/MiZou'

bugs.python.org fields:

activity = <Date 2016-05-16.09:44:51.644>
actor = 'martin.panter'
assignee = 'none'
closed = True
closed_date = <Date 2016-05-16.09:44:51.646>
closer = 'martin.panter'
components = ['Library (Lib)']
creation = <Date 2013-02-16.10:52:39.782>
creator = 'Mi.Zou'
dependencies = []
files = ['30964', '30976', '40907']
hgrepos = []
issue_num = 17214
keywords = ['patch']
message_count = 17.0
messages = ['182216', '182218', '182679', '193183', '193279', '193286', '193306', '193312', '193346', '193352', '239648', '253409', '253410', '253771', '265521', '265682', '265690']
nosy_count = 11.0
nosy_names = ['orsenthil', 'christian.heimes', 'ezio.melotti', 'python-dev', 'berker.peksag', 'martin.panter', 'Mi.Zou', 'vajrasky', 'LDTech', 'Uche Ogbuji', 'Strecke']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17214'
versions = ['Python 3.5', 'Python 3.6']

[MiZou]

while urllib following the redirection(302):
urllib.client.HTTPConnection.putrequest raise an error:
#

---

File "D:\Program Files\Python32\lib\http\client.py", line 1004, in _send_request
self.putrequest(method, url, **skips)
File "D:\Program Files\Python32\lib\http\client.py", line 868, in putrequest
self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 108-111: ordinal not in range(128)
#----------------------------------------------------------
in the sourcode i found that:
at line 811
def putrequest(self, method, url, skip_host=0,skip_accept_encoding=0)...
the argument url may be a unicode,and it was unquoted..

i think we should replace:
request = '%s %s %s' (method,url,self._http_vsn_str)
with:
import urllib.parse
request = '%s %s %s' (method,urllib.parse.quote(url),self._http_vsn_str)

[MiZou]

while urllib following the redirection(302):
http.client.HTTPConnection.putrequest raise an error:
#

---

...
File "D:\Program Files\Python32\lib\http\client.py", line 1004, in _send_request
self.putrequest(method, url, **skips)
File "D:\Program Files\Python32\lib\http\client.py", line 868, in putrequest
self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 108-111: ordinal not in range(128)
#----------------------------------------------------------
in the sourcode i found that:
at line 811

def putrequest(self, method, url, skip_host=0,skip_accept_en...)
...

the argument url may be a unicode,and it was unquoted..
----------------------------note----------------------------------------
in my case:
...
purl="http://bbs.dospy.com/1111258attachdown.php?aid=14361277&bbsid=349"
req=urllib.request.Request(purl,headers=headers)
response=urllib.request.urlopen(req)
...

then,the http serve redirect me to a file download url...
and the url contains some Chinese word....
i have print out the argument url:

/f/1ba1f70606223af2aa5c3aeff6c6a46a/511f7b4c/day_111015/20111015_5949e996881b2e28403d26Ch6dOfj6LZ.rar/p/ÒâÁÖ03-08.part1.rar

[terryjreedy]

Please give us

1. the exact Python version used. 3.2.3? or something earlier?
2. A minimal but complete example that we can run. What is 'headers'?
3. The complete traceback, not just the last two entries.
4. The result of running with the newer 3.3.0, if you possibly can. Perhaps the problem has already been fixed.

While line numbers have changed, even in 3.2.4 in repository, 3.2-3.4 all have

    request = '%s %s %s' % (method, url, self._http_vsn_str)
    # Non-ASCII characters should have been eliminated earlier
    self._output(request.encode('ascii'))

Since there is nothing earlier in the function that would eliminate non-ascii, there must be an assumption about what happens earlier in the call chain. That might have already been fixed, which is why we need an example to test.

[LDTech]

This problem still exist in Python 3.3.2. The following code gives you an example:

import urllib.request
url = "http://www.libon.it/libon/search/isbn/3499155443"
req = urllib.request.Request(url)
response = urllib.request.urlopen(req, timeout=30)
the_page = response.read().decode('utf-8')
print(the_page)

Traceback (most recent call last):
  File "C:\X\webpy.py", line 4, in <module>
    response = urllib.request.urlopen(req, timeout=30)
  File "C:\Python33\lib\urllib\request.py", line 156, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Python33\lib\urllib\request.py", line 475, in open
    response = meth(req, response)
  File "C:\Python33\lib\urllib\request.py", line 587, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python33\lib\urllib\request.py", line 507, in error
    result = self._call_chain(*args)
  File "C:\Python33\lib\urllib\request.py", line 447, in _call_chain
    result = func(*args)
  File "C:\Python33\lib\urllib\request.py", line 692, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python33\lib\urllib\request.py", line 469, in open
    response = self._open(req, data)
  File "C:\Python33\lib\urllib\request.py", line 487, in _open
    '_open', req)
  File "C:\Python33\lib\urllib\request.py", line 447, in _call_chain
    result = func(*args)
  File "C:\Python33\lib\urllib\request.py", line 1268, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "C:\Python33\lib\urllib\request.py", line 1248, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "C:\Python33\lib\http\client.py", line 1061, in request
    self._send_request(method, url, body, headers)
  File "C:\Python33\lib\http\client.py", line 1089, in _send_request
    self.putrequest(method, url, **skips)
  File "C:\Python33\lib\http\client.py", line 953, in putrequest
    self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 78-79: ordinal not in range(128)

[vajrasky]

The script for demonstrating bug can be simplified to:

-----------------------------------------------------------------------

import urllib.request
url = "http://www.libon.it/ricerca/7817940/3499155443/dettaglio/3102314/Onkel-Oswald-und-der-Sudan-Käfer/order/date_desc"

req = urllib.request.Request(url)
response = urllib.request.urlopen(req, timeout=30)
the_page = response.read().decode('utf-8')
print(the_page)

---

Attached the simple patch to solve this problem.

The question is whether we should fix this problem in urllib or not because strictly speaking the url should be ascii characters only. But if the Firefox can open this url, why not urllib?

I will contemplate about this problem and if I (or other people) think that urllib should handle url containing non-ascii characters, then I will add additional unit test.

Until then, people can use third party package, which is
request package from http://docs.python-requests.org/en/latest/

----------------------------------------------------------------

r = requests.get("http://www.libon.it/ricerca/7817940/3499155443/dettaglio/3102314/Onkel-Oswald-und-der-Sudan-Käfer/order/date_desc")
print(r.text)

---

[tiran]

The problem may not be a bug but a deliberate design choice. urllib is rather low level and doesn't implement some browser magic. Browsers handle stuff like 'ä' -> '%C3%A4', ' ' -> '%20' or IDNA but urllib doesn't. I always saw it as may responsibility to quote and encode everything myself. Higher level APIs such as requests are free to implement browser magic.

Contrary to common believes an URL with an umlaut or space is *not* a valid URI. From http://docs.python.org/3/library/urllib.request.html#urllib.request.Request

url should be a string containing a valid URL.

I suggest that this ticket shall be closed as "won't fix".