urllib2 fails against IIS 6.0 (No support for MD5-sess auth)

[bwmcadams]

BPO	2202
Nosy	@birkenfeld, @orsenthil, @kristjanvalur, @tiran, @cjw296, @bitdancer, @berkerpeksag, @vadmium, @deronnax
Files	urllib2-support_md5-sess.diff: add support for MD5-sess issue2202_py27.patch md5-sess_not_implem_cur.diff md5-sess_not_implem_27.diff md5-sess_not_implem_27_v2.diff: '%s' changed to %r md5-sess_not_implem_cur_v2.diff: '%s' changed to %r digest_md5sess_unittest.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = <Date 2016-03-06.14:31:42.530>
created_at = <Date 2008-02-28.18:26:33.408>
labels = ['type-bug', 'library']
title = 'urllib2 fails against IIS 6.0 (No support for MD5-sess auth)'
updated_at = <Date 2016-03-08.12:48:52.897>
user = 'https://bugs.python.org/bwmcadams'

bugs.python.org fields:

activity = <Date 2016-03-08.12:48:52.897>
actor = 'deronnax'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2016-03-06.14:31:42.530>
closer = 'berker.peksag'
components = ['Library (Lib)']
creation = <Date 2008-02-28.18:26:33.408>
creator = 'bwmcadams'
dependencies = []
files = ['15964', '21619', '36724', '36725', '41949', '41950', '42045']
hgrepos = []
issue_num = 2202
keywords = ['patch']
message_count = 33.0
messages = ['63101', '63102', '63108', '98115', '109874', '133535', '133540', '138847', '138848', '138849', '138855', '138865', '138866', '138869', '138992', '138995', '192499', '220429', '220433', '227540', '227541', '227884', '260423', '260437', '260438', '260448', '260971', '260972', '261163', '261252', '261253', '261254', '261356']
nosy_count = 14.0
nosy_names = ['georg.brandl', 'jjlee', 'orsenthil', 'kristjan.jonsson', 'christian.heimes', 'cjw296', 'bwmcadams', 'dieresys', 'r.david.murray', 'santoso.wijaya', 'python-dev', 'berker.peksag', 'martin.panter', 'deronnax']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue2202'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[bwmcadams]

URLLIB2 seems to have issues attempting to digest authenticate against a
Windows-based IIS 6.x server. The IIS server requests MD5-sess support,
and URLLIB2 throws an exception that ... far from explains this:

Traceback (most recent call last):
  File "/usr/lib64/python2.5/runpy.py", line 95, in run_module
    filename, loader, alter_sys)
  File "/usr/lib64/python2.5/runpy.py", line 52, in _run_module_code
    mod_name, mod_fname, mod_loader)
  File "/usr/lib64/python2.5/runpy.py", line 32, in _run_code
    exec code in run_globals
  File "/srv/pubPal/test/test_auth.py", line 16, in <module>
    print opener.open(uri)
  File "/usr/lib64/python2.5/urllib2.py", line 380, in open
    response = meth(req, response)
  File "/usr/lib64/python2.5/urllib2.py", line 491, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.5/urllib2.py", line 412, in error
    result = self._call_chain(*args)
  File "/usr/lib64/python2.5/urllib2.py", line 353, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.5/urllib2.py", line 992, in http_error_401
    host, req, headers)
  File "/usr/lib64/python2.5/urllib2.py", line 884, in http_error_auth_reqed
    return self.retry_http_digest_auth(req, authreq)
  File "/usr/lib64/python2.5/urllib2.py", line 889, in
retry_http_digest_auth
    auth = self.get_authorization(req, chal)
  File "/usr/lib64/python2.5/urllib2.py", line 920, in get_authorization
    H, KD = self.get_algorithm_impls(algorithm)
  File "/usr/lib64/python2.5/urllib2.py", line 972, in get_algorithm_impls
    return H, KD
UnboundLocalError: local variable 'H' referenced before assignment

The offending code is urllib2.py line 972:

    def get_algorithm_impls(self, algorithm):
        # lambdas assume digest modules are imported at the top level
        if algorithm == 'MD5':
            H = lambda x: hashlib.md5(x).hexdigest()
        elif algorithm == 'SHA':
            H = lambda x: hashlib.sha1(x).hexdigest()
        # XXX MD5-sess
        KD = lambda s, d: H("%s:%s" % (s, d))
        return H, KD

I'm not sure what's meant by the # XXX MD5-sess comment there... But
what ends up happening is that MD5-sess matches neither the if or the
elif, and when return attempts to return, H is undefined.

This could be easily patched, but support for MD5-sess would be great as
well.

In my estimation, at the least, urllib2 shouldn't crap out in such an
obscure way when encountering an unsupported algorithm.

I'd suggest changing line 970 (The # XXX MD5-sess line) to:

raise Exception, "Unsupported Digest Authentication Algorithm '%s'" %
(algorithm)

[bwmcadams]

Sorry, this isn't a crash. It doesn't crash the interpreter. I'll
assume "behavior' is the correct categorization.

[akuchling]

Marking as easy, assuming you add the exception; implementing MD5-sess
is probably not easy.

[dieresys]

Here is a patch for supporting MD5-sess, following RFC2617 specification.

Some comments/warnings:

• I've only tested the patch against IIS 6.0. I don't know about other servers supporting MD5-sess.
• IIS 6.0 expects the User Agent to send the URI (in the Authorization header) without the query string.
• This patch doesn't add support for Digest sessions. For each request we make, we get a new [401|407] message with a new nonce (depending if we're talking about a proxy with digest authentication or a web server). Then we generate another authenticated request using that nonce. For Digest sessions to be fully supported, we should be adding an [WWW|Proxy]-Authenticate header in each following request we made to the server using the last nonce. This includes both MD5-sess and MD5 digest implementations.
• A1 is being recalculated for every request. Given the above, this is not a real problem.

I'll open a new ticket regarding Digest sessions.

[BreamoreBoy]

I tried to test the patch but got:-
c:\Python26>python.exe lib\test\test_urllib2_localnet.py
Traceback (most recent call last):
  File "lib\test\test_urllib2_localnet.py", line 325, in <module>
    class DigestMD5sessClientAuthTests(BaseTestCase):
NameError: name 'BaseTestCase' is not defined

[kristjanvalur]

Is this moving anywhere? I was asked to apply this patch locally and am curious as to whether it is moving into the stdlib.

[santosowijaya]

Adding some minor fixes to the original patch to apply cleanly to current 2.7 tip.

Also, add a conditional branch in the case that the server returns none of the supported algorithm strings, to raise a ValueError (the previous patch adds support for 'MD5-sess' but the original non-descript stack trace would be seen again with some other unsupported algorithm string).