Urllib2 authentication memory.

[bradobro]

BPO	7159
Nosy	@terryjreedy, @orsenthil, @bitdancer, @axitkhurana, @vadmium
Files	auth.patch auth10.patch prior_auth_docs.patch auth11.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = <Date 2016-05-24.16:41:29.167>
created_at = <Date 2009-10-17.15:32:38.254>
labels = ['type-feature', 'library']
title = 'Urllib2 authentication memory.'
updated_at = <Date 2016-05-24.16:41:29.166>
user = 'https://bugs.python.org/bradobro'

bugs.python.org fields:

activity = <Date 2016-05-24.16:41:29.166>
actor = 'terry.reedy'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2016-05-24.16:41:29.167>
closer = 'terry.reedy'
components = ['Library (Lib)']
creation = <Date 2009-10-17.15:32:38.254>
creator = 'bradobro'
dependencies = []
files = ['39045', '39062', '39074', '39075']
hgrepos = []
issue_num = 7159
keywords = ['patch']
message_count = 12.0
messages = ['94180', '236620', '240367', '241136', '241180', '241182', '241258', '241259', '241260', '241271', '266231', '266246']
nosy_count = 11.0
nosy_names = ['terry.reedy', 'orsenthil', 'dstanek', 'r.david.murray', 'cvrebert', 'bradobro', 'BreamoreBoy', 'python-dev', 'axitkhurana', 'martin.panter', 'mvolz']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue7159'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[bradobro]

For each request requiring HTTP authentication, urllib2 submits the
request without authentication, receives the server's 401
error/challenge, then re-submits the request with authentication.

This is compliant behavior. The problem comes in that urllib2 repeats
this for every ensuing request to the same namespace.

At times this is just an inefficiency--every request gets sent twice,
often with POST data (which can be sizeable).

Sometimes, especially with large POST bodies, this causes a connection
failure.

(Mercurial suffers greatly from this (and I have suggested workaround to
that team.)

This isn't non-compliant behavior, but RFC2617 (sections 2, 3.3)
suggests that once an HTTP client authenticates, it pre-emptively send
authentication with ensuing requests.

More analysis and fix suggestions at
bitbucket.org/bradobro/liquidhg/wiki/Home.

[BreamoreBoy]

@senthil what is your opinion of this?

[bitdancer]

See also bpo-19494. I think this would be a new feature, and it may be that it should leverge the feature added in bpo-19494. The difference here is that we are proposing to allow it to happen automatically after the initial 401, whereas in 19494 we send it pre-emptively even on the first call.

[axitkhurana]

Adding a patch for Python 3+

Some notes:

• Adding a new password manager to handle this case

• The new handler added in bpo-19494 had couple of issues

  • test passes even if we use the old handler in added test
  • uses request.host instead of request.full_url as key for password

• The new handler did assume realm = None before and still does.

I'm using the same logic for adding keys for adding authenticated urls/realm as for login credentials in basic auth handler.

[bitdancer]

Added some review comments.

I think the urllib documentation does not really explain how to *use* these classes, and it should, but that is a separate issue.

[axitkhurana]

Updated patch with review changes

[bitdancer]

Here is the doc patch for the design that Akshit and I agreed to after re-consideration of the API. This eliminates the HTTPBasicPriorAuthHandler in favor of the HTTPPasswdMgrWithPriorAuth password manager and putting support for prior auth on the AbstractBasicAuthHandler based on whether or not the password_mgr supports the is_authenticated and updated_authenticated methods. This redesign means that the Proxy handler automatically gets support for prior auth.

[bitdancer]

Actually attaching the patch this time.