urllib.request.getproxies() misparses Windows registry proxy settings

[benrg]

BPO	42627
Nosy	@pfmoore, @tjguk, @zware, @zooba, @corona10, @chrisxiao, @NateScarlet, @CrazyBoyFeng, @keuin
PRs	bpo-42627: Fix wrong parsing of Windows registry proxy settings #26307

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-12-12.20:27:52.875>
labels = ['type-bug', '3.9', '3.10', '3.11', 'library', 'OS-windows']
title = 'urllib.request.getproxies() misparses Windows registry proxy settings'
updated_at = <Date 2021-11-20.14:38:47.145>
user = 'https://bugs.python.org/benrg'

bugs.python.org fields:

activity = <Date 2021-11-20.14:38:47.145>
actor = 'CrazyBoyFeng'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2020-12-12.20:27:52.875>
creator = 'benrg'
dependencies = []
files = []
hgrepos = []
issue_num = 42627
keywords = ['patch']
message_count = 10.0
messages = ['382921', '387468', '388149', '388152', '393927', '394668', '395276', '401215', '401995', '406658']
nosy_count = 11.0
nosy_names = ['paul.moore', 'tim.golden', 'benrg', 'zach.ware', 'steve.dower', 'corona10', 'chrisxiao', 'kotori', 'NateScarlet', 'CrazyBoyFeng', 'keuin']
pr_nums = ['26307']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42627'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[benrg]

If HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyServer contains the string http=host:123;https=host:456;ftp=host:789, then getproxies_registry() should return

{'http': 'http://host:123', 'https': 'http://host:456', 'ftp': 'http://host:789'}

for consistency with WinInet and Chromium, but it actually returns

{'http': 'http://host:123', 'https': 'https://host:456', 'ftp': 'ftp://host:789'}

This bug has existed for a very long time (since Python 2.0.1 if not earlier), but it was exposed recently when urllib3 added support for HTTPS-in-HTTPS proxies in version 1.26. Before that, an https prefix on the HTTPS proxy url was silently treated as http, accidentally resulting in the correct behavior.

There are additional bugs in the treatment of single-proxy strings (the case when the string contains no = character).

The Chromium code for parsing the ProxyServer string can be found here: https://source.chromium.org/chromium/chromium/src/+/refs/tags/89.0.4353.1:net/proxy_resolution/proxy_config.cc;l=86

Below is my attempt at modifying the code from getproxies_registry to approximately match Chromium's behavior. I could turn this into a patch, but I'd like feedback on the corner cases first.

    if '=' not in proxyServer and ';' not in proxyServer:
        # Use one setting for all protocols.
        # Chromium treats this as a separate category, and some software
        # uses the ALL_PROXY environment variable for a similar purpose,
        # so arguably this should be 'all={}'.format(proxyServer),
        # but this is more backward compatible.
        proxyServer = 'http={0};https={0};ftp={0}'.format(proxyServer)

    for p in proxyServer.split(';'):
        # Chromium and WinInet are inconsistent in their treatment of
        # invalid strings with the wrong number of = characters. It
        # probably doesn't matter.
        protocol, addresses = p.split('=', 1)
        protocol = protocol.strip()

        # Chromium supports more than one proxy per protocol. I don't
        # know how many clients support the same, but handling it is at
        # least no worse than leaving the commas uninterpreted.
        for address in addresses.split(','):
            if protocol in {'http', 'https', 'ftp', 'socks'}:
                # See if address has a type:// prefix
                if not re.match('(?:[^/:]+)://', address):
                    if protocol == 'socks':
                        # Chromium notes that the correct protocol here
                        # is SOCKS4, but "socks://" is interpreted
                        # as SOCKS5 elsewhere. I don't know whether
                        # prepending socks4:// here would break code.
                        address = 'socks://' + address
                    else:
                        address = 'http://' + address

            # A string like 'http=foo;http=bar' will produce a
            # comma-separated list, while previously 'bar' would
            # override 'foo'. That could potentially break something.
            if protocol not in proxies:
                proxies[protocol] = address
            else:
                proxies[protocol] += ',' + address

[kotori]

I came across this issue as well.
I checked Microsoft documentations and it seems InternetGetProxyInfo in WinInet is deprecated, while WinHttpGetIEProxyConfigForCurrentUser in WinHTTP will return the exact same string as what it stored registery.

https://docs.microsoft.com/en-US/troubleshoot/windows-client/networking/configure-client-proxy-server-settings-by-registry-file
Also from this documentation, a proxy server could have "http://" prefix, so I guess it could also support "https://" prefix if a user set a https proxy.

[CrazyBoyFeng]

We know Windows reslove:
http=host:port;https=host:port;ftp=host:port;socks=host:port
as:
http=http://host:port;https=http://host:port;ftp=http://host:port;socks=socks://host:port
means:
Using HTTP type proxy for HTTP, HTTPS and FTP requests, but Socks4/4a type proxy for the other TCP requests.
We notice that socks are different from the others.

Now I want to know what if Windows slove this:
http=socks://host:port;ftp=https://host:port;socks=https://host:port
Does it mean using Socks4/4a type proxy for HTTP requests, HTTPS type proxy for FTP requests, and HTTP type proxy for the other TCP requests?
Or just invalid settings?

[CrazyBoyFeng]

I make some black box tests with the HTTPS proxy.
Set system proxy http=https://host:port and start a HTTPS proxy, then IE, Edge (Chromium) and benrg's code (using urllib3) work fine while fetching http://website. Then I shutdown the HTTPS proxy and start a HTTP proxy on the same port, I get SSLError. That's it.

benrg should add your patch as soon as possible.
Too few people actually use HTTPS proxy. Even if there is a bug with HTTPS proxy, it will not be too late to fix it when it is discovered.