ImageImage

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Function ssl.create_default_context raises exception on Windows 10 when called with ssl.Purpose.SERVER_AUTH) attribute
Type: behavior Stage: resolved
Components: SSL Versions: Python 3.9
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: MDM-1, christian.heimes, pedja, pervlad, pukkandan
Priority: normal Keywords:

Created on 2019-01-05 12:24 by pervlad, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
cacerts.pem pervlad, 2019-01-07 18:44 pub certs
Untitled.png MDM-1, 2021-09-28 15:39 MUPCA Root - Certificates
Messages (16)
msg333054 - (view) Author: Vladimir Perić (pervlad) Date: 2019-01-05 12:24
In Python 3.7.1 on Windows 10 ssl library function call ssl.create_default_context(ssl.Purpose.SERVER_AUTH) raises an ssl error:

File "C:\Python37\lib\ssl.py", line 471, in _load_windows_store_certs
    self.load_verify_locations(cadata=certs)
ssl.SSLError: nested asn1 error (_ssl.c:3926)

In Python 3.6.4 same function call raises no error.
msg333055 - (view) Author: Vladimir Perić (pervlad) Date: 2019-01-05 13:36
Same outcome in Python 3.7.2.
See first comment for detailed explanation of issue.
msg333158 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-07 14:49
> self.load_verify_locations(cadata=certs)
> ...
> ssl.SSLError: nested asn1 error (_ssl.c:3926)

It seems like one of your certificate is invalid.

> In Python 3.6.4 same function call raises no error.

We frequently update OpenSSL in Python. You can get OpenSSL version using:

$ python3
Python 3.7.2 (default, Jan  3 2019, 09:14:01) 
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.1 FIPS  11 Sep 2018'
>>> ssl.OPENSSL_VERSION_INFO
(1, 1, 1, 0, 15)
>>> ssl.OPENSSL_VERSION_NUMBER
269488143
>>> hex(ssl.OPENSSL_VERSION_NUMBER)
'0x1010100f'
msg333159 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-07 14:52
Would it be possible to attach the certification to the issue so someone can try to reproduce the issue? (but don't attach any private key ;-))
msg333168 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-01-07 15:27
The certs are coming from Windows' trust store. Could you please dump the trust store for me and attach the result to the bug tracker. The following script is untested but should work. I don't have access to a Windows machine at the moment.

ctx = ssl.SSLContext(ssl.PROTOCOL_TLS)
certs = []
for storename in ("CA", "ROOT"):
    certs.append(storename)
    for cert, encoding, trust in ssl.enum_certificates(storename):
        if encoding == "x509_asn":
            if trust is True or ssl.Purpose.SERVER_AUTH.oid in trust:
                try:
                    ctx.load_verify_locations(cadata=cert)
                except Exception as e:
                    certs.append(str(e))
                certs.append(ssl.DER_cert_to_PEM_cert(cert))

with open('cacerts.pem', 'w') as f:
    f.write('\n'.join(certs))
msg333181 - (view) Author: Vladimir Perić (pervlad) Date: 2019-01-07 18:44
Public Certificate file cert.pem is attached.

Version of ssl lib in pythons on my machine:
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.0j  20 Nov 2018'

Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2q  20 Nov 2018'
msg333183 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-01-07 19:53
Your Windows cert store contains multiple invalid certificates. The first failing certificate is the custom "MUPCA Root", which looks like a certificate from http://ca.mup.gov.rs/sertifikati.html. The serial number seems to be badly formated or padded. There is nothing we can do about erroneous and bad certificates.

$ openssl x509 -in ca.pem 
unable to load certificate
140613019477824:error:0D0E20DD:asn1 encoding routines:c2i_ibuf:illegal padding:crypto/asn1/a_int.c:187:
140613019477824:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:crypto/asn1/tasn_dec.c:627:Field=serialNumber, Type=X509_CINF
140613019477824:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:crypto/asn1/tasn_dec.c:627:Field=cert_info, Type=X509
140613019477824:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:crypto/pem/pem_oth.c:33:

$ openssl asn1parse -in ca.pem  
    0:d=0  hl=4 l=1300 cons: SEQUENCE          
    4:d=1  hl=4 l= 764 cons: SEQUENCE          
    8:d=2  hl=2 l=   3 cons: cont [ 0 ]        
   10:d=3  hl=2 l=   1 prim: INTEGER           :02
   13:d=2  hl=2 l=   4 prim: INTEGER           :BAD INTEGER:[00000066]
   19:d=2  hl=2 l=  13 cons: SEQUENCE          
   21:d=3  hl=2 l=   9 prim: OBJECT            :sha1WithRSAEncryption
   32:d=3  hl=2 l=   0 prim: NULL              
   34:d=2  hl=2 l=  83 cons: SEQUENCE          
   36:d=3  hl=2 l=  19 cons: SET               
   38:d=4  hl=2 l=  17 cons: SEQUENCE          
   40:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   45:d=5  hl=2 l=  10 prim: UTF8STRING        :MUPCA Root
   57:d=3  hl=2 l=  29 cons: SET               
   59:d=4  hl=2 l=  27 cons: SEQUENCE          
   61:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
   66:d=5  hl=2 l=  20 prim: UTF8STRING        :MUP Republike Srbije
   88:d=3  hl=2 l=  16 cons: SET               
   90:d=4  hl=2 l=  14 cons: SEQUENCE          
   92:d=5  hl=2 l=   3 prim: OBJECT            :localityName
   97:d=5  hl=2 l=   7 prim: UTF8STRING        :Beograd
  106:d=3  hl=2 l=  11 cons: SET               
  108:d=4  hl=2 l=   9 cons: SEQUENCE          
  110:d=5  hl=2 l=   3 prim: OBJECT            :countryName
  115:d=5  hl=2 l=   2 prim: PRINTABLESTRING   :RS
  119:d=2  hl=2 l=  30 cons: SEQUENCE          
  121:d=3  hl=2 l=  13 prim: UTCTIME           :100227161918Z
  136:d=3  hl=2 l=  13 prim: UTCTIME           :200227161918Z
  ...

$ wget http://ca.mup.gov.rs/MUPCARoot.crt
$ openssl x509 -in MUPCARoot.crt -inform DER
unable to load certificate
140699773712192:error:0D0E20DD:asn1 encoding routines:c2i_ibuf:illegal padding:crypto/asn1/a_int.c:187:
140699773712192:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:crypto/asn1/tasn_dec.c:627:Field=serialNumber, Type=X509_CINF
140699773712192:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:crypto/asn1/tasn_dec.c:627:Field=cert_info, Type=X509
msg333184 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-01-07 20:05
OpenSSL 1.1.0 is more strict than OpenSSL 1.0.2. That's why you don't see the issue with Python 3.6 but with 3.7. The problem is explained in https://mta.openssl.org/pipermail/openssl-dev/2016-February/005100.html

The CA has encoded the integer 102 (0x66) as "02 04 00
00 00 66", which violates the DER standard. The correct encoding is "02 01  66".

>>> from asn1crypto.core import Integer
>>> import binascii
>>> binascii.hexlify(Integer(102).dump())
b'020166'
msg333215 - (view) Author: Vladimir Perić (pervlad) Date: 2019-01-08 09:38
Thank you all for this expeditive help. Sorry for taking your time.
I will remove bad certificates from my machine.
Thanks again.

I will try to close this one.
msg333224 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-01-08 11:45
I also checked how other implementations deal with invalid DER encoding. NSS 3.41, Firefox, and Chromium accept the certifiate.

NSS shows the serial number as "102 (0x66)"
Firefox and Chromium display the serial number as "00:00:00:66".

$ echo "password" > passwd
$ certutil -d . -f passwd -N
$ certutil -d . -f passwd -A -n ca -i ../ca.pem -t C,C,C
$ certutil -d . -L -n ca
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 102 (0x66)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "C=RS,L=Beograd,O=MUP Republike Srbije,CN=MUPCA Root"
        Validity:
            Not Before: Sat Feb 27 16:19:18 2010
            Not After : Thu Feb 27 16:19:18 2020
        Subject: "C=Re...,L=Beograd,O=MUP Republike Srbije,CN=MUPCA Resursi"
        Subject Public Key Info:
            Public Key Algorithm: PKCS #1 RSA Encryption
            RSA Public Key:
                Modulus:
                    ea:69:46:bc:c7:70:00:d5:f5:32:8d:c7:4e:ad:3a:a5:
                    d3:29:7e:a2:46:12:a9:dd:57:75:b1:49:95:80:20:ed:
                    9b:68:6b:e3:c5:55:d8:64:15:68:42:ab:a3:f7:c0:96:
                    37:08:51:cb:05:ca:b5:99:f6:07:a6:8b:f2:cd:d2:f5:
                    d6:16:12:da:bf:a8:0b:9c:45:5d:ac:79:1d:a8:67:47:
                    ee:7f:83:40:f8:58:00:d5:dd:c4:c9:52:1b:d2:f4:ce:
                    e1:fa:8a:66:d3:18:86:1e:ea:fc:0a:8b:b5:ec:49:cd:
                    86:bf:8b:7e:b0:61:81:ec:ea:99:4f:64:82:96:93:9d:
                    ab:80:7d:a7:27:65:00:d4:12:26:98:45:64:7e:76:0b:
                    98:ff:16:50:49:0c:45:20:82:ce:2e:23:a2:65:3a:b7:
                    44:cd:51:00:d9:bf:e3:1f:de:23:1d:57:e9:32:c3:55:
                    f0:24:af:d4:cf:cd:9e:77:1f:19:7e:1c:03:5b:7a:e4:
                    75:84:3b:d4:1d:e9:23:d6:8c:f2:8f:b2:0d:e3:79:df:
                    9e:03:1e:0e:15:5b:7b:0c:dd:6e:4d:82:86:5a:63:79:
                    64:b5:07:79:dd:fd:08:e3:d6:cb:60:01:fd:82:11:59:
                    2c:8d:22:f8:f9:91:59:b1:cd:12:7b:39:6d:08:82:5d
                Exponent: 65537 (0x10001)
        Signed Extensions:
            Name: Certificate Basic Constraints
            Critical: True
            Data: Is a CA with no maximum path length.

            Name: Certificate Key Usage
            Critical: True
            Usages: Certificate Signing
                    CRL Signing

            Name: Authority Information Access
            Method: PKIX CA issuers access method
            Location: 
                URI: "http://ca.mup.gov.rs/MUPCARoot.crt"

            Name: Certificate Subject Key ID
            Data:
                cb:f9:00:a9:b7:b6:c1:6f:44:43:d0:22:ad:fc:0e:6e:
                cc:8f:f6:0f

            Name: Certificate Authority Key Identifier
            Key ID:
                3f:66:b0:0f:66:fb:f0:10:2e:61:a4:6f:ef:2c:95:8a:
                14:72:6f:71

            Name: CRL Distribution Points
            Distribution point:
                URI: "http://ca.mup.gov.rs/MUPCARoot.crl"

    Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
msg360050 - (view) Author: Pedja (pedja) Date: 2020-01-15 12:58
This is still an issue. Serious one. People are unable to just remove this certificate as it is needed for everyday use.

It is reasonable that application does not deal with invalid certificates. 
I can understand application to breaks if one tries to use invalid certificate. 

But this is not that case. Application breaks on just enumerating certificates even if it does not need it at all.

It is advisable for application just to skip and ignore invalid certificate unless it is required for application to work.

Please reconsider action on this issue to prevent application breaking when it is not necessary.
msg402782 - (view) Author: Dimitrije Milović (MDM-1) * Date: 2021-09-28 15:36
Just to ad to the issue, and to further update the importance of those certificates...

I came to this issue (still persistent with all python versions since 3.6) while using yt-dlp: https://github.com/yt-dlp/yt-dlp/issues/1060

I obviously have the SAME problem than the guy in your link since I am from Serbia too, and those certificates "MUPCA Root" are (unfortunately-badly executed) crucial (issued by the ministry of interior - police 🙄) ones to be able too read ID cards and use personal signing certificates, and they're are all valid...
So the option to remove the faulty certificates, is a no go to me (or anyone in Serbia using their ID card - individuals, companies and entrepreneurs like me)...

Please help!
msg402792 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-09-28 16:57
We cannot fix the issue in Python. Please report the problem to OpenSSL and to your government. Either OpenSSL needs to relax its cert parser again or your government has to replace the broken certificates with correct certificates.
msg402794 - (view) Author: Dimitrije Milović (MDM-1) * Date: 2021-09-28 17:17
Maybe better to continue in my newly opened tread https://bugs.python.org/issue45312 since I suppose I wasn't correctly specific (read I am a noob!), and pukkandan was more so.

And my government fixing their certificates?! No chance i hell, they are like this for more of a decade! :smirk:
msg402805 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-09-28 20:49
Could you please open an OpenSSL bug on the projects bug tracker https://github.com/openssl/openssl/ and explain the issue there? They might be able to implement a workaround for the broken certificates or advise you how to handle the invalid certificates.

A workaround for Python would require a major rewrite of the Windows CA store integration. We don't have any capacity to work on that area. Even if we had capacity, a workaround would land in Python 3.11 earliest (October 2022).
msg402809 - (view) Author: pukkandan (pukkandan) Date: 2021-09-28 21:47
> A workaround for Python would require a major rewrite of the Windows CA store integration. We don't have any capacity to work on that area

In theory, the issue can be worked around by simply loading each certificate separately. See https://github.com/yt-dlp/yt-dlp/pull/1118/commits/599ca418ac75ab1c0baf97f184f32ac48aa759ed
History
Date User Action Args
2022-04-11 14:59:10adminsetgithub: 79846
2021-09-28 21:47:58pukkandansetnosy: + pukkandan
messages: + msg402809
2021-09-28 20:49:29christian.heimessetmessages: + msg402805
2021-09-28 17:17:29MDM-1setmessages: + msg402794
2021-09-28 16:57:43christian.heimessetassignee: christian.heimes ->
resolution: third party
messages: + msg402792
2021-09-28 16:54:09christian.heimeslinkissue45312 superseder
2021-09-28 16:54:09christian.heimeslinkissue45312 dependencies
2021-09-28 15:41:47MDM-1setfiles: - Untitled.png
2021-09-28 15:39:56MDM-1setfiles: + Untitled.png
2021-09-28 15:39:05MDM-1setfiles: + Untitled.png
2021-09-28 15:36:59MDM-1setnosy: + MDM-1

messages: + msg402782
versions: + Python 3.9, - Python 3.7
2020-01-15 21:29:37vstinnersetnosy: - vstinner
2020-01-15 12:58:40pedjasetnosy: + pedja
messages: + msg360050
2019-01-08 11:45:02christian.heimessetmessages: + msg333224
2019-01-08 09:38:24pervladsetstatus: open -> closed

messages: + msg333215
stage: resolved
2019-01-07 20:05:02christian.heimessetmessages: + msg333184
2019-01-07 19:53:42christian.heimessetmessages: + msg333183
2019-01-07 18:44:27pervladsetfiles: + cacerts.pem

messages: + msg333181
2019-01-07 15:27:38christian.heimessetmessages: + msg333168
2019-01-07 14:52:14vstinnersetmessages: + msg333159
2019-01-07 14:49:04vstinnersetnosy: + vstinner
messages: + msg333158
2019-01-05 13:36:32pervladsetmessages: + msg333055
2019-01-05 12:24:45pervladcreate