expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)

[NatanaelCopa]

BPO	29591
Nosy	@birkenfeld, @vstinner, @larryhastings, @tiran, @benjaminp, @jkloth, @ned-deily, @matrixise, @yan12125
PRs	bpo-29591: Upgrade Modules/expat to libexpat 2.2 #2164 [3.6] bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) #2200 [3.5] bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) #2201 [2.7] bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) #2202 [3.4] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (#2164) #2203 [3.3] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (#2164) #2204 bpo-29591: [2.7] Update VS project files #2310 bpo-30726: PCbuild _elementtree: remove duplicate defines #2348 [3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) #2349 [3.5] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) #2350
Files	rebuild_expat_dir.sh

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2017-07-12.12:35:46.638>
created_at = <Date 2017-02-17.15:39:39.676>
labels = ['type-security', 'expert-XML', '3.7']
title = 'expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)'
updated_at = <Date 2019-05-10.18:12:37.639>
user = 'https://bugs.python.org/NatanaelCopa'

bugs.python.org fields:

activity = <Date 2019-05-10.18:12:37.639>
actor = 'ned.deily'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2017-07-12.12:35:46.638>
closer = 'larry'
components = ['XML']
creation = <Date 2017-02-17.15:39:39.676>
creator = 'Natanael Copa'
dependencies = []
files = ['46949']
hgrepos = []
issue_num = 29591
keywords = []
message_count = 35.0
messages = ['288014', '288016', '288017', '288018', '288296', '295841', '295924', '296045', '296046', '296047', '296048', '296050', '296051', '296067', '296255', '296537', '296552', '296574', '296579', '296585', '296689', '296713', '296716', '297932', '298157', '298178', '298179', '298180', '298182', '298206', '298209', '298210', '298215', '298216', '298428']
nosy_count = 11.0
nosy_names = ['georg.brandl', 'vstinner', 'larry', 'christian.heimes', 'benjamin.peterson', 'jkloth', 'ned.deily', 'jeremy.kloth', 'matrixise', 'yan12125', 'Natanael Copa']
pr_nums = ['2164', '2200', '2201', '2202', '2203', '2204', '2310', '2348', '2349', '2350']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue29591'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[NatanaelCopa]

cpython bundles expat in Modules/expat/ and needs to be updated to expat-2.2.0 to fix various security vulnerabilities.

21 June 2016, Expat 2.2.0 released.
Release 2.2.0 includes security & other bug fixes.

Security fixes

CVE-2016-0718 (issue 537)
Fix crash on malformed input

CVE-2016-4472
Improve insufficient fix to CVE-2015-1283 / CVE-2015-2716 introduced with Expat 2.1.1

CVE-2016-5300 (issue 499)
Use more entropy for hash initialization than the original fix to CVE-2012-0876

CVE-2012-6702 (issue 519)
Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

Fix should be applied to all maintained python branches.

[vstinner]

CVE-2012-6702 (issue 519)
Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-6702

Extract of Modules/pyexpat.c:
---

#if ((XML_MAJOR_VERSION >= 2) && (XML_MINOR_VERSION >= 1)) || defined(XML_HAS_SET_HASH_SALT)
    /* This feature was added upstream in libexpat 2.1.0.  Our expat copy
     * has a backport of this feature where we also define XML_HAS_SET_HASH_SALT
     * to indicate that we can still use it. */
    XML_SetHashSalt(self->itself,
                    (unsigned long)_Py_HashSecret.prefix);
#endif

---

Python 2.7, 3.5, 3.6 and 3.7 have this call at least (I didn't check other versions).

[vstinner]

You may want to look also at https://pypi.python.org/pypi/defusedxml

[tiran]

CVE-2016-0718 and CVE-2016-4472 might be relevant for Python. CVE-2016-5300 and CVE-2012-6702 are irrelevant. As Victor already pointed out, Python seeds libexpat from a good CPRNG.

[vstinner]

I'm working on a new documentation of Python vulnerabilities to help to handle such issue:
http://python-security.readthedocs.io/en/latest/vulnerabilities.html

[ned-deily]

Note that a duplicate of this issue was opened as bpo-30610 and @matrixise was working on a PR there to update the embedded expat to 2.2.0. Since there are CVE's and a demo crash supplied in bpo-30610, it seems to me we need to fix this for 3.6.2rc1 so I'm making this a "release blocker" and delaying the release. I'm willing to be convinced otherwise. Christian or Victor, can one of you please follow up on this for the 3.6 branch ASAP? Thanks!

[vstinner]

I upgraded Modules/expat/ to expat 2.2 using attached rebuild_expat_dir.sh script:
#2164

TODO: Should be done later in the master branch, once the security fix is handled.

• Drop support for VMS? VMS support removed from Python 3.4. Remove Modules/expat/expat_config.h
• Drop support for the Open Watcom compiler? Compiler not supported by Python. Remove Modules/expat/watcomconfig.h
• Send Python downstream changes to expat upstream?

[vstinner]

New changeset 23ec4b5 by Victor Stinner in branch 'master':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (bpo-2164)
23ec4b5

[vstinner]

Python 3.3 currently embeds a copy of libexpat 2.1.0, wheras other branches have libexpat 2.1.1:
http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html