Update embeded copy of libexpat from 2.2.1 to 2.2.3

[vstinner]

BPO	30947
Nosy	@gpshead, @vstinner, @larryhastings, @tiran, @ned-deily, @serhiy-storchaka
PRs	bpo-30947: Update libexpat from 2.2.1 to 2.2.3 #3106 [3.6] bpo-30947: Update libexpat from 2.2.1 to 2.2.3 (#3106) #3143 [3.5] bpo-30947: Update libexpat from 2.2.1 to 2.2.3 (#3106) #3144 [2.7] bpo-30947: Update libexpat from 2.2.1 to 2.2.3 (#3106) #3145 [3.3] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3352 [3.4][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3353 [3.5][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3354
Files	cpython_rebuild_expat_dir.sh

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-09-25.08:26:11.982>
created_at = <Date 2017-07-17.14:18:15.790>
labels = ['type-security', '3.7']
title = 'Update embeded copy of libexpat from 2.2.1 to 2.2.3'
updated_at = <Date 2019-05-10.18:02:34.803>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-05-10.18:02:34.803>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2017-09-25.08:26:11.982>
closer = 'vstinner'
components = []
creation = <Date 2017-07-17.14:18:15.790>
creator = 'vstinner'
dependencies = []
files = ['47088']
hgrepos = []
issue_num = 30947
keywords = []
message_count = 22.0
messages = ['298525', '298528', '298529', '300365', '300367', '300368', '300369', '300414', '300437', '300535', '300547', '300548', '301269', '301423', '302834', '302899', '302924', '302929', '325910', '325922', '325925', '325934']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'vstinner', 'larry', 'christian.heimes', 'ned.deily', 'serhiy.storchaka']
pr_nums = ['3106', '3143', '3144', '3145', '3352', '3353', '3354']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue30947'
versions = ['Python 3.7']

[vstinner]

libexpat released a new version 2.2.2 which seems to contain 2 or 3 security fixes. I'm not sure that Python is affected by these bugs.

https://github.com/libexpat/libexpat/blob/R_2_2_2/expat/Changes#L5

Release 2.2.2 Wed July 12 2017
Security fixes:
#43 Protect against compilation without any source of high
quality entropy enabled, e.g. with CMake build system;
commit ff0207e6076e9828e536b8d9cd45c9c92069b895
#60 Windows with _UNICODE:
Unintended use of LoadLibraryW with a non-wide string
resulted in failure to load advapi32.dll and degradation
in quality of used entropy when compiled with _UNICODE for
Windows; you can launch existing binaries with
EXPAT_ENTROPY_DEBUG=1 in the environment to inspect the
quality of entropy used during runtime; commits
* 95b95032f907ef1cd17ee7a9a1768010a825d61d
* 73a5a2e9c081f49f2d775cf7ced864158b68dc80
[MOX-006] Fix non-NULL parser parameter validation in XML_Parse;
resulted in NULL dereference, previously;
commit ac256dafdffc9622ab0dc2c62fcecb0dfcfa71fe

    Bug fixes:
         #69  Fix improper use of unsigned long long integer literals

    Other changes:
         #73  Start requiring a C99 compiler
         #49  Fix "==" Bashism in configure script
         #50  Fix too eager getrandom detection for Debian GNU/kFreeBSD
         #52    and macOS
         #51  Address lack of stdint.h in Visual Studio 2003 to 2008
         #58  Address compile warnings
         #68  Fix "./buildconf.sh && ./configure" for some versions
                of Dash for /bin/sh
         #72  CMake: Ease use of Expat in context of a parent project
                with multipe CMakeLists.txt files
         #72  CMake: Resolve mistaken executable permissions
         #76  Address compile warning with -DNDEBUG (not recommended!)
         #77  Address compile warning about macro redefinition

    Special thanks to:
        Alexander Bluhm
        Ben Boeckel
        Cătălin Răceanu
        Kerin Millar
        László Böszörményi
        S. P. Zeidler
        Segev Finer
        Václav Slavík
        Victor Stinner
        Viktor Szakats
             and

Radically Open Security

--

Previous issue for expat 2.2.1: issue bpo-30694.

[vstinner]

#51 Address lack of stdint.h in Visual Studio 2003 to 2008

FYI this change only impacts Python 2.7, since Python 3.3 and newer requires Visual Studio 2010 or newer, and I already backported (cherry-picked) this specific commit in Python 2.7:
https://github.com/python/cpython/pull/2312/commits

#58 Address compile warnings

That's my small contribution, so coming from CPython :-)
libexpat/libexpat#58

#76 Address compile warning with -DNDEBUG (not recommended!)

Nice contributions from Segev Finer, coming from CPython ;-)
libexpat/libexpat#76

#77 Address compile warning about macro redefinition

Another contribution of Segev Finer, already fixed downstream (in Python):
libexpat/libexpat#77

[vstinner]

About the 3 security fixes (is the last change a security fix?).

"""
#43 Protect against compilation without any source of high
quality entropy enabled, e.g. with CMake build system;
commit ff0207e6076e9828e536b8d9cd45c9c92069b895
"""

Since Python uses its own entropy source, I don't think that this change impacts us.

libexpat/libexpat@ff0207e

"""
#60 Windows with _UNICODE:
Unintended use of LoadLibraryW with a non-wide string
resulted in failure to load advapi32.dll and degradation
in quality of used entropy when compiled with _UNICODE for
Windows; you can launch existing binaries with
EXPAT_ENTROPY_DEBUG=1 in the environment to inspect the
quality of entropy used during runtime; commits
* 95b95032f907ef1cd17ee7a9a1768010a825d61d
* 73a5a2e9c081f49f2d775cf7ced864158b68dc80
"""

I don't understand the consequence of this specific bug.

libexpat/libexpat@95b9503
libexpat/libexpat@73a5a2e

"""
[MOX-006] Fix non-NULL parser parameter validation in XML_Parse;
resulted in NULL dereference, previously;
commit ac256dafdffc9622ab0dc2c62fcecb0dfcfa71fe
"""

I'm not sure that it's possible to call XML_Parse() with NULL in Python.

libexpat/libexpat@ac256da

[vstinner]

Expat 2.2.3 was released:

Release 2.2.3 Wed August 2 2017
Security fixes:
#82 CVE-2017-11742 -- Windows: Fix DLL hijacking vulnerability
using Steve Holme's LoadLibrary wrapper for/of cURL

    Bug fixes:
         #85  Fix a dangling pointer issue related to realloc

    Other changes:
              Increase code coverage
         #91  Linux: Allow getrandom to fail if nonblocking pool has not
                yet been initialized and read /dev/urandom then, instead.
                This is in line with what recent Python does.
         #81  Pre-10.7/Lion macOS: Support entropy from arc4random
         #86  Check that a UTF-16 encoding in an XML declaration has the
                right endianness
    #4 #5 #7  Recover correctly when some reallocations fail
              Repair "./configure && make" for systems without any
                provider of high quality entropy
                and try reading /dev/urandom on those
              Ensure that user-defined character encodings have converter
                functions when they are needed
              Fix mis-leading description of argument -c in xmlwf.1
              Rely on macro HAVE_ARC4RANDOM_BUF (rather than __CloudABI__)
                for CloudABI
        #100  Fix use of SIPHASH_MAIN in siphash.h
         #23  Test suite: Fix memory leaks
              Version info bumped from 7:4:6 to 7:5:6

    Special thanks to:
        Chanho Park
        Joe Orton
        Pascal Cuoq
        Rhodri James
        Simon McVittie
        Vadim Zeitlin
        Viktor Szakats
             and

Core Infrastructure Initiative

[vstinner]

Previous update: bpo-30694.

[vstinner]

cpython_rebuild_expat_dir.sh: Script used to update Modules/expat/ to 2.2.3. The script now uses the libexpat Git repository. Previously, I used tarballs.

[vstinner]

#82 CVE-2017-11742 -- Windows: Fix DLL hijacking vulnerability using Steve Holme's LoadLibrary wrapper for/of cURL

libexpat/libexpat#82

I don't think that this bug affects Python since Python sets a hash secret.

[serhiy-storchaka]

Could the updating script be added into the CPython repository?

[vstinner]

If libexpat is upgraded in Python 2.7, the new Modules/expat/loadlibrary.c should also be added to PC/VS9.0/ project files, as I did for PCbuild.

Note: PC/VS7.1/ and PC/VS8.0/ are likely broken and don't need to be updated, right?

[vstinner]

New changeset 93d0cb5 by Victor Stinner in branch 'master':
bpo-30947: Update libexpat from 2.2.1 to 2.2.3 (bpo-3106)
93d0cb5

[vstinner]

New changeset 83e37e1 by Victor Stinner in branch '3.6':
bpo-30947: Update libexpat from 2.2.1 to 2.2.3 (bpo-3106) (bpo-3143)
83e37e1

[vstinner]

New changeset ec4ab09 by Victor Stinner in branch '2.7':
bpo-30947: Update libexpat from 2.2.1 to 2.2.3 (bpo-3106) (bpo-3145)
ec4ab09