gh-98739: Update libexpat from 2.4.9 to 2.5.0#98742

scdub · 2022-10-26T21:03:07Z

Based this PR on the recent earlier upgrade at #97006 by @corona10

Issue: Upgrade bundled expat to 2.5.0 #98739

bedevere-bot · 2022-10-26T21:03:12Z

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

scdub · 2022-10-27T02:02:46Z

@hartwork I'm not sure if there are other changes that you think would be necessary for Python to upgrade to your latest release, but here's a first attempt at to get Python synced up. I believe there is a constellation of security fixes which Python will be requiring in the near term (SQLite, OpenSSL, ...) and this could potentially tag along for one of those releases.

hartwork · 2022-10-27T14:47:47Z

@scdub I don't really understand your message. Can you maybe rephrase your question for me to better understand?

hartwork

Verified using this Dockerfile:

# Copyright (c) 2022 Sebastian Pipping <sebastian@pipping.org>
# Licensed under the Apache License version 2.0

FROM alpine
RUN apk add --update \
            diffutils \
            git \
            sed \
        && \
    git clone --depth 1 https://github.com/python/cpython cpython-main \
        && \
    ( cd cpython-main && git rev-parse HEAD ) \
        && \
    git clone --depth 1 --branch expat-2.5.0 https://github.com/scdub/cpython scdub-expat-2-5-0 \
        && \
    ( cd scdub-expat-2-5-0 && git rev-parse HEAD ) \
        && \
    git config --global advice.detachedHead false \
        && \
    git clone --depth 1 --branch R_2_4_9 https://github.com/libexpat/libexpat libexpat_2_4_9 \
        && \
    git clone --depth 1 --branch R_2_5_0 https://github.com/libexpat/libexpat libexpat_2_5_0 \
        && \
    diff -r -u libexpat_2_4_9/expat/lib/ cpython-main/Modules/expat/ | tee 2-4-9.diff \
        && \
    diff -r -u libexpat_2_5_0/expat/lib/ scdub-expat-2-5-0/Modules/expat/ | tee 2-5-0.diff \
        && \
    sed -e '/^Only in /d' -e '/^\(+++\|---\) /d' -e '/^diff /d' -i 2-4-9.diff 2-5-0.diff \
        && \
    diff -u 2-4-9.diff 2-5-0.diff \
        && \
    echo 'Diff is good.'

scdub · 2022-10-27T17:52:25Z

@scdub I don't really understand your message. Can you maybe rephrase your question for me to better understand?

I applogize, I wasn't clear. I noticed that with 2.5.0, there is a new EXPAT_CONFIG_H definition for preventing multiple inclusions and wasn't sure if there was any changes required for pyexpatns.h or they can remain independent approaches.

hartwork · 2022-10-27T18:16:18Z

@scdub thanks for elaborating. The added EXPAT_CONFIG_H protection originally came up in relation to refactoring the test suite that triggered compiler warnings on redefining existing macros. I think it can safely be ignored in context of CPython.

scdub · 2022-10-27T18:26:55Z

@gpshead I'm not sure if this PR is of interest to you, but saw that you did initiate the work for the previous expat upgrade to 2.4.9.

gpshead · 2022-10-27T20:43:25Z

Confirmed that these match the libexpat 2.5.0 upstream sources from SHA256(expat-2.5.0.tar.xz)= ef2420f0232c087801abf705e89ae65f6257df6b7931d37846a193ef2e8cdcbe

with our pyexpatns.h addition and that this adds no new C APIs that need including in that "namespace" header hack.

miss-islington · 2022-10-27T20:45:16Z

Thanks @scdub for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8, 3.9, 3.10, 3.11.
🐍🍒⛏🤖

bedevere-bot · 2022-10-27T20:45:33Z

GH-98784 is a backport of this pull request to the 3.11 branch.

bedevere-bot · 2022-10-27T20:45:37Z

GH-98785 is a backport of this pull request to the 3.10 branch.

* Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com> (cherry picked from commit 3e07f82) Co-authored-by: Shaun Walbridge <46331011+scdub@users.noreply.github.com>

bedevere-bot · 2022-10-27T20:45:43Z

GH-98786 is a backport of this pull request to the 3.9 branch.

bedevere-bot · 2022-10-27T20:45:49Z

GH-98787 is a backport of this pull request to the 3.8 branch.

* Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com> (cherry picked from commit 3e07f82) Co-authored-by: Shaun Walbridge <46331011+scdub@users.noreply.github.com>

bedevere-bot · 2022-10-27T20:45:55Z

GH-98788 is a backport of this pull request to the 3.7 branch.

* Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com> (cherry picked from commit 3e07f82) Co-authored-by: Shaun Walbridge <46331011+scdub@users.noreply.github.com>

bedevere-bot · 2022-10-27T21:57:50Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Fedora Stable Clang Installed 3.11 has failed when building commit c5f3f29.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1004/builds/393) and take a look at the build logs.
Check if the failure is related to this commit (c5f3f29) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1004/builds/393

Failed tests:

test_nntplib

Failed subtests:

setUpClass - test.test_nntplib.NetworkedNNTP_SSLTests
test_with_statement - test.test_nntplib.NetworkedNNTP_SSLTests.test_with_statement

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

416 tests OK.

1 test failed:
test_nntplib

17 tests skipped:
test_asdl_parser test_check_c_globals test_clinic test_devpoll
test_gdb test_ioctl test_kqueue test_launcher test_msilib
test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
test_winreg test_winsound test_zipfile64

1 re-run test:
test_nntplib

Total duration: 2 min 41 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/test/test_nntplib.py", line 252, in wrapped
    meth(self)
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/test/test_nntplib.py", line 298, in test_with_statement
    if re.search(r'(?i)KEY.TOO.SMALL', ssl_err.reason):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/re/__init__.py", line 176, in search
    return _compile(pattern, flags).search(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'NoneType'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/test/test_nntplib.py", line 286, in test_with_statement
    server = self.NNTP_CLASS(self.NNTP_HOST, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 1028, in __init__
    super().__init__(host, port, user, password, readermode,
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 337, in __init__
    self.sock = self._create_socket(timeout)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 1034, in _create_socket
    sock = _encrypt_on(sock, self.ssl_context, self.host)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 295, in _encrypt_on
    return context.wrap_socket(sock, server_hostname=hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 1075, in _create
    self.do_handshake()
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 1346, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:992)


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
KeyError: '/psm_d193db14'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
KeyError: '/psm_f4548318'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/test/test_nntplib.py", line 335, in setUpClass
    if re.search(r'(?i)KEY.TOO.SMALL', ssl_err.reason):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/re/__init__.py", line 176, in search
    return _compile(pattern, flags).search(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'NoneType'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
KeyError: '/psm_5be85ea9'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/test/test_nntplib.py", line 332, in setUpClass
    cls.server = cls.NNTP_CLASS(cls.NNTP_HOST, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 1028, in __init__
    super().__init__(host, port, user, password, readermode,
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 337, in __init__
    self.sock = self._create_socket(timeout)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 1034, in _create_socket
    sock = _encrypt_on(sock, self.ssl_context, self.host)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 295, in _encrypt_on
    return context.wrap_socket(sock, server_hostname=hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 1075, in _create
    self.do_handshake()
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 1346, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:992)

Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com> (cherry picked from commit 3e07f82)

* Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com>

Update libexpat from 2.4.9 to 2.5.0

a73f8ee

bedevere-bot added the awaiting review label Oct 26, 2022

📜🤖 Added by blurb_it.

3db91f1

hartwork mentioned this pull request Oct 27, 2022

Release Expat 2.5.0 libexpat/libexpat#667

Closed

27 tasks

hartwork approved these changes Oct 27, 2022

View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels Oct 27, 2022

gpshead added type-security A security issue release-blocker topic-XML needs backport to 3.7 needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes labels Oct 27, 2022

gpshead self-assigned this Oct 27, 2022

gpshead approved these changes Oct 27, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Oct 27, 2022

gpshead merged commit 3e07f82 into python:main Oct 27, 2022

bedevere-bot removed the awaiting merge label Oct 27, 2022

bedevere-bot removed the needs backport to 3.11 only security fixes label Oct 27, 2022

bedevere-bot removed the needs backport to 3.10 only security fixes label Oct 27, 2022

bedevere-bot removed the needs backport to 3.9 label Oct 27, 2022

bedevere-bot removed the needs backport to 3.8 label Oct 27, 2022

bedevere-bot removed the needs backport to 3.7 label Oct 27, 2022

illia-v mentioned this pull request Dec 26, 2022

Create a GitHub Actions workflow for verifying bundled libexpat files #100538

Open

This was referenced Feb 13, 2024

Please upgrade bundled Expat to 2.6.0 (e.g. for the fix to CVE-2023-52425) #115399

Closed

gh-115399: Upgrade bundled libexpat to 2.6.0 #115431

Merged

[3.11] Upgrade bundled libexpat to 2.6.0 (GH-115399) #115468

Merged

Uh oh!

Conversation

scdub commented Oct 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Oct 26, 2022

Uh oh!

scdub commented Oct 27, 2022

Uh oh!

hartwork commented Oct 27, 2022

Uh oh!

hartwork left a comment

Choose a reason for hiding this comment

Uh oh!

scdub commented Oct 27, 2022

Uh oh!

hartwork commented Oct 27, 2022

Uh oh!

scdub commented Oct 27, 2022

Uh oh!

gpshead commented Oct 27, 2022

Uh oh!

miss-islington commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

scdub commented Oct 26, 2022 •

edited

Loading