bpo-44394: Update libexpat copy to 2.4.1 (GH-26945) (GH-28032)

Update the vendored copy of libexpat to 2.4.1 (from 2.2.8) to get the
fix for the CVE-2013-0340 "Billion Laughs" vulnerability. This copy
is most used on Windows and macOS.

Co-authored-by: Łukasz Langa <[email protected]>
(cherry picked from commit 3fc5d84)

Co-authored-by: Victor Stinner <[email protected]>

Author: miss-islington

Doc/library/xml.rst

@@ -60,22 +60,26 @@ circumvent firewalls.
 The following table gives an overview of the known attacks and whether
 the various modules are vulnerable to them.
 
-=========================  ==============   ===============   ==============   ==============   ==============
-kind                       sax              etree             minidom          pulldom          xmlrpc
-=========================  ==============   ===============   ==============   ==============   ==============
-billion laughs             **Vulnerable**   **Vulnerable**    **Vulnerable**   **Vulnerable**   **Vulnerable**
-quadratic blowup           **Vulnerable**   **Vulnerable**    **Vulnerable**   **Vulnerable**   **Vulnerable**
-external entity expansion  Safe (4)         Safe    (1)       Safe    (2)      Safe (4)         Safe    (3)
-`DTD`_ retrieval           Safe (4)         Safe              Safe             Safe (4)         Safe
-decompression bomb         Safe             Safe              Safe             Safe             **Vulnerable**
-=========================  ==============   ===============   ==============   ==============   ==============
-
-1. :mod:`xml.etree.ElementTree` doesn't expand external entities and raises a
+=========================  ==================  ==================  ==================  ==================  ==================
+kind                       sax                 etree               minidom             pulldom             xmlrpc
+=========================  ==================  ==================  ==================  ==================  ==================
+billion laughs             **Vulnerable** (1)  **Vulnerable** (1)  **Vulnerable** (1)  **Vulnerable** (1)  **Vulnerable** (1)
+quadratic blowup           **Vulnerable** (1)  **Vulnerable** (1)  **Vulnerable** (1)  **Vulnerable** (1)  **Vulnerable** (1)
+external entity expansion  Safe (5)            Safe (2)            Safe (3)            Safe (5)            Safe (4)
+`DTD`_ retrieval           Safe (5)            Safe                Safe                Safe (5)            Safe
+decompression bomb         Safe                Safe                Safe                Safe                **Vulnerable**
+=========================  ==================  ==================  ==================  ==================  ==================
+
+1. Expat 2.4.1 and newer is not vulnerable to the "billion laughs" and
+   "quadratic blowup" vulnerabilities. Items still listed as vulnerable due to
+   potential reliance on system-provided libraries. Check
+   :data:`pyexpat.EXPAT_VERSION`.
+2. :mod:`xml.etree.ElementTree` doesn't expand external entities and raises a
    :exc:`ParserError` when an entity occurs.
-2. :mod:`xml.dom.minidom` doesn't expand external entities and simply returns
+3. :mod:`xml.dom.minidom` doesn't expand external entities and simply returns
    the unexpanded entity verbatim.
-3. :mod:`xmlrpclib` doesn't expand external entities and omits them.
-4. Since Python 3.7.1, external general entities are no longer processed by
+4. :mod:`xmlrpclib` doesn't expand external entities and omits them.
+5. Since Python 3.7.1, external general entities are no longer processed by
    default.
 
 

Misc/NEWS.d/next/Security/2021-06-29-02-45-53.bpo-44394.A220N1.rst

@@ -0,0 +1,3 @@
+Update the vendored copy of libexpat to 2.4.1 (from 2.2.8) to get the fix
+for the CVE-2013-0340 "Billion Laughs" vulnerability. This copy is most used
+on Windows and macOS.

Modules/expat/COPYING

@@ -1,5 +1,5 @@
 Copyright (c) 1998-2000 Thai Open Source Software Center Ltd and Clark Cooper
-Copyright (c) 2001-2017 Expat maintainers
+Copyright (c) 2001-2019 Expat maintainers
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

Modules/expat/ascii.h

@@ -6,8 +6,11 @@
                         \___/_/\_\ .__/ \__,_|\__|
                                  |_| XML parser
 
-   Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
-   Copyright (c) 2000-2017 Expat development team
+   Copyright (c) 1999-2000 Thai Open Source Software Center Ltd
+   Copyright (c) 2000      Clark Cooper <[email protected]>
+   Copyright (c) 2002      Fred L. Drake, Jr. <[email protected]>
+   Copyright (c) 2007      Karl Waclawek <[email protected]>
+   Copyright (c) 2017      Sebastian Pipping <[email protected]>
    Licensed under the MIT license:
 
    Permission is  hereby granted,  free of charge,  to any  person obtaining

Modules/expat/asciitab.h

@@ -7,7 +7,9 @@
                                  |_| XML parser
 
    Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
-   Copyright (c) 2000-2017 Expat development team
+   Copyright (c) 2000      Clark Cooper <[email protected]>
+   Copyright (c) 2002      Fred L. Drake, Jr. <[email protected]>
+   Copyright (c) 2017      Sebastian Pipping <[email protected]>
    Licensed under the MIT license:
 
    Permission is  hereby granted,  free of charge,  to any  person obtaining

Modules/expat/expat.h

@@ -7,7 +7,14 @@
                                  |_| XML parser
 
    Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
-   Copyright (c) 2000-2017 Expat development team
+   Copyright (c) 2000      Clark Cooper <[email protected]>
+   Copyright (c) 2000-2005 Fred L. Drake, Jr. <[email protected]>
+   Copyright (c) 2001-2002 Greg Stein <[email protected]>
+   Copyright (c) 2002-2016 Karl Waclawek <[email protected]>
+   Copyright (c) 2016-2021 Sebastian Pipping <[email protected]>
+   Copyright (c) 2016      Cristian Rodríguez <[email protected]>
+   Copyright (c) 2016      Thomas Beutlich <[email protected]>
+   Copyright (c) 2017      Rhodri James <[email protected]>
    Licensed under the MIT license:
 
    Permission is  hereby granted,  free of charge,  to any  person obtaining
@@ -115,7 +122,11 @@ enum XML_Error {
   XML_ERROR_RESERVED_PREFIX_XMLNS,
   XML_ERROR_RESERVED_NAMESPACE_URI,
   /* Added in 2.2.1. */
-  XML_ERROR_INVALID_ARGUMENT
+  XML_ERROR_INVALID_ARGUMENT,
+  /* Added in 2.3.0. */
+  XML_ERROR_NO_BUFFER,
+  /* Added in 2.4.0. */
+  XML_ERROR_AMPLIFICATION_LIMIT_BREACH
 };
 
 enum XML_Content_Type {
@@ -318,7 +329,7 @@ typedef void(XMLCALL *XML_EndDoctypeDeclHandler)(void *userData);
 
    For internal entities (<!ENTITY foo "bar">), value will
    be non-NULL and systemId, publicID, and notationName will be NULL.
-   The value string is NOT nul-terminated; the length is provided in
+   The value string is NOT null-terminated; the length is provided in
    the value_length argument. Since it is legal to have zero-length
    values, do not use this argument to test for internal entities.
 
@@ -513,7 +524,7 @@ typedef struct {
    Otherwise it must return XML_STATUS_ERROR.
 
    If info does not describe a suitable encoding, then the parser will
-   return an XML_UNKNOWN_ENCODING error.
+   return an XML_ERROR_UNKNOWN_ENCODING error.
 */
 typedef int(XMLCALL *XML_UnknownEncodingHandler)(void *encodingHandlerData,
                                                  const XML_Char *name,
@@ -707,7 +718,7 @@ XML_GetBase(XML_Parser parser);
 /* Returns the number of the attribute/value pairs passed in last call
    to the XML_StartElementHandler that were specified in the start-tag
    rather than defaulted. Each attribute/value pair counts as 2; thus
-   this correspondds to an index into the atts array passed to the
+   this corresponds to an index into the atts array passed to the
    XML_StartElementHandler.  Returns -1 if parser == NULL.
 */
 XMLPARSEAPI(int)
@@ -716,7 +727,7 @@ XML_GetSpecifiedAttributeCount(XML_Parser parser);
 /* Returns the index of the ID attribute passed in the last call to
    XML_StartElementHandler, or -1 if there is no ID attribute or
    parser == NULL.  Each attribute/value pair counts as 2; thus this
-   correspondds to an index into the atts array passed to the
+   corresponds to an index into the atts array passed to the
    XML_StartElementHandler.
 */
 XMLPARSEAPI(int)
@@ -997,7 +1008,10 @@ enum XML_FeatureEnum {
   XML_FEATURE_SIZEOF_XML_LCHAR,
   XML_FEATURE_NS,
   XML_FEATURE_LARGE_SIZE,
-  XML_FEATURE_ATTR_INFO
+  XML_FEATURE_ATTR_INFO,
+  /* Added in Expat 2.4.0. */
+  XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFAULT,
+  XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAULT
   /* Additional features must be added to the end of this enum. */
 };
 
@@ -1010,12 +1024,24 @@ typedef struct {
 XMLPARSEAPI(const XML_Feature *)
 XML_GetFeatureList(void);
 
+#ifdef XML_DTD
+/* Added in Expat 2.4.0. */
+XMLPARSEAPI(XML_Bool)
+XML_SetBillionLaughsAttackProtectionMaximumAmplification(
+    XML_Parser parser, float maximumAmplificationFactor);
+
+/* Added in Expat 2.4.0. */
+XMLPARSEAPI(XML_Bool)
+XML_SetBillionLaughsAttackProtectionActivationThreshold(
+    XML_Parser parser, unsigned long long activationThresholdBytes);
+#endif
+
 /* Expat follows the semantic versioning convention.
    See http://semver.org.
 */
 #define XML_MAJOR_VERSION 2
-#define XML_MINOR_VERSION 2
-#define XML_MICRO_VERSION 8
+#define XML_MINOR_VERSION 4
+#define XML_MICRO_VERSION 1
 
 #ifdef __cplusplus
 }

Modules/expat/expat_external.h

@@ -7,7 +7,14 @@
                                  |_| XML parser
 
    Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
-   Copyright (c) 2000-2017 Expat development team
+   Copyright (c) 2000      Clark Cooper <[email protected]>
+   Copyright (c) 2000-2004 Fred L. Drake, Jr. <[email protected]>
+   Copyright (c) 2001-2002 Greg Stein <[email protected]>
+   Copyright (c) 2002-2006 Karl Waclawek <[email protected]>
+   Copyright (c) 2016      Cristian Rodríguez <[email protected]>
+   Copyright (c) 2016-2019 Sebastian Pipping <[email protected]>
+   Copyright (c) 2017      Rhodri James <[email protected]>
+   Copyright (c) 2018      Yury Gribov <[email protected]>
    Licensed under the MIT license:
 
    Permission is  hereby granted,  free of charge,  to any  person obtaining

Modules/expat/iasciitab.h

@@ -7,7 +7,9 @@
                                  |_| XML parser
 
    Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
-   Copyright (c) 2000-2017 Expat development team
+   Copyright (c) 2000      Clark Cooper <[email protected]>
+   Copyright (c) 2002      Fred L. Drake, Jr. <[email protected]>
+   Copyright (c) 2017      Sebastian Pipping <[email protected]>
    Licensed under the MIT license:
 
    Permission is  hereby granted,  free of charge,  to any  person obtaining

Modules/expat/internal.h

@@ -25,8 +25,12 @@
                         \___/_/\_\ .__/ \__,_|\__|
                                  |_| XML parser
 
-   Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
-   Copyright (c) 2000-2017 Expat development team
+   Copyright (c) 2002-2003 Fred L. Drake, Jr. <[email protected]>
+   Copyright (c) 2002-2006 Karl Waclawek <[email protected]>
+   Copyright (c) 2003      Greg Stein <[email protected]>
+   Copyright (c) 2016-2021 Sebastian Pipping <[email protected]>
+   Copyright (c) 2018      Yury Gribov <[email protected]>
+   Copyright (c) 2019      David Loffredo <[email protected]>
    Licensed under the MIT license:
 
    Permission is  hereby granted,  free of charge,  to any  person obtaining
@@ -101,22 +105,58 @@
 #  endif
 #endif
 
+#include <limits.h> // ULONG_MAX
+
+#if defined(_WIN32) && ! defined(__USE_MINGW_ANSI_STDIO)
+#  define EXPAT_FMT_ULL(midpart) "%" midpart "I64u"
+#  if defined(_WIN64) // Note: modifiers "td" and "zu" do not work for MinGW
+#    define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "I64d"
+#    define EXPAT_FMT_SIZE_T(midpart) "%" midpart "I64u"
+#  else
+#    define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
+#    define EXPAT_FMT_SIZE_T(midpart) "%" midpart "u"
+#  endif
+#else
+#  define EXPAT_FMT_ULL(midpart) "%" midpart "llu"
+#  if ! defined(ULONG_MAX)
+#    error Compiler did not define ULONG_MAX for us
+#  elif ULONG_MAX == 18446744073709551615u // 2^64-1
+#    define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "ld"
+#    define EXPAT_FMT_SIZE_T(midpart) "%" midpart "lu"
+#  else
+#    define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
+#    define EXPAT_FMT_SIZE_T(midpart) "%" midpart "u"
+#  endif
+#endif
+
 #ifndef UNUSED_P
 #  define UNUSED_P(p) (void)p
 #endif
 
+/* NOTE BEGIN If you ever patch these defaults to greater values
+              for non-attack XML payload in your environment,
+              please file a bug report with libexpat.  Thank you!
+*/
+#define EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFAULT   \
+  100.0f
+#define EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAULT    \
+  8388608 // 8 MiB, 2^23
+/* NOTE END */
+
+#include "expat.h" // so we can use type XML_Parser below
+
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-#ifdef XML_ENABLE_VISIBILITY
-#  if XML_ENABLE_VISIBILITY
-__attribute__((visibility("default")))
-#  endif
+void _INTERNAL_trim_to_complete_utf8_characters(const char *from,
+                                                const char **fromLimRef);
+
+#if defined(XML_DTD)
+unsigned long long testingAccountingGetCountBytesDirect(XML_Parser parser);
+unsigned long long testingAccountingGetCountBytesIndirect(XML_Parser parser);
+const char *unsignedCharToPrintable(unsigned char c);
 #endif
-void
-_INTERNAL_trim_to_complete_utf8_characters(const char *from,
-                                           const char **fromLimRef);
 
 #ifdef __cplusplus
 }

Modules/expat/latin1tab.h

@@ -7,7 +7,9 @@
                                  |_| XML parser
 
    Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
-   Copyright (c) 2000-2017 Expat development team
+   Copyright (c) 2000      Clark Cooper <[email protected]>
+   Copyright (c) 2002      Fred L. Drake, Jr. <[email protected]>
+   Copyright (c) 2017      Sebastian Pipping <[email protected]>
    Licensed under the MIT license:
 
    Permission is  hereby granted,  free of charge,  to any  person obtaining