Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer)

[tianlynn]

BPO	31170
Nosy	@birkenfeld, @vstinner, @larryhastings, @benjaminp, @ned-deily, @serhiy-storchaka, @hartwork, @tianlynn
PRs	bpo-31170: Update libexpat from 2.2.3 to 2.2.4 #3315 [3.6] bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315) #3350 [2.7] bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315) #3351 [3.3] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3352 [3.4][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3353 [3.5][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3354 bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug #3570 [2.7] bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570) #3745 [3.6] bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570) #3746 [3.5] bpo-31170: Fix inclusion of expat in Windows build projects. #3751 [3.4] bpo-31170: Fix inclusion of expat in Windows build projects #3785
Files	cpython_rebuild_expat_dir.sh

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-01-04.15:52:09.415>
created_at = <Date 2017-08-10.04:48:25.430>
labels = ['type-bug', 'library', 'release-blocker']
title = 'Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer)'
updated_at = <Date 2019-05-10.18:04:35.827>
user = 'https://github.com/tianlynn'

bugs.python.org fields:

activity = <Date 2019-05-10.18:04:35.827>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2018-01-04.15:52:09.415>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2017-08-10.04:48:25.430>
creator = 'tianlynn'
dependencies = []
files = ['47118']
hgrepos = []
issue_num = 31170
keywords = ['patch']
message_count = 26.0
messages = ['300043', '300044', '300268', '300309', '300605', '301011', '301012', '301084', '301270', '301287', '301399', '301422', '301424', '302159', '302483', '302833', '302898', '302925', '302927', '302930', '302931', '302975', '303037', '307254', '309469', '309471']
nosy_count = 8.0
nosy_names = ['georg.brandl', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'serhiy.storchaka', 'sping', 'tianlynn']
pr_nums = ['3315', '3350', '3351', '3352', '3353', '3354', '3570', '3745', '3746', '3751', '3785']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31170'
versions = ['Python 3.4', 'Python 3.5']

[tianlynn]

utf8_toUtf8(const ENCODING *UNUSED_P(enc),
            const char **fromP, const char *fromLim,
            char **toP, const char *toLim)
{
  char *to;
  const char *from;
  const char *fromLimInitial = fromLim;

/* Avoid copying partial characters. */
align_limit_to_full_utf8_characters(*fromP, &fromLim);

for (to = *toP, from = *fromP; (from < fromLim) && (to < toLim); from++, to++)
*to = *from;
*fromP = from;
*toP = to;

  if (fromLim < fromLimInitial)
    return XML_CONVERT_INPUT_INCOMPLETE;
  else if ((to == toLim) && (from < fromLim))
    // <===== Bug is here. In case (to == toLim), it's possible that
    //        from is still pointing to partial character. For example,
    //        a character with 3 bytes (A, B, C) and form is pointing to C.
    //        It means only A and B is copied to output buffer. Next
    //        scanning will start with C which could be considered as invalid
    //        byte and got dropped. After this, only "AB" is kept in memory
    //        and thus it will lead to invalid continuation byte.
    return XML_CONVERT_OUTPUT_EXHAUSTED;
  else
    return XML_CONVERT_COMPLETED;
}

[serhiy-storchaka]

This is not our code. Please use corresponding bug tracker [1] if you have found a bug in Expat.

But I think there is not a bug here.

[1] https://github.com/libexpat/libexpat/issues

[tianlynn]

Reactive this issue as to let you know that libexpat has confirmed and fixed the bug and they are interested in porting the fix to python. Reactive this in case you want to know what's going on and make a decision accordingly. (Sorry, I'm not very familiar with process here)

[hartwork]

For the record, the upstream fix is commit libexpat/libexpat@74a7090 and it will be part of Expat 2.2.4.

[hartwork]

Expat 2.2.4 with a fix has been released now: https://github.com/libexpat/libexpat/releases

[serhiy-storchaka]

I confirm the regression (see bpo-31303 for reproducer). Victor, do you mind to update expat to 2.2.4?

This issue can be classified as a security issue, since a regression was added in security update.

[serhiy-storchaka]

Corresponding Expat issue: libexpat/libexpat#115.

[benjaminp]

We can put expat 2.2.4 in 2.7.14 final.