Plistlib: Half of the double width characters are missing when writing binary plist

[kuglee]

BPO	28321
Nosy	@ronaldoussoren, @serhiy-storchaka
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	input.plist: This is the input file output.plist: This is the output file. plistlib-astral-characters.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-10-04.17:10:19.838>
created_at = <Date 2016-09-30.22:57:38.388>
labels = ['3.7', 'type-bug', 'library']
title = 'Plistlib: Half of the double width characters are missing when writing binary plist'
updated_at = <Date 2017-03-31.16:36:39.033>
user = 'https://bugs.python.org/kuglee'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:39.033>
actor = 'dstufft'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-10-04.17:10:19.838>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2016-09-30.22:57:38.388>
creator = 'kuglee'
dependencies = []
files = ['44897', '44898', '44901']
hgrepos = []
issue_num = 28321
keywords = ['patch']
message_count = 3.0
messages = ['277781', '277800', '278062']
nosy_count = 4.0
nosy_names = ['ronaldoussoren', 'python-dev', 'serhiy.storchaka', 'kuglee']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28321'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[kuglee]

I read an emoji character from a plist file. The emoji printed correctly to stdout. However when I dump the file to a binary plist only the half of the emoji was present.

[serhiy-storchaka]

The simplest reproducer:

>>> import plistlib
>>> plistlib.loads(plistlib.dumps('\U0001f40d', fmt=plistlib.FMT_BINARY))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/plistlib.py", line 1006, in loads
    fp, fmt=fmt, use_builtin_types=use_builtin_types, dict_type=dict_type)
  File "/usr/lib/python3.5/plistlib.py", line 997, in load
    return p.parse(fp)
  File "/usr/lib/python3.5/plistlib.py", line 623, in parse
    return self._read_object(self._object_offsets[top_object])
  File "/usr/lib/python3.5/plistlib.py", line 704, in _read_object
    return self._fp.read(s * 2).decode('utf-16be')
  File "/usr/lib/python3.5/encodings/utf_16_be.py", line 16, in decode
    return codecs.utf_16_be_decode(input, errors, True)
UnicodeDecodeError: 'utf-16-be' codec can't decode bytes in position 0-1: unexpected end of data

Proposed patch fixes this issue.

[python-dev]

New changeset 381ef0f08f89 by Serhiy Storchaka in branch '3.5':
Issue bpo-28321: Fixed writing non-BMP characters with binary format in plistlib.
https://hg.python.org/cpython/rev/381ef0f08f89

New changeset 3a7234d04fe9 by Serhiy Storchaka in branch '3.6':
Issue bpo-28321: Fixed writing non-BMP characters with binary format in plistlib.
https://hg.python.org/cpython/rev/3a7234d04fe9

New changeset b6c85e7e558a by Serhiy Storchaka in branch 'default':
Issue bpo-28321: Fixed writing non-BMP characters with binary format in plistlib.
https://hg.python.org/cpython/rev/b6c85e7e558a