PyUnicode_AsDecodedObject can only return unicode now

[zhangyangyu]

BPO	28426
Nosy	@malemburg, @ncoghlan, @vstinner, @larryhastings, @benjaminp, @ezio-melotti, @serhiy-storchaka, @zhangyangyu
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Dependencies	bpo-28526: Use PyUnicode_AsEncodedString instead of PyUnicode_AsEncodedObject
Files	PyUnicode_AsDecodedObject-crash.patch deprecate-str-to-str-coding-unicode-api.patch deprecate-str-to-str-coding-unicode-api-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-10-27.18:17:34.417>
created_at = <Date 2016-10-13.04:30:06.104>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'PyUnicode_AsDecodedObject can only return unicode now'
updated_at = <Date 2017-03-31.16:36:09.529>
user = 'https://github.com/zhangyangyu'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:09.529>
actor = 'dstufft'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-10-27.18:17:34.417>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2016-10-13.04:30:06.104>
creator = 'xiang.zhang'
dependencies = ['28526']
files = ['45070', '45071', '45211']
hgrepos = []
issue_num = 28426
keywords = ['patch']
message_count = 17.0
messages = ['278545', '278546', '278550', '278551', '278552', '278554', '278628', '279133', '279337', '279344', '279363', '279368', '279375', '279378', '279557', '279559', '279718']
nosy_count = 10.0
nosy_names = ['lemburg', 'ncoghlan', 'vstinner', 'larry', 'benjamin.peterson', 'ezio.melotti', 'Arfrever', 'python-dev', 'serhiy.storchaka', 'xiang.zhang']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28426'
versions = ['Python 3.6', 'Python 3.7']

[zhangyangyu]

PyUnicode_AsDecodedObject was added in f46d49e2e0f0 and became an API in 2284fa89ab08. It seems its intention is to return a Python object. But during evolution, with commits 5f11621a6f51 and 123f2dc08b3e, it can only return unicode now, becoming another version of PyUnicode_AsDecodedUnicode. Is this the wanted behaviour?

[serhiy-storchaka]

Indeed. The only difference is that PyUnicode_AsDecodedUnicode fails for most encodings (except rot13), but PyUnicode_AsDecodedObject just crashes in debug build. It seems to me that these functions (as well as PyUnicode_AsEncodedUnicode) shouldn't exist it Python 3. None of these functions are documented. PyUnicode_AsDecodedObject emits Py3k warning in 2.7. PyUnicode_AsDecodedUnicode and PyUnicode_AsEncodedUnicode were added in Python 3 (2284fa89ab08), and the purpose of this is not clear to me. They work only with rot13, but general PyCodec_Decode and PyCodec_Encode can be used instead. Could you please explain Marc-André?

[malemburg]

PyUnicode_AsDecodedObject() and PyUnicode_AsEncodedObject() were meant as C API implementations of the unicode.decode() and unicode.encode() methods in Python2. Not having PyUnicode_AsDecodedObject() documented was likely an oversight on my part.

In Python2, unicode.decode() and unicode.encode() were more or less direct interfaces to the codec registry. In Python 2.7 this was changed to issue a warning for porting to Python 3.

In Python3, the methods were changed to only return unicode objects and to reflect this change without breaking the C API, the new PyUnicode_AsDecodedUnicode() and PyUnicode_AsEncodedUnicode() were added.

I guess the more recent changes simply didn't pay attention to this difference anymore and put restrictions on the output of PyUnicode_AsDecodedObject() and PyUnicode_AsEncodedObject() which were not originally intended, hence the crash you are seeing, Serhiy.

Going forward, C extensions in Python3 could indeed use the PyCodec_*() APIs directly.

[serhiy-storchaka]

Shouldn't all these function be deprecated in favour of PyCodec_*() APIs?

[malemburg]

On 13.10.2016 10:10, Serhiy Storchaka wrote:

Shouldn't all these function be deprecated in favour of PyCodec_*() APIs?

Not all of them, since you still want to have a C API for
unicode.encode(). PyUnicode_AsEncodedUnicode() would have
to stay.

As for the others, I don't know how much use they get.

[serhiy-storchaka]

I propose following:

1. Fix a crash in PyUnicode_AsDecodedObject by removing unicode_result() in all maintained 3.x versions (starting from 3.4? or 3.3?).

2. Deprecate PyUnicode_AsDecodedObject, PyUnicode_AsDecodedUnicode and PyUnicode_AsEncodedUnicode in 3.6, make they always failing in 3.7 and remove them in future versions. They shouldn't be widely used since they are not documented, PyUnicode_AsDecodedObject already is deprecated in 2.7, and the only supported standard encoding is rot13.

[zhangyangyu]

+1 for 2). Patch looks good.

[serhiy-storchaka]

Marc-Andre, what are your thoughts about this?

[malemburg]

As I already mentioned, PyUnicode_AsEncodedUnicode() needs to stay, since it's the C API for unicode.encode(). The others can be deprecated.

[zhangyangyu]

Marc-Andre, shouldn't the C API of unicode.encode() be PyUnicode_AsEncodedString instead of PyUnicode_AsEncodedUnicode now?

BTW Serhiy, how about PyUnicode_AsEncodedObject? Not see it in your deprecate list.

[malemburg]

On 25.10.2016 04:16, Xiang Zhang wrote:

Marc-Andre, shouldn't the C API of unicode.encode() be PyUnicode_AsEncodedString instead of PyUnicode_AsEncodedUnicode now?

You're right. I got confused with all the slight variations.

BTW Serhiy, how about PyUnicode_AsEncodedObject? Not see it in your deprecate list.

Let's see what we have:

PyUnicode_AsEncodedString(): encode to bytes (unicode.encode())
PyUnicode_AsEncodedUnicode(): encode to unicode (for e.g. rot13)
PyUnicode_AsEncodedObject(): encode to whatever the codec returns

codecs.encode() can be used for the last two.

[python-dev]

New changeset 71dce630dc02 by Serhiy Storchaka in branch '3.4':
Issue bpo-28426: Fixed potential crash in PyUnicode_AsDecodedObject() in debug build.
https://hg.python.org/cpython/rev/71dce630dc02

New changeset 74569ecd67e4 by Serhiy Storchaka in branch '3.5':
Issue bpo-28426: Fixed potential crash in PyUnicode_AsDecodedObject() in debug build.
https://hg.python.org/cpython/rev/74569ecd67e4

New changeset 6af1a26e655f by Serhiy Storchaka in branch '3.6':
Issue bpo-28426: Fixed potential crash in PyUnicode_AsDecodedObject() in debug build.
https://hg.python.org/cpython/rev/6af1a26e655f

New changeset 1ab1fd00e9d6 by Serhiy Storchaka in branch 'default':
Issue bpo-28426: Fixed potential crash in PyUnicode_AsDecodedObject() in debug build.
https://hg.python.org/cpython/rev/1ab1fd00e9d6