Multiple type confusions in unicode error handlers

[pkt]

BPO	24102
Nosy	@malemburg, @doerwalter, @vstinner, @tiran, @ezio-melotti, @serhiy-storchaka
Files	poc_unicode_errors.py codecs_error_handlers_issubclass.patch codecs_error_handlers_issubclass_2.patch codecs_error_handlers_issubclass_3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-05-18.13:13:20.206>
created_at = <Date 2015-05-01.14:14:06.688>
labels = ['extension-modules', 'interpreter-core', 'expert-unicode', 'type-crash']
title = 'Multiple type confusions in unicode error handlers'
updated_at = <Date 2015-05-18.13:13:38.627>
user = 'https://bugs.python.org/pkt'

bugs.python.org fields:

activity = <Date 2015-05-18.13:13:38.627>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-05-18.13:13:20.206>
closer = 'serhiy.storchaka'
components = ['Extension Modules', 'Interpreter Core', 'Unicode']
creation = <Date 2015-05-01.14:14:06.688>
creator = 'pkt'
dependencies = []
files = ['39253', '39265', '39266', '39287']
hgrepos = []
issue_num = 24102
keywords = ['patch']
message_count = 9.0
messages = ['242319', '242391', '242393', '242395', '242397', '242398', '242556', '243476', '243477']
nosy_count = 9.0
nosy_names = ['lemburg', 'doerwalter', 'vstinner', 'christian.heimes', 'ezio.melotti', 'Arfrever', 'python-dev', 'serhiy.storchaka', 'pkt']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue24102'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[pkt]

# Breakpoint 1, PyUnicodeEncodeError_GetEnd (exc=<X at remote 0x405730e4>, end=0xbf9e8f7c) at Objects/exceptions.c:1643
\bpo-1643 PyObject *obj = get_unicode(((PyUnicodeErrorObject *)exc)->object,
# (gdb) s
# get_unicode (attr=<unknown at remote 0x8c6a120>, name=0x82765ea "object") at Objects/exceptions.c:1516
\bpo-1516 if (!attr) {
# (gdb) print *attr
# $4 = {_ob_next = 0xfefefefe, _ob_prev = 0xfefefefe, ob_refcnt = -16843010, ob_type = 0xfefefefe}
# (gdb) c
# Continuing.
#
# Program received signal SIGSEGV, Segmentation fault.
# 0x080bc7d9 in get_unicode (attr=<unknown at remote 0x8cbe250>, name=0x82765ea "object") at Objects/exceptions.c:1521
\bpo-1521 if (!PyUnicode_Check(attr)) {

# Type confusion. IsInstance check is ineffective because of custom
# __getattribute__ method. Contents of string instance is interpreted as
# an exception object.

[serhiy-storchaka]

Here is simpler reproducer:

import codecs

class X(str):
    __class__ = UnicodeEncodeError

codecs.ignore_errors(X())

The problem is that PyObject_IsInstance() is fooled by custom __class__, but then builtin error handlers handle error object as having UnicodeEncodeError layout, while it doesn't.

Here is a patch that fixes the issue by using PyObject_IsSubclass() of exc->ob_type instead of PyObject_IsInstance().

[doerwalter]

The patch does indeed fix the segmentation fault. However the exception message looks confusing:

TypeError: don't know how to handle UnicodeEncodeError in error callback

[serhiy-storchaka]

Here is a patch that makes error message consistent with type checking.

[doerwalter]

Looks much better. However shouldn't:

exc->ob_type->tp_name

be:

Py_TYPE(exc)->tp_name

(although there are still many spots in the source that still use ob_type->tp_name)

[serhiy-storchaka]

Py_TYPE() is necessary when the argument is not of type PyObject* (e.g. PyUnicodeObject*).

[serhiy-storchaka]

Also fixed handling errors of PyObject_IsSubclass() (bpo-24115) in the _codecs module.

[python-dev]

New changeset 547bc11e3357 by Serhiy Storchaka in branch '2.7':
Issue bpo-24102: Fixed exception type checking in standard error handlers.
https://hg.python.org/cpython/rev/547bc11e3357

New changeset 68eaa9409818 by Serhiy Storchaka in branch '3.4':
Issue bpo-24102: Fixed exception type checking in standard error handlers.
https://hg.python.org/cpython/rev/68eaa9409818

New changeset 510819e5855e by Serhiy Storchaka in branch 'default':
Issue bpo-24102: Fixed exception type checking in standard error handlers.
https://hg.python.org/cpython/rev/510819e5855e

[serhiy-storchaka]

Greg Ewing suggested to use PyObject_TypeCheck (http://permalink.gmane.org/gmane.comp.python.devel/153216).