assertion failure in Python/ast.c in case of a bad unicodedata.normalize()

[orenmn]

BPO	31592
Nosy	@brettcannon, @ncoghlan, @vstinner, @benjaminp, @serhiy-storchaka, @1st1, @orenmn
PRs	bpo-31592: Fix an assertion failure in Python/ast.c in case of a bad unicodedata.normalize() #3767 [3.6] bpo-31592: Fix an assertion failure in Python parser in case of a bad unicodedata.normalize(). (GH-3767) #3836

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-09-30.19:54:36.492>
created_at = <Date 2017-09-26.12:54:08.971>
labels = ['interpreter-core', '3.7', 'type-crash']
title = 'assertion failure in Python/ast.c in case of a bad unicodedata.normalize()'
updated_at = <Date 2017-09-30.19:54:36.491>
user = 'https://github.com/orenmn'

bugs.python.org fields:

activity = <Date 2017-09-30.19:54:36.491>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2017-09-30.19:54:36.492>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2017-09-26.12:54:08.971>
creator = 'Oren Milman'
dependencies = []
files = []
hgrepos = []
issue_num = 31592
keywords = ['patch']
message_count = 4.0
messages = ['303036', '303043', '303416', '303420']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'ncoghlan', 'vstinner', 'benjamin.peterson', 'serhiy.storchaka', 'yselivanov', 'Oren Milman']
pr_nums = ['3767', '3836']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue31592'
versions = ['Python 3.6', 'Python 3.7']

[orenmn]

The following code causes an assertion failure:

import unicodedata
def bad_normalize(*args):
    return None

unicodedata.normalize = bad_normalize
import ast
ast.parse('\u03D5')

This is because init_normalization() (in Python/ast.c) assumes that
unicodedata.normalize() is valid, and stores it in the compiling struct.
Later, new_identifier() calls the stored function, assumes it returned a
string, and passes it to PyUnicode_InternInPlace(), which asserts it is a
string.

[serhiy-storchaka]

Oh, and passing arguments to this function is tricky. If faked unicodedata.normalize save the args tuple it will see a mutated tuple containing a hanging reference.

This is a good opportunity of using the fast call protocol.

[serhiy-storchaka]

New changeset 7dc46d8 by Serhiy Storchaka (Oren Milman) in branch 'master':
bpo-31592: Fix an assertion failure in Python parser in case of a bad unicodedata.normalize(). (bpo-3767)
7dc46d8

[serhiy-storchaka]

New changeset a4dfe1c by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31592: Fix an assertion failure in Python parser in case of a bad unicodedata.normalize(). (GH-3767) (bpo-3836)
a4dfe1c