itertools.groupby() can fail a C assert()

[arigo]

BPO	30347
Nosy	@arigo, @rhettinger, @serhiy-storchaka
PRs	bpo-30347: Fixed a crash in itertools.groupby() when use the grouper #1557 [3.6] bpo-30347: Stop crashes when concurrently iterate over itertools.groupby() iterators. (GH-1557) #3770 [2.7] bpo-30347: Stop crashes when concurrently iterate over itertools.groupby() iterators. (GH-1557). #3772

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2017-09-26.20:16:27.494>
created_at = <Date 2017-05-11.21:10:21.469>
labels = ['extension-modules', '3.7', 'type-crash']
title = 'itertools.groupby() can fail a C assert()'
updated_at = <Date 2017-09-26.20:16:27.493>
user = 'https://github.com/arigo'

bugs.python.org fields:

activity = <Date 2017-09-26.20:16:27.493>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2017-09-26.20:16:27.494>
closer = 'serhiy.storchaka'
components = ['Extension Modules']
creation = <Date 2017-05-11.21:10:21.469>
creator = 'arigo'
dependencies = []
files = []
hgrepos = []
issue_num = 30347
keywords = ['patch']
message_count = 8.0
messages = ['293517', '293519', '293549', '293579', '302948', '303062', '303065', '303071']
nosy_count = 3.0
nosy_names = ['arigo', 'rhettinger', 'serhiy.storchaka']
pr_nums = ['1557', '3770', '3772']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue30347'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

[arigo]

This triggers an assert() failure on debug-mode Python (or a leak in release Python):

from itertools import groupby

def f(n):
    print("enter:", n)
    if n == 5:
        list(b)
    print("leave:", n)
    return n != 6

for (k, b) in groupby(range(10), f):
    print(list(b))

With current trunk we get: python: ./Modules/itertoolsmodule.c:303: _grouper_next: Assertion `gbo->currkey == NULL' failed.

[rhettinger]

Do you have a suggested fix?

[serhiy-storchaka]

PR 1557 implements the simplest fix -- remove the assert and use Py_XSETREF() instead of simple assignment. Since the resulting code in _grouper_next() is identical to the code in groupby_next(), it was shared between these two functions.

[rhettinger]

At first glance, the suggested PR looks fine. Unless there is a rush, I would like to hold-off spend more time thinking about bpo-30346 before moving forward with this one.

[serhiy-storchaka]

Hmm, the sole bpo-30346 is not enough for fixing this crash. I don't know if there is a simpler fix based on bpo-30346, but since PR 1557 removes the code duplication I'm going to merge it in any case.

[serhiy-storchaka]

New changeset c740e4f by Serhiy Storchaka in branch 'master':
bpo-30347: Stop crashes when concurrently iterate over itertools.groupby() iterators. (bpo-1557)
c740e4f

[serhiy-storchaka]

New changeset 69b2dc8 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-30347: Stop crashes when concurrently iterate over itertools.groupby() iterators. (GH-1557) (bpo-3770)
69b2dc8