PEP 479: Change StopIteration handling inside generators

[Rosuav]

BPO	22906
Nosy	@gvanrossum, @rhettinger, @ncoghlan, @abalkin, @scoder, @vstinner, @bitdancer, @ethanfurman, @schlamar, @Rosuav, @berkerpeksag, @serhiy-storchaka, @1st1, @NeilGirdhar
Files	pep0479.patch: POC patch for PEP 479 itertoolsdoc.diff pep0479.patch: Replacement POC patch stopiter.py: Simple demo of new behaviour pep0479.patch: Real patch, creates future directive and all test_pep479.py pep0479.patch pep0479.patch pep0479.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/1st1'
closed_at = <Date 2015-05-09.18:45:52.986>
created_at = <Date 2014-11-20.10:43:22.982>
labels = ['interpreter-core', 'type-feature', 'library', 'release-blocker']
title = 'PEP 479: Change StopIteration handling inside generators'
updated_at = <Date 2015-10-07.01:43:49.577>
user = 'https://github.com/Rosuav'

bugs.python.org fields:

activity = <Date 2015-10-07.01:43:49.577>
actor = 'john.black.3k'
assignee = 'yselivanov'
closed = True
closed_date = <Date 2015-05-09.18:45:52.986>
closer = 'berker.peksag'
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2014-11-20.10:43:22.982>
creator = 'Rosuav'
dependencies = []
files = ['37233', '37297', '37640', '37641', '37646', '39079', '39316', '39317', '39323']
hgrepos = []
issue_num = 22906
keywords = ['patch']
message_count = 70.0
messages = ['231422', '231447', '231478', '231481', '231482', '231483', '231485', '231486', '231487', '231675', '231779', '231826', '231828', '231852', '231853', '231855', '231856', '231857', '231858', '233620', '233621', '233622', '233629', '233640', '233645', '233665', '237223', '237247', '237277', '237278', '238449', '241255', '241290', '241999', '242115', '242630', '242632', '242633', '242698', '242709', '242732', '242733', '242734', '242736', '242740', '242741', '242743', '242746', '242752', '242773', '242785', '242795', '242814', '242815', '242817', '242819', '242820', '242821', '242822', '242823', '242824', '242825', '242828', '242849', '242863', '243578', '243830', '243832', '252429', '252443']
nosy_count = 17.0
nosy_names = ['gvanrossum', 'rhettinger', 'ncoghlan', 'belopolsky', 'scoder', 'vstinner', 'r.david.murray', 'ethan.furman', 'Yury.Selivanov', 'python-dev', 'schlamar', 'Rosuav', 'berker.peksag', 'serhiy.storchaka', 'yselivanov', 'NeilGirdhar', 'john.black.3k']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22906'
versions = ['Python 3.5']

[Rosuav]

See PEP for full details. Attached is POC patch: behaviour is altered globally (rather than respecting a __future__ directive), and minimal changes are made elsewhere to make the test suite mostly pass (test_generators does not - it'll need more comprehensive edits). Note that PEP-8 is deliberately not followed here; some of the edits ought to involve indenting slabs of code, but instead, "half-level" indentation is used, to keep the patch visibly minimal.

[python-dev]

New changeset dd19add74b21 by Guido van Rossum in branch 'default':
PEP-479: Add link to bpo-22906.
https://hg.python.org/peps/rev/dd19add74b21

[schlamar]

AFAIS this would break all existing code for yield-based coroutine schedulers (Tornado, Twisted, Trollius, monocle, ...) when a coroutine is exited with raise StopIteration in client code.

And this seems like a lot, a quick GitHub code search gives various examples, e.g.

• https://github.com/circus-tent/circus-web/blob/3b76c83d1eb984a78ed69a9abcb13054cde78f89/circusweb/circushttpd.py#L170
• https://github.com/kzahel/falcon-api/blob/5c62454db971bc99c52694cf2ce0613fb1504d80/python/falcon_api/session.py#L159
• https://github.com/peterhajas/Genesis/blob/2f1a03c38934b892ee6bb04e07e867bf463b7ae5/servers/genesis/clients.py#L24

[Rosuav]

Marc, are those all cases where the "raise StopIteration" is actually inside a generator? If so, it can be trivially replaced with "return". Yes, it'll break that way of spelling it, but it's a completely mechanical transformation, and making the change won't break your code for previous versions of Python. Personally, I would recommend using "return" there anyway, regardless of this proposal.

[schlamar]

Yes, all yield-based coroutines are generators. I know that there is a backward compatible upgrade path, but this might have a huge impact on existing code.

Interestingly, I didn't know before researching this PEP that you can actually use return without arguments in generators before Python 3.3 (even in 2.3) and I have worked a lot with coroutines/generators.

So I'm not even against this proposal and using return instead of raise StopIteration seems the right way to exit a generator/coroutine, but there could be lots of affected users...

[Rosuav]

Yep, the question is whether any of the "raise StopIteration" lines are actually non-local flow control. If they're local, then it's easy: mechanical replacement with "return" and it becomes compatible with all versions (unless it has a value attached to it, as "return x" doesn't work in Py2). But if they're non-local, some refactoring will need to be done.

In any case, my line of argument is: A generator function is not an iterator's __next__ method, ergo iterator protocol does not apply. Use of StopIteration is a hack that happens to work because of how generator functions are implemented (a thin wrapper around an iterator), but it's not part of the *concept* of a generator function.

[gvanrossum]

If all examples were just using "raise StopIteration" instead of "return"
in a generator I would be very happy. Such code can be trivially fixed by
using "return" and the __future__ import will help the eventual transition.

It's sad that apparently this use of return hasn't been better advertised
-- it has existed since generators were first introduced.

On Fri, Nov 21, 2014 at 9:44 AM, Chris Angelico <report@bugs.python.org>
wrote:

Chris Angelico added the comment:

Yep, the question is whether any of the "raise StopIteration" lines are
actually non-local flow control. If they're local, then it's easy:
mechanical replacement with "return" and it becomes compatible with all
versions (unless it has a value attached to it, as "return x" doesn't work
in Py2). But if they're non-local, some refactoring will need to be done.

In any case, my line of argument is: A generator function is not an
iterator's __next__ method, ergo iterator protocol does not apply. Use of
StopIteration is a hack that happens to work because of how generator
functions are implemented (a thin wrapper around an iterator), but it's not
part of the *concept* of a generator function.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue22906\>

---

[Rosuav]

Sadly, I don't know of a way to check if that's the case, other than by manually going through and eyeballing the code - if there's "raise StopIteration", see if there's also "yield" in the same function. The three cited examples are (I checked those straight away), but frankly, I am not volunteering to go manually through all of github in search of examples :|

[serhiy-storchaka]

I suggest to apply right now those changes which are compatible with current behavior and don't make code more cumbersome. E.g.

•    while True:
  

•        yield next(line_pair_iterator)
  

+ yield from line_pair_iterator

and

•    raise StopIteration
  

+ return

To me these changes make code even better and are worth to be applied even if PEP-479 will be rejected.

[Rosuav]

Known issues with the current patch, if anyone feels like playing with this who better knows the code:

1. Needs a __future__ directive to control behaviour
2. test_generators needs to be heavily reworked
3. The test of what exception was thrown needs to also handle StopIteration subclasses
4. Check for refleaks and/or over-freeing
5. Properly provide a traceback for the original StopIteration (not always happening)

Any others?

[gvanrossum]

Here's a doc patch for itertools.

[rhettinger]

FYI: I applied these two changes right after Guido pronounced on PEP-479:

https://mail.python.org/pipermail/python-checkins/2014-November/133252.html

https://mail.python.org/pipermail/python-checkins/2014-November/133253.html

Also, I'm submitting a patch to fix the code in Django that will be broken by PEP-479.

[vstinner]

FYI: I applied these two changes right after Guido pronounced on PEP-479:

Extract of emails:

changeset: 93542:9eb0d0eb0992
parent: 93540:23f8a511050a
user: Raymond Hettinger <python at rcn.com>
date: Sat Nov 22 21:56:23 2014 -0800

PEP-479: Don't let StopIteration bubble out of calls to next() inside a generator.

changeset: 93543:e8b3083bb148
user: Raymond Hettinger <python at rcn.com>
date: Sat Nov 22 22:14:41 2014 -0800

PEP-479: Use the return-keyword instead of raising StopIteration inside a generators.