yield expression inside generator expression does nothing

[inyeollee]

BPO	10544
Nosy	@gvanrossum, @birkenfeld, @rhettinger, @ncoghlan, @abalkin, @benjaminp, @glyph, @serhiy-storchaka, @1st1, @ajdavis, @DimitrisJim
PRs	[3.8] bpo-10544: Disallow "yield" in comprehensions and generator expressions. #4564 bpo-10544: Deprecate "yield" in comprehensions and generator expressions. #4579 [2.7] bpo-10544: Deprecate "yield" in comprehensions and generator expressions in Py3k mode. (GH-4579) #4676
Files	yield-in-comprehensions.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-12-02.19:01:38.844>
created_at = <Date 2010-11-26.19:21:17.627>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'yield expression inside generator expression does nothing'
updated_at = <Date 2018-02-04.08:53:53.562>
user = 'https://bugs.python.org/InyeolLee'

bugs.python.org fields:

activity = <Date 2018-02-04.08:53:53.562>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-02.19:01:38.844>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2010-11-26.19:21:17.627>
creator = 'Inyeol.Lee'
dependencies = []
files = ['47296']
hgrepos = []
issue_num = 10544
keywords = ['patch']
message_count = 66.0
messages = ['122475', '122512', '122645', '122658', '122705', '122759', '122760', '122762', '122765', '123641', '123649', '241410', '286273', '286274', '286280', '286281', '286282', '286283', '286407', '286408', '286409', '306784', '306802', '306804', '306805', '306806', '306807', '306808', '306810', '306812', '306829', '306831', '306832', '306833', '306834', '306835', '306836', '306837', '306838', '306840', '306841', '306842', '306843', '306844', '306845', '306846', '306847', '306939', '306940', '306963', '306964', '306983', '306995', '307013', '307033', '307035', '307039', '307051', '307067', '307091', '307095', '307359', '307361', '307426', '307449', '311593']
nosy_count = 15.0
nosy_names = ['gvanrossum', 'georg.brandl', 'rhettinger', 'ncoghlan', 'belopolsky', 'benjamin.peterson', 'erickt', 'glyph', 'Inyeol.Lee', 'serhiy.storchaka', 'yselivanov', 'esc24', 'danielsh', 'emptysquare', 'Jim Fasarakis-Hilliard']
pr_nums = ['4564', '4579', '4676']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10544'
versions = ['Python 3.7']

[inyeollee]

Simple coroutine with for loop works:

>>> def pack_a():
        while True:
            L = []
            for i in range(2):
                L.append((yield))
            print(L)

>>> pa = pack_a()
>>> next(pa)
>>> pa.send(1)
>>> pa.send(2)
[1, 2]
>>>

If using list comprehension (generator expression), it fails:

>>> def pack_b():
        while True:
            L = [(yield) for i in range(2)]
            print(L)

>>> pb = pack_b()
<endless loop here>

I understand what's going on here - generator expression is converted to nested function and there's no way to either stop the execution inside nested function (since it's not started yet!) or send() a value to its yield expression. Still I think this behavior is a bug and needs fixed.

• best fix would make it behave the same as for loop.
• if it's not fixable, yield expression inside genexp should not be allowed.

[rhettinger]

Hmm, what an interesting and unexpected side-effect of the efforts to hide the loop induction variable.

[birkenfeld]

While the behavior is confusing, I don't think yield inside comprehensions should be disallowed. Rather, the fact that comprehensions have their own scope should be stated clearer.

[benjaminp]

I think I can probably fix it, but it's debatable whether it should be done, since it'd make list comps more of "quasi" functions.

[rhettinger]

This discussion should probably be moved to python-dev. With tools like Twisted's inlineDefer or the Monocle package, there is a growing need to be able to use yield in complex expressions. Yet, that goes against the trend toward making lists comps more like genexps and less like sugar for a simple for-loop accumulator.

Guido, do you have any thoughts on the subject? Mark it a documentation issue or try out Benjamin's fix?

[gvanrossum]

I think it is definitely wrong the way it works in 3.x. (Especially since it works as expected in 2.x.)

I agree with Inyeol's preference of fixes: (1) make it work properly for listcomps as well as genexps, (2) if that's not possible, forbid yield in a genexp or listcomp.

Note that even though yield in a genexp could be considered as having a well-defined meaning, that meaning is not useful and I would consider it as merely a coincidence of the specification, not an intentional effect. So I would be fine changing its meaning. (My assumption is that since it is not useful there is -- almost -- no code depending on that meaning.)

[gvanrossum]

PS. Wasn't there a similar issue with something inside a genexp that raises StopIteration? Did we ever solve that?

[abalkin]

Isn't this the same issue as bpo-3267?

[gvanrossum]

Yes it is, but I was never asked about it back then.

[terryjreedy]

bpo-3267 did not expose endless loop possibility and was closed as won't fix.
Rather than reopen that and close this and move nosy list back, I added to nosy list here.

[birkenfeld]

FWIW, the "endless loop possibility" is not of issue here, and is simply an artifact of the specific generator function the OP uses.