Merge generator.gi_running and frame executing flag into single frame state

[markshannon]

BPO	40941
Nosy	@vstinner, @cjerdonek, @markshannon
PRs	bpo-40941: Unify implicit and explicit state in the frame and generator objects into a single value. #20803 bpo-40941: Fix stackdepth compiler warnings #22377 bpo-40941: Fix fold_tuple_on_constants() compiler warnings #22378

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/markshannon'
closed_at = <Date 2020-09-23.12:08:01.765>
created_at = <Date 2020-06-10.13:33:12.111>
labels = ['interpreter-core', '3.10', 'performance']
title = 'Merge generator.gi_running and frame executing flag into single frame state'
updated_at = <Date 2020-09-23.15:53:08.630>
user = 'https://github.com/markshannon'

bugs.python.org fields:

activity = <Date 2020-09-23.15:53:08.630>
actor = 'vstinner'
assignee = 'Mark.Shannon'
closed = True
closed_date = <Date 2020-09-23.12:08:01.765>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2020-06-10.13:33:12.111>
creator = 'Mark.Shannon'
dependencies = []
files = []
hgrepos = []
issue_num = 40941
keywords = ['patch']
message_count = 11.0
messages = ['371194', '371277', '373388', '373401', '373813', '376292', '377369', '377370', '377371', '377399', '377402']
nosy_count = 3.0
nosy_names = ['vstinner', 'chris.jerdonek', 'Mark.Shannon']
pr_nums = ['20803', '22377', '22378']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue40941'
versions = ['Python 3.10']

[markshannon]

Generators have a "gi_running" attribute (coroutines have an equivalent cr_running flag). Internally frame also has an executing flag.

We use these, plus the test f_stacktop pointer == NULL, to determine what state a generator or frame is in.

It would be much cleaner to maintain a single f_state field in the frame to track the state of a frame, reducing the change of ambiguity and error.

The possible states of a frame are:
CREATED -- Frame exists but has not been executed at all
SUSPENDED -- Frame has been executed, and has been suspended by a yield
EXECUTING -- Frame is being executed
RETURNED -- Frame has completed by a RETURN_VALUE instruction
RAISED -- Frame has completed as a result of an exception being raised
CLEARED -- Frame has been cleared, either by explicit call to clear or by the GC.

[markshannon]

This change combines the explicit state in PyFrameObject.f_excuting and PyGenObject.gi_running, and the implicit state in PyFrameObject.f_stacktop == NULL and PyFrameObject.f_last == -1 into a single enum field in PyFrameObject.

Since we no longer need to test PyFrameObject.f_stacktop == NULL , The f_stacktop pointer can be replaced with a f_stackdepth integer, which make for simpler code when iterating over the stack and avoids the potential hazard of NULL pointers.

There are three benefits to these changes:

1. The code is more robust and, IMO, easier to understand, as all state is now explicit.
2. It carries additional information about the state of the frame. Information about whether a frame exiting by return or by raise is available.
3. A modest reduction in size of frame and generator objects.

[cjerdonek]

Mark, I want to flag bpo-29590 for you ("Incorrect stack traces when re-entering a generator/coroutine stack via .throw()") in case this issue relates to or would help at all with that.

[markshannon]

bpo-29590 is unrelated

[markshannon]

New changeset cb9879b by Mark Shannon in branch 'master':
bpo-40941: Unify implicit and explicit state in the frame and generator objects into a single value. (GH-20803)
cb9879b

[vstinner]

The change introduced compiler warnings on Windows:

c:\vstinner\python\master\python\ceval.c(1448): warning C4244: '=': conversion
from '__int64' to 'int', possible loss of data [C:\vstinner\python\master\PCbui
ld\pythoncore.vcxproj]
c:\vstinner\python\master\python\ceval.c(2254): warning C4244: '=': conversion
from '__int64' to 'int', possible loss of data [C:\vstinner\python\master\PCbui
ld\pythoncore.vcxproj]
c:\vstinner\python\master\python\ceval.c(2271): warning C4244: '=': conversion
from '__int64' to 'int', possible loss of data [C:\vstinner\python\master\PCbui
ld\pythoncore.vcxproj]

[vstinner]

New changeset 71f2ff4 by Victor Stinner in branch 'master':
bpo-40941: Fix fold_tuple_on_constants() compiler warnings (GH-22378)
71f2ff4

[vstinner]

New changeset b7d8d8d by Victor Stinner in branch 'master':
bpo-40941: Fix stackdepth compiler warnings (GH-22377)
b7d8d8d

[vstinner]

The main change is merged, and I fixed compiler warnings. It seems like there is no remaining thing to do, so I close the issue.