3.11: unused generator comprehensions cause f_lineno==None

[nedbat]

BPO	46389
Nosy	@scoder, @nedbat, @markshannon

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/markshannon'
closed_at = <Date 2022-03-03.14:12:07.700>
created_at = <Date 2022-01-15.12:26:22.492>
labels = ['interpreter-core', '3.11']
title = '3.11: unused generator comprehensions cause f_lineno==None'
updated_at = <Date 2022-03-04.16:56:53.732>
user = 'https://github.com/nedbat'

bugs.python.org fields:

activity = <Date 2022-03-04.16:56:53.732>
actor = 'Mark.Shannon'
assignee = 'Mark.Shannon'
closed = True
closed_date = <Date 2022-03-03.14:12:07.700>
closer = 'Mark.Shannon'
components = ['Interpreter Core']
creation = <Date 2022-01-15.12:26:22.492>
creator = 'nedbat'
dependencies = []
files = []
hgrepos = []
issue_num = 46389
keywords = ['3.11regression']
message_count = 6.0
messages = ['410643', '410763', '413972', '413984', '414420', '414439']
nosy_count = 3.0
nosy_names = ['scoder', 'nedbat', 'Mark.Shannon']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue46389'
versions = ['Python 3.11']

[nedbat]

In Python 3.11, unused generator comprehensions cause trace events with f_lineno set to None.

---- %< -------------------------------------------------

import linecache, sys

def trace(frame, event, arg):
    # The weird globals here is to avoid a NameError on shutdown...
    if frame.f_code.co_filename == globals().get("__file__"):
        lineno = frame.f_lineno
        if lineno is not None:
            line = linecache.getline(__file__, lineno).rstrip()
        else:
            line = "<none>"
        print("{} {!s:4}: {}".format(event[:4], lineno, line))
    return trace

print(sys.version)
sys.settrace(trace)

(i for i in [1])
------------------------------------------------------

$ python3.10 /tmp/runbpo.py
3.10.2 (main, Jan 15 2022, 05:51:59) [Clang 12.0.0 (clang-1200.0.32.29)]
call 17  : (i for i in [1])
exce 17  : (i for i in [1])
retu 17  : (i for i in [1])

$ python3.11 /tmp/runbpo.py
3.11.0a4 (main, Jan 14 2022, 18:14:29) [Clang 12.0.0 (clang-1200.0.32.29)]
call None: <none>
exce None: <none>
retu None: <none>

[markshannon]

This has the same root cause as https://bugs.python.org/issue46374:
closing the unused generator expression exposes the generator's frame before it has been initialized.

[scoder]

Possibly also related, so I though I'd mention it here (sorry if this is hijacking the ticket, seems difficult to tell). We're also seeing None values in f_lineno in Cython's test suite with 3.11a5:

  File "<doctest line_trace.run_trace[2]>", line 1, in <module>
    run_trace(py_add, 1, 2)
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "tests/run/line_trace.pyx", line 231, in line_trace.run_trace (line_trace.c:7000)
    func(*args)
  File "tests/run/line_trace.pyx", line 60, in line_trace.trace_trampoline (line_trace.c:3460)
    raise
  File "tests/run/line_trace.pyx", line 54, in line_trace.trace_trampoline (line_trace.c:3359)
    result = callback(frame, what, arg)
  File "tests/run/line_trace.pyx", line 81, in line_trace._create_trace_func._trace_func (line_trace.c:3927)
    trace.append((map_trace_types(event, event), frame.f_lineno - frame.f_code.co_firstlineno))
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

https://github.com/cython/cython/blob/7ab11ec473a604792bae454305adece55cd8ab37/tests/run/line_trace.pyx

No generator expressions involved, though. (Much of that test was written while trying to get the debugger in PyCharm to work with Cython compiled modules.)

There is a chance that Cython is doing something wrong in its own line tracing code, obviously.
(I also remember seeing other tracing issues before, where the line reported was actually in the trace function itself rather than the code to be traced. We haven't caught up with the frame-internal changes yet.)

[markshannon]

Stefan,

f_lineno can be None for some opcodes, but there shouldn't be trace events if it is.

E.g.
>>> def f():
...     try:
...          1/0
...     finally:
...          pass
...

>>> list(f.__code__.co_lines())
[(0, 2, 1), (2, 4, 2), (4, 12, 3), (12, 16, 5), (16, 18, None), (18, 26, 5)]

Note that the bytecode at offset 16 has no line number, but there should be no events for it.

BUT, before you try and fix your tracing emulation, I repeat my plea.

Please stop trying to mimic CPython internals, and ask for the APIs that you need.

[markshannon]

Ned, is this fixed for you now?

[nedbat]

Yes, this is fixed, thanks.