match/case statements trace incorrectly in 3.10.0b4

[nedbat]

BPO	44600
Nosy	@nedbat, @ambv, @markshannon, @ammaraskar, @pablogsal, @miss-islington, @brandtbucher, @chaburkland, @jdevries3133
PRs	bpo-44600: Pattern-matching: line-numbers incorrectly traced on default case #27346 [3.10] bpo-44600: Fix line numbers for pattern matching cleanup code (GH-27346) #27356 bpo-44600: Refactor new tracing tests #27396 [3.10] bpo-44600: Refactor new tracing tests (GH-27396) #27397 [3.10] bpo-44600: Refactor new tracing tests (GH-27396) #27404

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brandtbucher'
closed_at = <Date 2021-07-25.23:43:03.716>
created_at = <Date 2021-07-11.21:02:55.662>
labels = ['interpreter-core', 'type-bug', '3.10', 'release-blocker']
title = 'match/case statements trace incorrectly in 3.10.0b4'
updated_at = <Date 2021-07-28.12:38:57.459>
user = 'https://github.com/nedbat'

bugs.python.org fields:

activity = <Date 2021-07-28.12:38:57.459>
actor = 'lukasz.langa'
assignee = 'brandtbucher'
closed = True
closed_date = <Date 2021-07-25.23:43:03.716>
closer = 'brandtbucher'
components = ['Interpreter Core']
creation = <Date 2021-07-11.21:02:55.662>
creator = 'nedbat'
dependencies = []
files = []
hgrepos = []
issue_num = 44600
keywords = ['patch']
message_count = 13.0
messages = ['397264', '397266', '397346', '397574', '397738', '397756', '398091', '398145', '398163', '398199', '398200', '398335', '398368']
nosy_count = 9.0
nosy_names = ['nedbat', 'lukasz.langa', 'Mark.Shannon', 'ammar2', 'pablogsal', 'miss-islington', 'brandtbucher', 'chaburkland', 'jack__d']
pr_nums = ['27346', '27356', '27396', '27397', '27404']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44600'
versions = ['Python 3.10']

[nedbat]

Some simple match/case statements show incorrect tracing. Below is the code to run, as well as the output. Output lines with initial stars are incorrect: they incorrectly indicate that case bodies are executing when they are not. Sorry for the bulk here, I wanted to give you all the cases I had.

-- 8< -------------------------------------

import linecache, sys

def trace(frame, event, arg):
    # The weird globals here is to avoid a NameError on shutdown...
    if frame.f_code.co_filename == globals().get("__file__"):
        lineno = frame.f_lineno
        print("{} {}: {}".format(event[:4], lineno, linecache.getline(__file__, lineno).rstrip()))
    return trace

def match_with_default():
    for command in ["huh", "go home", "go n"]:
        print(command)
        match command.split():
            case ["go", direction] if direction in "nesw":
                match = f"go: {direction}"
            case ["go", _]:
                match = "no go"
            case _:
                match = "default"
        print(match)

def match_with_wildcard():
    for command in ["huh", "go home", "go n"]:
        print(command)
        match command.split():
            case ["go", direction] if direction in "nesw":
                match = f"go: {direction}"
            case ["go", _]:
                match = "no go"
            case x:
                match = f"default: {x}"
        print(match)

def match_without_wildcard():
    match = None
    for command in ["huh", "go home", "go n"]:
        print(command)
        match command.split():
            case ["go", direction] if direction in "nesw":
                match = f"go: {direction}"
            case ["go", _]:
                match = "no go"
        print(match)

print(sys.version)
sys.settrace(trace)
match_with_default()
match_with_wildcard()
match_without_wildcard()
-- 8< 

---

Output:

3.10.0b4 (default, Jul 11 2021, 13:51:53) [Clang 12.0.0 (clang-1200.0.32.29)]
call 10: def match_with_default():
line 11: for command in ["huh", "go home", "go n"]:
line 12: print(command)
huh
line 13: match command.split():
line 14: case ["go", direction] if direction in "nesw":
line 15: match = f"go: {direction}"
line 16: case ["go", _]:

• line 17: match = "no go"
  line 19: match = "default"
  line 20: print(match)
  default
  line 11: for command in ["huh", "go home", "go n"]:
  line 12: print(command)
  go home
  line 13: match command.split():
  line 14: case ["go", direction] if direction in "nesw":
  line 16: case ["go", _]:
  line 17: match = "no go"
  line 20: print(match)
  no go
  line 11: for command in ["huh", "go home", "go n"]:
  line 12: print(command)
  go n
  line 13: match command.split():
  line 14: case ["go", direction] if direction in "nesw":
  line 15: match = f"go: {direction}"
  line 20: print(match)
  go: n
  line 11: for command in ["huh", "go home", "go n"]:
  retu 11: for command in ["huh", "go home", "go n"]:
  call 22: def match_with_wildcard():
  line 23: for command in ["huh", "go home", "go n"]:
  line 24: print(command)
  huh
  line 25: match command.split():
  line 26: case ["go", direction] if direction in "nesw":
• line 27: match = f"go: {direction}"
  line 28: case ["go", _]:
• line 29: match = "no go"
  line 30: case x:
  line 31: match = f"default: {x}"
  line 32: print(match)
  default: ['huh']
  line 23: for command in ["huh", "go home", "go n"]:
  line 24: print(command)
  go home
  line 25: match command.split():
  line 26: case ["go", direction] if direction in "nesw":
  line 28: case ["go", _]:
  line 29: match = "no go"
  line 32: print(match)
  no go
  line 23: for command in ["huh", "go home", "go n"]:
  line 24: print(command)
  go n
  line 25: match command.split():
  line 26: case ["go", direction] if direction in "nesw":
  line 27: match = f"go: {direction}"
  line 32: print(match)
  go: n
  line 23: for command in ["huh", "go home", "go n"]:
  retu 23: for command in ["huh", "go home", "go n"]:
  call 34: def match_without_wildcard():
  line 35: match = None
  line 36: for command in ["huh", "go home", "go n"]:
  line 37: print(command)
  huh
  line 38: match command.split():
  line 39: case ["go", direction] if direction in "nesw":
• line 40: match = f"go: {direction}"
  line 41: case ["go", _]:
• line 42: match = "no go"
  line 43: print(match)
  None
  line 36: for command in ["huh", "go home", "go n"]:
  line 37: print(command)
  go home
  line 38: match command.split():
  line 39: case ["go", direction] if direction in "nesw":
  line 41: case ["go", _]:
  line 42: match = "no go"
  line 43: print(match)
  no go
  line 36: for command in ["huh", "go home", "go n"]:
  line 37: print(command)
  go n
  line 38: match command.split():
  line 39: case ["go", direction] if direction in "nesw":
  line 40: match = f"go: {direction}"
  line 43: print(match)
  go: n
  line 36: for command in ["huh", "go home", "go n"]:
  retu 36: for command in ["huh", "go home", "go n"]:

[brandtbucher]

Thanks, I'll take a closer look at this soon (most likely this coming week).

Out of curiosity, do we have a standard way of writing regression tests for correct line numbers like this? Of the top of my head, it seems like we could either statically compare the dis output (fragile) or actually execute some code with a trace function enabled and check the line hits (tricky).

Neither really seems ideal, so I'm wondering if there's be a better way?

[ammaraskar]

Brandt, maybe this regression test from a previous tracing bug might be useful for the test writing: https://github.com/python/cpython/pull/22026/files#diff-8b73bfc55514d8add8904c5f4d1d24b7b644ebfccba8d846085303577aa94dd6

[brandtbucher]

Thanks, that test framework looks good for this.

My initial hunch (just from looking at this) is that this has to do with how we handle cleanup after failed matches. Our "fail pop" blocks probably have whatever the last line number compiled was (which I think is always the last line of the preceding case block).

The fix *should* be as simple as calling "SET_LOC(c, whatever_the_last_pattern_was)" before compiling these blocks.

I have some new-ish contributors who might want to take this. As far as compiler issues go, this one seems pretty straightforward.

[jdevries3133]

@brandtbucher, is anyone working on this yet? I'd like to take a crack at it this week if it's still available!

[brandtbucher]

Yeah, this is actively being worked on. Thanks for asking.

If anything changes, I’ll let you know and you can pick it up then!

[brandtbucher]

Two other things we realized while working on this:

• The first occurrence of line 15 in the example output should be marked as incorrectly traced.

• We should emit a NOP to show coverage of "case _" (that fix will be part of the same PR).