Cyclic import and Unpack raises INTERNAL ERROR

[hoxbro]

Crash Report

Running the following code raises INTERNAL ERROR. Should likely give a better warning.

An MRE example of problem first raised here: #20486 (comment)

Ref: bokeh/bokeh#14817

Traceback

/home/shh/Downloads/mre_bokeh/mre_pkg/callbacks.pyi:10: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.19.1
/home/shh/Downloads/mre_bokeh/mre_pkg/callbacks.pyi:10: : note: use --pdb to drop into pdb
Traceback (most recent call last):
  File "mypy/semanal.py", line 7568, in accept
  File "mypy/nodes.py", line 995, in accept
  File "mypy/semanal.py", line 978, in visit_func_def
  File "mypy/semanal.py", line 1023, in analyze_func_def
  File "mypy/semanal.py", line 7214, in defer
AssertionError: Must not defer during final iteration

To Reproduce

#!/usr/bin/env bash
set -euo pipefail

rm -rf mre_pkg mre.py .mypy_cache
mkdir -p mre_pkg

cat >mre.py <<'PY'
import mre_pkg.model
PY

cat >mre_pkg/model.pyi <<'PY'
from typing import TYPE_CHECKING, TypedDict

from typing_extensions import Unpack

from .callbacks import Callback

class _ModelInit(TypedDict, total=False):
    callback: Callback

class Model:
    def __init__(self, **kwargs: Unpack[_ModelInit]) -> None: ...
PY

cat >mre_pkg/callbacks.pyi <<'PY'
from typing import TYPE_CHECKING, TypedDict

from typing_extensions import Unpack

from .model import Model, _ModelInit

class _CallbackInit(_ModelInit, total=False): ...

class Callback(Model):
    def __init__(self, **kwargs: Unpack[_CallbackInit]) -> None: ...
PY

mypy mre.py --show-traceback

Your Environment

• Mypy version used: mypy 1.19.1 (compiled: yes)
• Mypy command-line flags:
• Mypy configuration options from mypy.ini (and other config files):
• Python version used:
• Operating system and version:

[bryevdv]

@ilevkivskyi I can confirm this seems to fix the issue with mypy-1.20.0+dev.5917b80053e58c585ad2854af288e9c797edce48

Do you have any suggestions for workarounds? As it is we cannot release Bokeh with a rather long-awaited set of typing improvements if it is going to break everyone's jobs in CI if they pin older versions of mypy

[hauntsaninja]

A little ugly, but could you redefine the TypedDict fields instead of inheriting?

[bryevdv]

In theory, but Bokeh has hundreds of models with thousands of properties total so realistically, probably not. If we did that we'd might as well just write out all the model __init__ signtatures in full by hand, which is what we were trying to avoid and simplify (all these models are themselves basically types in a traits-like type system so inheritance actually fits the bill here)

[hauntsaninja]

Hm maybe you could unravel the import cycle and move the problem-triggering TypedDict definitions into a single file? (And then re-export them from the files that they were previously defined in)

[ilevkivskyi]

A possible hacky workaround would be to add this before every problematic __init__ (this should force mypy to re-analyse the TypedDict):

class Callback(Model):
    _init_args_internal: _CallbackInit
    def __init__(self, **kwargs: Unpack[_CallbackInit]) -> None: ...

[bryevdv]

@ilevkivskyi that looks promising, thank you!