X | Y union syntax breaks GenericModel

[squarebridges]

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.9.1
            pydantic compiled: True
                 install path: /home/paystone/.local/lib/python3.10/site-packages/pydantic
               python version: 3.10.2 (main, Jan 29 2022, 02:55:36) [GCC 10.2.1 20210110]
                     platform: Linux-5.18.2-arch1-1-x86_64-with-glibc2.31
     optional deps. installed: ['typing-extensions']

Python 3.10 incorporated PEP 604, which added syntax for writing Unions as X | Y, instead of typing.Union[X, Y]. It seems that when this syntax is used in combination with TypeVars as part of a GenericModel, the binding of TypeVars to their types fails.

Below is first a working example using typing.Union, followed by a non-working example using | syntax, with the stack trace. The example seeks to construct a Relationship GenericModel, and a BidirectionalMultiRelationship model which contains a list of either Relationship[A, B] or Relationship[B, A] models:

from typing import Union, Generic, TypeVar
from pydantic.generics import GenericModel

SourceT = TypeVar("SourceT")
TargetT = TypeVar("TargetT")

class Relationship(GenericModel, Generic[SourceT, TargetT]):
    source: SourceT
    target: TargetT

class BidirectionalMultiRelationship(GenericModel, Generic[SourceT, TargetT]):
    relationships: list[Union[Relationship[SourceT, TargetT], Relationship[TargetT, SourceT]]]
    
BidirectionalMultiRelationship[str, int]

from typing import Generic, TypeVar
from pydantic.generics import GenericModel

SourceT = TypeVar("SourceT")
TargetT = TypeVar("TargetT")

class Relationship(GenericModel, Generic[SourceT, TargetT]):
    source: SourceT
    target: TargetT

class BidirectionalMultiRelationship(GenericModel, Generic[SourceT, TargetT]):
    relationships: list[Relationship[SourceT, TargetT] | Relationship[TargetT, SourceT]]
    
BidirectionalMultiRelationship[str, int]

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [2], in <cell line: 14>()
     11 class BidirectionalMultiRelationship(GenericModel, Generic[SourceT, TargetT]):
     12     relationships: Relationship[SourceT, TargetT] | Relationship[TargetT, SourceT]
---> 14 BidirectionalMultiRelationship[str, int]

File ~/.local/lib/python3.10/site-packages/pydantic/generics.py:137, in GenericModel.__class_getitem__(cls, params)
    133     _generic_types_cache[(cls, params[0])] = created_model
    135 # Recursively walk class type hints and replace generic typevars
    136 # with concrete types that were passed.
--> 137 _prepare_model_fields(created_model, fields, instance_type_hints, typevars_map)
    139 return created_model

File ~/.local/lib/python3.10/site-packages/pydantic/generics.py:356, in _prepare_model_fields(created_model, fields, instance_type_hints, typevars_map)
    353 assert field.type_.__class__ is DeferredType, field.type_.__class__
    355 field_type_hint = instance_type_hints[key]
--> 356 concrete_type = replace_types(field_type_hint, typevars_map)
    357 field.type_ = concrete_type
    358 field.outer_type_ = concrete_type

File ~/.local/lib/python3.10/site-packages/pydantic/generics.py:263, in replace_types(type_, type_map)
    261         origin_type = getattr(typing, type_._name)
    262     assert origin_type is not None
--> 263     return origin_type[resolved_type_args]
    265 # We handle pydantic generic models separately as they don't have the same
    266 # semantics as "typing" classes or generic aliases
    267 if not origin_type and lenient_issubclass(type_, GenericModel) and not type_.__concrete__:

TypeError: 'type' object is not subscriptable

On inspection I found that at the point that the exception is thrown, origin_type is types.UnionType, where it should be Relationship.

Apologies if I missed an existing issue for this. I couldn't find anything in my search.

[AMDmi3]

Same thing with subscriptable types, with somehow simpler example:

# 1.py
from __future__ import annotations

from pydantic import BaseModel

class Model(BaseModel):
    foo: list[str]

% python3.8 1.py
Traceback (most recent call last):
  File "1.py", line 5, in <module>
    class Model(BaseModel):
  File "pydantic/main.py", line 188, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/typing.py", line 419, in pydantic.typing.resolve_annotations
    For example:
  File "/usr/local/lib/python3.8/typing.py", line 270, in _eval_type
    return t._evaluate(globalns, localns)
  File "/usr/local/lib/python3.8/typing.py", line 518, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
TypeError: 'type' object is not subscriptable

Obviously, python3.8 cannot evaluate list[str] here: while from __future__ import annotations (which is postponed evaluation of annotations) allows to use it in annotations, it's not a valid construct in python < 3.9 and cannot be evaluated directly. Thus I'm not sure if it's even fixable. The solution is to either use newer python, or use typing.List instead of list in annotations.

[thenx]

While i believe #4146 (comment) is answered in #2314 (comment), I'd love to bump the original issue with a slightly reduced example.

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.10.4
            pydantic compiled: True
                 install path: {project_path}/.venv/lib/python3.10/site-packages/pydantic
               python version: 3.10.8 (main, Nov 18 2022, 12:43:38) [Clang 14.0.0 (clang-1400.0.29.102)]
                     platform: macOS-13.1-arm64-arm-64bit
     optional deps. installed: ['dotenv', 'email-validator', 'typing-extensions']

from typing import Generic, TypeVar

import pydantic
from pydantic import generics

DataT = TypeVar("DataT")

class MyGenericModel(generics.GenericModel, Generic[DataT]):
    generic: list[DataT] | None


class A(pydantic.BaseModel):
    wrapper: MyGenericModel[int]

results in

Traceback (most recent call last):
  File "/<>/.pyenv/versions/3.10.8/lib/python3.10/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 12, in <module>
  File "<input>", line 13, in A
  File "<>/.venv/lib/python3.10/site-packages/pydantic/generics.py", line 145, in __class_getitem__
    _prepare_model_fields(created_model, fields, instance_type_hints, typevars_map)
  File "<>/.venv/lib/python3.10/site-packages/pydantic/generics.py", line 364, in _prepare_model_fields
    concrete_type = replace_types(field_type_hint, typevars_map)
  File "<>/.venv/lib/python3.10/site-packages/pydantic/generics.py", line 271, in replace_types
    return origin_type[resolved_type_args]
TypeError: 'type' object is not subscriptable

The same with Union

from typing import Generic, TypeVar, Union

import pydantic
from pydantic import generics

DataT = TypeVar("DataT")

class MyGenericModel(generics.GenericModel, Generic[DataT]):
    generic: Union[list[DataT], None]


class A(pydantic.BaseModel):
    wrapper: MyGenericModel[int]

seems to be working without the issue.

[samuelcolvin]

Confirmed, should be fixed in V2, @dmontagu might be worth checking this is fixed in #4970.

Happy to review a PR to fix this in V1.10, but I suspect it's not trivial.

[thenx]

A slightly more reduced example would be:

from pydantic import generics

generics.replace_types(list[str] | None, {str: int})

I guess the problem is that type(list | int) is types.UnionType and a dum-dum approach could be to translate types.UnionType back to typing.Union or just to do smth like functools.reduce(operator.or_, type_args) if origin_type is types.UnionType?

[dmontagu]

For what it's worth, on the #4970 PR, I checked that the following code runs without issue (on Python 3.11 anyway):

from typing import Generic, TypeVar

import pydantic

DataT = TypeVar("DataT")

class MyGenericModel(pydantic.BaseModel, Generic[DataT]):
    generic: list[DataT] | None

class A(pydantic.BaseModel):
    wrapper: MyGenericModel[int]

m = MyGenericModel[int](generic=None)
print(A(wrapper=m))
# wrapper=MyGenericModel[int](generic=None)

[Kludex]

Since this is working on V2, I'll be closing this issue. 🙏

[Cjkjvfnby]

I can reproduce it on Mac and windows with Python 3.9

pydantic.version.VERSION =2.2.1
sys.version = 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)]

from __future__ import annotations

from pydantic import BaseModel

class Model(BaseModel):
    foo: list[str]
    boo: int | str

TypeError: unsupported operand type(s) for |: 'type' and 'type'

Traceback (most recent call last):
  File "sample.py", line 12, in <module>
    class Model(BaseModel):
  File "...\lib\site-packages\pydantic\_internal\_model_construction.py", line 177, in __new__
    set_model_fields(cls, bases, config_wrapper, types_namespace)
  File "...\lib\site-packages\pydantic\_internal\_model_construction.py", line 405, in set_model_fields
    fields, class_vars = collect_model_fields(cls, bases, config_wrapper, types_namespace, typevars_map=typevars_map)
  File "...\lib\site-packages\pydantic\_internal\_fields.py", line 98, in collect_model_fields
    type_hints = get_cls_type_hints_lenient(cls, types_namespace)
  File "...\lib\site-packages\pydantic\_internal\_typing_extra.py", line 212, in get_cls_type_hints_lenient
    hints[name] = eval_type_lenient(value, globalns, localns)
  File "...\lib\site-packages\pydantic\_internal\_typing_extra.py", line 224, in eval_type_lenient
    return typing._eval_type(value, globalns, localns)  # type: ignore
  File "...\Python39\lib\typing.py", line 290, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
  File "...\Python39\lib\typing.py", line 546, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
TypeError: unsupported operand type(s) for |: 'type' and 'type'

[thenx]

I can reproduce it on Mac and windows with Python 3.9

PEP604-syntax requires python>=3.10