Parameter substitution in the union type does not work with typing.Union

[serhiy-storchaka]

BPO	44653
Nosy	@gvanrossum, @ambv, @serhiy-storchaka, @miss-islington, @uriyyo, @Fidget-Spinner
PRs	bpo-44653: Support typing.Union in parameter substitution of the union type #27232 bpo-44653: Support typing types in parameter substitution in the union type. #27247 [3.10] bpo-44653: Support typing types in parameter substitution in the union type. (GH-27247) #27296

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-07-22.22:21:37.910>
created_at = <Date 2021-07-16.13:32:27.733>
labels = ['interpreter-core', 'type-bug', '3.10', '3.11']
title = 'Parameter substitution in the union type does not work with typing.Union'
updated_at = <Date 2021-07-22.22:34:04.923>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-07-22.22:34:04.923>
actor = 'gvanrossum'
assignee = 'none'
closed = True
closed_date = <Date 2021-07-22.22:21:37.910>
closer = 'lukasz.langa'
components = ['Interpreter Core']
creation = <Date 2021-07-16.13:32:27.733>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 44653
keywords = ['patch']
message_count = 8.0
messages = ['397624', '397803', '397827', '397828', '398009', '398011', '398012', '398014']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'lukasz.langa', 'serhiy.storchaka', 'miss-islington', 'uriyyo', 'kj']
pr_nums = ['27232', '27247', '27296']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44653'
versions = ['Python 3.10', 'Python 3.11']

[serhiy-storchaka]

>>> import typing
>>> T = typing.TypeVar('T')
>>> (int | T)[typing.Union[str, list]]
NotImplemented

See also bpo-44633. But in this case the expected result is int | str | list or typing.Union[init, str, list].

[Fidget-Spinner]

@serhiy, this doesn't just affect typing.Union, it seems that the rest of the typing types don't substitute:

>>> (int | T)[typing.List[str]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Each union arg must be a type, got typing.List[str]

We should probably loosen the check during the union_getitem substitution -- no need to raise the TypeError or check for is_union, just blindly replace the TypeVar. We already do this for types.GenericAlias:

>>> list[T][1]
list[1]

Or if you want to continue checking, maybe checking for PyCallable_Check(obj) in substitution is enough - typing internally accepts callable(o) too:

https://github.com/python/cpython/blob/3.10/Lib/typing.py#L146

[serhiy-storchaka]

There is also problem with other typing types:

>>> (int | T)[typing.List]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Each union argument must be a type, got typing.List
>>> (int | T)[typing.List[int]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Each union argument must be a type, got typing.List[int]
>>> (int | T)[typing.Hashable]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Each union argument must be a type, got typing.Hashable
>>> (int | T)[typing.Callable[[int], str]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Each union argument must be a type, got typing.Callable[[int], str]
>>> (int | T)[typing.ParamSpec('P')]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Each union argument must be a type, got ~P

Despite the fact that they support the | operator.

We can add one by one special support of different types supporting the | operator (ParamSpec, _GenericAlias, _CallableGenericAlias, _UnionGenericAlias, _LiteralGenericAlias, _ConcatenateGenericAlias, _AnnotatedAlias, _SpecialGenericAlias, _CallableType, _TupleType), but it is cumbersome and errorprone. We will need to synchronize code of unionobject.c with typing every time we add new kind of types.

PR 27247 uses more general approach. It calls the | operator for arguments after substitution. So all types which support the | operator are now automatically supported. But the result of parameter substitution can now be typing.Union instead of types.Union.

>>> import typing
>>> import collections.abc
>>> T = typing.TypeVar('T')
>>> (int | T)[list]
int | list
>>> (int | T)[typing.List]
typing.Union[int, typing.List]
>>> (int | T)[list[int]]
int | list[int]
>>> (int | T)[typing.List[int]]
typing.Union[int, typing.List[int]]
>>> (int | T)[collections.abc.Hashable]
int | collections.abc.Hashable
>>> (int | T)[typing.Hashable]
typing.Union[int, typing.Hashable]
>>> (int | T)[collections.abc.Sequence[int]]
int | collections.abc.Sequence[int]
>>> (int | T)[typing.Sequence[int]]
typing.Union[int, typing.Sequence[int]]
>>> (int | T)[collections.abc.Callable[[int], str]]
int | collections.abc.Callable[[int], str]
>>> (int | T)[typing.Callable[[int], str]]
typing.Union[int, typing.Callable[[int], str]]
>>> (int | T)[typing.TypeVar('S')]
int | ~S
>>> (int | T)[typing.ParamSpec('P')]
typing.Union[int, ~P]
>>> (int | T)[str | list]
int | str | list
>>> (int | T)[typing.Union[str, list]]
typing.Union[int, str, list]

[gvanrossum]

I need someone else to own this, sorry.

[ambv]

New changeset 2e3744d by Serhiy Storchaka in branch 'main':
bpo-44653: Support typing types in parameter substitution in the union type. (GH-27247)
2e3744d

[ambv]

New changeset 21db59f by Miss Islington (bot) in branch '3.10':
bpo-44653: Support typing types in parameter substitution in the union type. (GH-27247) (bpo-27296)
21db59f

[ambv]

Thanks Serhiy for the report and the fix, and Ken for reviews.