Add support for PEP 612 (Parameter Specification Variables) to typing.py

[gvanrossum]

BPO	41559
Nosy	@gvanrossum, @gvanrossum, @Fidget-Spinner
PRs	bpo-41559: Implement PEP 612 - Add ParamSpec and Concatenate to typing #23702 bpo-41559: Documentation for PEP 612 #24000 bpo-41559: Convert lists to tuples in genericalias to prepare for PEP 612 #24056 bpo-41559: Change PEP 612 implementation to pure Python #25449

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-28.15:47:23.848>
created_at = <Date 2020-08-15.18:36:04.607>
labels = ['type-bug', 'library', '3.10']
title = 'Add support for PEP 612 (Parameter Specification Variables) to typing.py'
updated_at = <Date 2021-04-28.16:14:24.696>
user = 'https://github.com/gvanrossum'

bugs.python.org fields:

activity = <Date 2021-04-28.16:14:24.696>
actor = 'gvanrossum'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-28.15:47:23.848>
closer = 'kj'
components = ['Library (Lib)']
creation = <Date 2020-08-15.18:36:04.607>
creator = 'gvanrossum'
dependencies = []
files = []
hgrepos = []
issue_num = 41559
keywords = ['patch']
message_count = 22.0
messages = ['375487', '382764', '382778', '382805', '382833', '382837', '382843', '382856', '382858', '382938', '383674', '383675', '383676', '384212', '384213', '384217', '386903', '391269', '391270', '392223', '392228', '392231']
nosy_count = 3.0
nosy_names = ['gvanrossum', 'Guido.van.Rossum', 'kj']
pr_nums = ['23702', '24000', '24056', '25449']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41559'
versions = ['Python 3.10']

[gvanrossum]

We need stub versions of ParamSpec and Concatenate added to typing.py, plus tests that ensure these actually work in all situations required by the PEP. (It's not so important to ensure that they raise exceptions at runtime in cases where the PEP says they needn't work -- static type checkers will flag those better.)

[gvanrossum]

Thanks for working on this -- let me know when you have a question for me. Once this is ready we should also add it to the typing_extensions module so people can use it on older Python versions. (https://github.com/python/typing/tree/master/typing_extensions)

[Fidget-Spinner]

I have a one question:
Should ParamSpec be treated as a type of TypeVar? This mostly pertains to
adding it to __parameters__ of generic aliases. Type substitution/chaining,
seems inappropiate for it right now.

Thanks for your help!

[gvanrossum]

I wrote

class C(Generic[T, P]): ...

and was surprised that C.__parameters__ was (T,) instead of (T, P).

[Fidget-Spinner]

and was surprised that C.__parameters__ was (T,) instead of (T, P).
Alright, I just fixed that :).

I'm now encountering many problems with how typing works which prevent what PEP-612 declare as valid from not throwing an error (these are all examples copied from the PEP)::

class X(Generic[T, P]): ...

1. X[int, [int, bool]] raises TypeError because second arg isn't a type (this is easy to fix).

2. X[int, ...] raises TypeError because for the same reason. One way to fix this is to automatically convert Ellipsis to EllipsisType just like how we already convert None to NoneType. Only problem now is that Callable[[...], int] doesn't raise TypeError. Should we just defer the problem of validating Callable to static type checkers instead?

class Z(Generic[P]): ...

3. Z[[int, str, bool]] raises TypeError, same as 1.

4. Z[int, str, bool] raises TypeError, but for too many parameters (not enough TypeVars). This one troubles me the most, current TypeVar substitution checks for len(args) == len(parameters).

I have a feeling your wish of greatly loosening type checks will come true ;). Should we proceed with that? That'd fix 1, 2, 3. The only showstopper now is 4. A quick idea is to just disable the check for len(args) == len(parameters) if there's only a single ParamSpec in __parameters__.

[gvanrossum]

class X(Generic[T, P]): ...

1. X[int, [int, bool]] raises TypeError because second arg isn't a type (this is easy to fix).

How would you fix it? By just allowing it? But then X.__args__ would be unhashable (see the other issue we're working on together).

2. X[int, ...] raises TypeError because for the same reason. One way to fix this is to automatically convert Ellipsis to EllipsisType just like how we already convert None to NoneType. Only problem now is that Callable[[...], int] doesn't raise TypeError. Should we just defer the problem of validating Callable to static type checkers instead?

I don't like using EllipsisType here, because this notation doesn't mean that we accept an ellipsis here -- it means (if I understand the PEP correctly) that we don't care about the paramspec.

class Z(Generic[P]): ...

3. Z[[int, str, bool]] raises TypeError, same as 1.

Same comment.

4. Z[int, str, bool] raises TypeError, but for too many parameters (not enough TypeVars). This one troubles me the most, current TypeVar substitution checks for len(args) == len(parameters).

The code should check that __parameters__ is (P,) for some ParamSpec P, and then transform this internally as if it was written Z[[int, str, bool]] and then typecheck that.

**BUT...**

How about an alternative implementation where as soon as we see that there's a ParamSpec in __parameters__ we just don't type-check at all, and accept anything? Leave it to the static type checker. Our goal here should be to support the syntax that PEP-612 describes so static type checkers can implement it, not to implement all the requirements described by the PEP at runtime.

I wonder what Pyre has in its stub files for ParamSpec -- maybe we can borrow that?

[gvanrossum]

For reference, here's what Pyre has (though it's an older version):

https://github.com/facebook/pyre-check/tree/master/pyre_extensions

[Fidget-Spinner]

The pyre version in their __init__.py looks like they took your advice for
letting the static checker do the work wholeheartedly.

I'm not in favour of type checking either. Just that the pre-existing code
does it for me.

Not type checking when seeing ~P in __parameters__ would work, just that
__args__ will be unhashable (like you mentioned) so things will be slower
due to no type cache. Maybe we can cast the [int, str] to (int, str), that
should work with cache for most cases. And unlike the Callable issue -
since we don't need to ensure runtime correctness - we can ignore any weird
effects to __args__ and __parameters__ in ParamSpec, like TypeVars not
collecting etc.

[gvanrossum]

I started doing this in the original code (long ago when PEP-484 was brand
new) but have since realized that this makes the typing module both slow
and hard to maintain. We should not follow this example. I do think we
should try to keep __args__ hashable, casting [int, str] to (int, str) sounds right.