Defaults for Generics?

[srittau]

Consider the following (simplified) typeshed definition for email.feedparser.FeedParser:

class FeedParser:
    def __init__(self, _factory: Callable[[], Message] = ...) -> None: ...
    def close(self) -> Message: ...

FeedParser allows a message factory to be passed in and will use it to construct the return value to close(). Unfortunately you have to cast the return value of close() to the correct type:

parser = FeedParser(MyMessage)
message = cast(MyMessage, parser.close())

The following type definition fixes this problem:

M = TypeVar("M", bound=Message)

class FeedParser(Generic[M]):
    def __init__(self, _factory: Callable[[], M] = ...) -> None: ...
    def close(self) -> M: ...

Now the example above does not require the cast to make mypy happy. But unfortunately using the default factory will not work anymore:

parser = FeedParser()
message = parser.close()

This will cause mypy to complain error: Need type annotation for variable. Is there any solution for this conundrum?

[emmatyping]

#4227 is somewhat related, but the underlying issue is that since mypy doesn't know what the return type of the _factory() Callable is (since the default is of the uninhabited type), mypy cannot deduce the return type to close().

It seems to me we might be able to fall back to the bound type, but I'm not sure about that.

[srittau]

Would having a "default" type for TypeVars make sense? Falling back to the bound type makes sense with my example above, but I am also unsure whether that is the right thing to do for the general case.

[chadrik]

I've run into a similar problem, where I intuitively wanted to add a default value in my stubbed function to affect the outcome:

M = TypeVar('M', bound=Mapping)

def asdict(inst: Any, recurse: bool = ..., dict_factory: M = dict) -> M: ...

reveal_type(asdict(foo))  # should be `dict`

[JukkaL]

There seem to be two separate things going on here:

• Infer a default value for type variable from the type of a default argument value (@chadrik's example above). I've wanted this once or twice. Overloading seems like a reasonable workaround, though.
• Give an explicit default value for a type variable. The scope could be for all instances of type variable or just for a particular class/function.
  • If it's for all instances, we may need a change to TypeVar, such as T = TypeVar('T', default=str). This needs to be proposed at https://github.com/python/typing.
  • If it's for a particular definition, we could use a class/function decorator such as @mypy_extensions.type_var_default(T, str). This could well be mypy-specific.

I think that the first change is not very controversial, but pretty low priority since there is a workaround. For the second change we'd need more evidence of it being generally useful.

[chadrik]

I think that the first change is not very controversial, but pretty low priority since there is a workaround.

I can agree with this as long as this use case is taken into consideration during the overflow overhaul (henceforth known as the overflowhaul). Currently I can't solve this without running into overlapping overload errors.

[srittau]

If I understand correctly, the first suggestion would also allow us to write the FeedParser definition as:

M = TypeVar("M", bound=Message)

class FeedParser(Generic[M]):
    def __init__(self, _factory: Callable[[], M] = Message) -> None: ...
    def close(self) -> M: ...

I think that's a fine solution.

[ilevkivskyi]

@JukkaL

For the second change we'd need more evidence of it being generally useful.

It looks like this pattern is used extensively in SQLAlchemy. A simplified example:

class Null: ...

class Clause(Generic[T]):
    def __init__(self, type: Optional[Type[T]] = None) -> None:
        if type is None:
            type = Null  # Something like this happens at runtime
        self.type = type

reveal_type(Clause(int))  # OK, `Clause[int]`
reveal_type(Clause())  # Bad, `Clause[<nothing>]`, should be `Clause[Null]`

A potential solution is to use an overloaded __new__ but unfortunately it doesn't work.

I think this can be raised to at least normal priority.

[bryanforbes]

IMO, I think the T = TypeVar('T', default=str) syntax would be the most robust in the long-run. TypeScript has a similar syntax and it's very handy:

class Clause<T = Null> {
    type: T;

    constructor(type?: T) {
        this.type = type ? type() : Null();
    }
}

let c = new Clause()
c.type // type is Null
let d = new Clause(Integer);
d.type // type is Integer

This could also potentially be combined with the bound parameter of TypeVar() to provide both the base type of the generic and a default value. @ilevkivskyi's example above would look like this:

class Null: ...

T = TypeVar('T', default=Null)
class Clause(Generic[T]):
    type: T
    def __init__(self, , type: Optional[Type[T]] = None) -> None:
        if type is None:
            type = Null  # Something like this happens at runtime
        self.type = type()

reveal_type(Clause(int))  # OK, `Clause[int]`
reveal_type(Clause()) # Ok, `Clause[Null]`

Then a developer (of either stubs or a library) wouldn't need to mess with __new__ to type this pattern.

[bryanforbes]

I forgot to give an example of default and bound:

class Comparator(Protocol):
    def compare(self, other: Any) -> bool: ...

class Null:
    def compare(self, other: Any) -> bool: ...

class Integer:
    def compare(self, other: Any) -> bool: ...

T = TypeVar('T', bound=Comparator, default=Null)
class Clause(Generic[T]):
    type: T
    def __init__(self, , type: Optional[Type[T]] = None) -> None:
        if type is None:
            type = Null  # Something like this happens at runtime
        self.type = type()

reveal_type(Clause(Integer))  # OK, `Clause[Integer]`
reveal_type(Clause()) # Ok, `Clause[Null]`
reveal_type(Clause(int)) # Value of type variable "T" of "Clause" cannot be "int"
reveal_type(c = cast(Clause[Integer], {}))  # Ok, `Clause[Integer]`
reveal_type(c = cast(Clause, {}))  # Ok, `Clause[Null]`
reveal_type(c = cast(Clause[int], {})) # Type argument "builtins.int" of "Clause" must be a subtype of "Comparator"