List[<nothing>] error

[jcrmatos]

Hello,

Already showed this in Gitter and was told to open the issue here.
My env is Win7P+SP1 x64, Py 3.5.2 32b and mypy 0.501.

I have the following situation:

With the following code mypy doesn't return an error (which is correct):

from datetime import datetime, timedelta

data_dic = {idx: []
            for idx in range(len(lcl.msgs['DATA_COL_NAMES']) - 1)}  # type: Dict[int, List[Union[datetime, float]]]

But with the following code mypy returns a strange error

from datetime import datetime, timedelta

data_dic = {idx: []
            for idx in range(len(lcl.msgs['DATA_COL_NAMES']) - 1)}  # type: Dict[int, Union[List[datetime], List[float]]]

rtg_main.py:197: error: Value expression in dictionary comprehension has incompatible type List[<uninhabited>]; expected type "Union[List[datetime], List[float]]"

Best regards,

JM

[gvanrossum]

Maybe @ddfisher understands this? IMO it's always an error when <uninhabited> shows up in error messages?

[ilevkivskyi]

@gvanrossum

IMO it's always an error when <uninhabited> shows up in error messages?

Unfortunately, currently it is quite easy to get <uninhabited> in an error (TBH I don't like this). For example:

T = TypeVar('T')
def fun(x: List[T]) -> T:
    ...
fun(1) # Error: Argument 1 to "fun" has incompatible type "int"; expected List[<uninhabited>]

I think such errors are quite cryptic, so I set priority to high here.

[ddfisher]

My understanding is that this is a failure of type variable inference -- the context mechanism isn't working fully here. Prior to #3024, I think these error messages would reference List[None] instead of List[<uninhabited>], but would be similarly cryptic otherwise. The fact that uninhabited type is inferred here is a bug IMO and is tracked in #3032.

The fact that there's an error here at all is a problem with the context mechanism (perhaps related to Unions?).

[JukkaL]

In the original example, it looks like we don't infer a type because it's ambiguous due to the union type context. This is a known issue, though it might only be mentioned in a comment in the code. Here's a simpler repro:

from typing import Union, List
a = []  # type: Union[List[int], List[str]]

Here I can see a few plausible options:

• Just arbitrarily infer List[int] or List[str] as the type of []. However, this might not work in general and brings non-determinism to type checking.
• Infer Union[List[int], List[str]] for the type of []. This might be tricky to implement -- basically we'd need to support multiple possible answers from type inference, whereas currently we assume there's just one.

In @ilevkivskyi's example there is no context that can be used to infer things, so maybe we should fail type inference and generate an error about not being able to infer a value for a type variable. Alternatively, we could pick an arbitrary type variable value such as inferring List[object] in the example. This would likely be better than the current situation, though still a bit problematic.

[pkch]

Infer Union[List[int], List[str]] for the type of []. This might be tricky to implement -- basically we'd need to support multiple possible answers from type inference, whereas currently we assume there's just one.

@JukkaL
Why would you consider inferring Union[List[int], List[str]] to be "multiple possible answers"? Isn't Union a first-class type? If it is, then we should be able to infer it and not view it as anything unusual.

If Union is not a first-class type in some sense, then what are the obstacles to making it such?

[gvanrossum]

That's what I thought too -- after all if you use PEP 526 and write

a: Union[List[int], List[str]]  # No value

then a's type is indeed Union[List[int], List[str]].

However the point here is more subtle -- with the union-of-lists type, any append to the variable later will then flagged as an error. But perhaps we can adapt the machinery for partial types to our needs here? If a later line appends an int, change a to List[int], if a later line appends a str, change it to List[str]. Once changed to a non-union type, type-check with that type (so that appending an int and then a str is invalid). This would have the same limitations of the existing partial type inferencing machinery (e.g. if you use an operation it doesn't understand, it falls back to "Need annotation for type variable" -- would have to be changed for this case) but I think it would work relatively well.

I'd also like to challenge this a bit: what would be a real-world use case for writing such a statement in the first place? If we can't think of any, we could just make it an error.

[JukkaL]

Why would you consider inferring Union[List[int], List[str]] to be "multiple possible answers"? Isn't Union a first-class type? If it is, then we should be able to infer it and not view it as anything unusual.

Mypy internally infers T in List[T]. To move from List[T] to Union[List[int], List[float]] mypy would need to understand that T can be int or float. Inferring a union would only result in List[Union[int, float]] which would be incorrect.

[JukkaL]

I'd also like to challenge this a bit: what would be a real-world use case for writing such a statement in the first place? If we can't think of any, we could just make it an error.

Something like this might be kind of reasonable:

def f(x: Union[List[str], List[bytes]]) -> None:
    if <whatever>:
        x = []
    ...

I think that we could just as well fail to infer a type for the expression and require a cast. This should be fine:

from typing import Union, List, cast
a = cast(List[int], [])  # type: Union[List[int], List[str]]

This would also be fine, without a cast:

from typing import Union, List, cast
temp = []  # type: List[int]
a = temp  # type: Union[List[int], List[str]]

We already require casts semi-frequently so requiring it for an unusual edge case doesn't sound too bad. However, it would be better to generate an error message that would at least hint at a solution.

[jcrmatos]

I'd also like to challenge this a bit: what would be a real-world use case for writing such a statement in the first place? If we can't think of any, we could just make it an error.

My use case is a sort of pandas dataframe.
The dict will have 1 list (column) of datetime data and several lists (columns) of float data (these are readings from sensors).
So the correct typing should be
# type: Dict[int, Union[List[datetime], List[float]]]
but that returns the error on the first email.

Because each list (column) only has one type (datetime or float, never both in the same list), the typing shouldn't be
# type: Dict[int, List[Union[datetime, float]]]
which doesn't return an error message, but it isn't what I want.

[ilevkivskyi]

The idea that we could allow union types appeared in three separate contexts recently:

• bind types on initial assignments for unions, compromise option (4) in Types not added to binder on initial assignment #2008
• support overloads with unions (maybe only in simple cases, like single argument functions and/or optional types) see Should have a way to annotate an Optional carryover method. #3295
• instance joins: see Make joins symmetric and transitive? #3257 and Protocols #3132

The general idea that I see in these three cases is that we could use unions in simple cases instead of avoiding them. It looks like users actually don't find them scary (and even like them in case of TypeScript).

[gvanrossum]

The Pandas dataframe example might be better served with TypedDict. ( @jcrmatos beware that TypedDict still has some unfinished features. They're on the road map.)

[JukkaL]

The general idea that I see in these three cases is that we could use unions in simple cases instead of avoiding them. It looks like users actually don't find them scary (and even like them in case of TypeScript).

I agree that we'll probably need to have more support for union use cases, at least because strict optional checking makes unions a very common thing. Every user who uses strict optional checking needs to be understand unions anyway so there's now less need to avoid them.

[pkch]

@gvanrossum

However the point here is more subtle -- with the union-of-lists type, any append to the variable later will then flagged as an error.

And it would be correct to do so, in general. For example:

def f(x: Union[List[int], List[str]]):
    y: Union[List[int], List[str]]
    y = x
    y.append(1) # type error, as it should be

Even this is perfrectly reasonable IMO; what does the user expect when they declare a union and try to use it as if it's not:

y: Union[List[int], List[str]]
y = []
y.append(1) # type error; mypy can't figure out something so complex

Of course, in some cases mypy may be able to bind it more narrowly, but that's like a bonus feature, not something that the user should expect to happen in arbitrary situations:

y: Union[List[int], List[str]]
y = [1]
reveal_type(y)  # List[int]
y.append(2) # ok

@JukkaL

Why would you consider inferring Union[List[int], List[str]] to be "multiple possible answers"? Isn't Union a first-class type? If it is, then we should be able to infer it and not view it as anything unusual.

Mypy internally infers T in List[T]. To move from List[T] to Union[List[int], List[float]] mypy would need to understand that T can be int or float. Inferring a union would only result in List[Union[int, float]] which would be incorrect.

Ah I see, we just use different wording. I completely agree with you, it's just that when I say Union I think of it as a single type that intrinsically allows multiple possible non-union types. And you are just emphasizing that point by saying that mypy needs to be able to infer "multiple possible answers". Then, yes, I agree it would be nice to be able to allow multiple possible answers = support Union as a return type from the inference engine.