AssertionError in visit_member_expr with mypy caching

[timabbott]

Since we upgraded to 0.4.6 a few weeks ago, I've been getting repeated reports of folks encountering this mypy caching bug when using mypy in the Zulip project. It's reproducible running on just one file with the cache file.

$ ./tools/run-mypy zproject/local_settings_template.py
# Expands to:
$ mypy --fast-parser --silent-imports --check-untyped-defs --scripts-are-modules -i --cache-dir=var/mypy-cache --show-traceback --py2 --disallow-untyped-defs zproject/local_settings.py
zproject/local_settings.py:13: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues
Traceback (most recent call last):
  File "/srv/zulip-py3-venv/bin/mypy", line 6, in <module>
    main(__file__)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/main.py", line 41, in main
    res = type_check_only(sources, bin_dir, options)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/main.py", line 86, in type_check_only
    options=options)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 183, in build
    dispatch(sources, manager)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1512, in dispatch
    process_graph(graph, manager)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1692, in process_graph
    process_stale_scc(graph, scc)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1767, in process_stale_scc
    graph[id].semantic_analysis()
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1425, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath, self.options)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 246, in visit_file
    self.accept(d)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 2758, in accept
    node.accept(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/nodes.py", line 812, in accept
    return visitor.visit_assignment_stmt(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 1153, in visit_assignment_stmt
    s.rvalue.accept(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/nodes.py", line 1247, in accept
    return visitor.visit_call_expr(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 2230, in visit_call_expr
    expr.callee.accept(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/nodes.py", line 1202, in accept
    return visitor.visit_member_expr(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 2330, in visit_member_expr
    base.accept(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/nodes.py", line 1202, in accept
    return visitor.visit_member_expr(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 2337, in visit_member_expr
    assert isinstance(file, MypyFile)
AssertionError: 
zproject/local_settings.py:13: note: use --pdb to drop into pdb

That's the error I got on 0.4.6. If I try using master instead, the error moves to running on scripts/setup/generate_secrets.py.

Here's a tarball of the mypy cache directory involved: mypy-cache-bug.tar.gz

I don't know exactly what gets the cache in this state (running mypy twice after deleting the cache doesn't do it), but the issue does keep coming back for folks if one just deletes the cache.

[ambv]

Note: the latest version is mypy==0.470.

[timabbott]

It's not on pypi though, so I just went to master rather than installing that release tarball: https://pypi.python.org/pypi/mypy-lang

[JukkaL]

Note that the PyPI package name was recently changed to mypy from mypy-lang.

[JukkaL]

(And mypy-lang won't be updated any more and will be stuck at 0.4.6.)

[timabbott]

Ahh. Is there an announcement list I can subscribe to in order to hear about things like this?

[JukkaL]

You can subscribe to the mypy blog or the mypy project twitter feed (the latter is very low-volume). We don't have an active mailing list right now.

[ddfisher]

Is this still an error on mypy>=0.470?

[timabbott]

Yep, as I mentioned it fails on master as well with that cache, though with a different file in the codebase.

…

-Tim Abbott (mobile) On Jan 24, 2017 15:13, "David Fisher" <notifications@github.com> wrote: Is this still an error on mypy>=0.470? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2751 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACnm2tMqi3S8TEKonj5Kynmi1Cbb_LpOks5rVoWAgaJpZM4Lr8lq> .

[ddfisher]

Ah, sorry that I missed that!

[gvanrossum]

Thanks for the report. I'm not sure that the cache dump is enough to repro the issue, because the cache stores the mtimes of all files, and if these don't match for a specific file, its cache entry is ignored. But there's a clue in the line number given at the top of the traceback -- it's

config_file = six.moves.configparser.RawConfigParser() # type: ignore # https://github.com/python/typeshed/pull/206

So I assume that the error you get with a newer mypy on scripts/setup/generate_secrets.py is for line 54:

    secrets_file = six.moves.configparser.RawConfigParser() # type: ignore # https://github.com/python/typeshed/issues/307

I expect I can take it from here.

[gvanrossum]

OK, so here's a repro. Using mypy --incremental --ignore-missing-imports run the following:

import six.moves.configparser

Then edit the file to this instead:

import six.moves.configparser
six.moves.configparser.RawConfigParser()  # type: ignore

and run again. You now get the traceback. Using --pdb we can see that file is None.

I will debug further when I can find some more time.