assert statements missed when loaded by zipimporter

[jaraco]

BPO	28131
Nosy	@Yhg1s, @brettcannon, @gpshead, @jaraco, @larryhastings, @benjaminp, @ned-deily, @berkerpeksag, @zooba, @zhangyangyu
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	issue28131.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-14.05:14:50.275>
created_at = <Date 2016-09-13.16:19:15.769>
labels = ['3.7', 'type-bug', 'library', 'release-blocker']
title = 'assert statements missed when loaded by zipimporter'
updated_at = <Date 2017-03-31.16:36:13.360>
user = 'https://github.com/jaraco'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:13.360>
actor = 'dstufft'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-14.05:14:50.275>
closer = 'berker.peksag'
components = ['Library (Lib)']
creation = <Date 2016-09-13.16:19:15.769>
creator = 'jaraco'
dependencies = []
files = ['44647']
hgrepos = []
issue_num = 28131
keywords = ['patch']
message_count = 10.0
messages = ['276296', '276311', '276359', '276360', '276369', '276371', '276372', '276376', '276377', '276378']
nosy_count = 11.0
nosy_names = ['twouters', 'brett.cannon', 'gregory.p.smith', 'jaraco', 'larry', 'benjamin.peterson', 'ned.deily', 'python-dev', 'berker.peksag', 'steve.dower', 'xiang.zhang']
pr_nums = ['552']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28131'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[jaraco]

Grabbing the recently released Python 3.6.0b1, I tried running one of my test suites, but found that some assertions were failing to assert when the package was loaded as a zip file (such as with pytest-runner installed dependencies). I distilled the issue to this:

$ cat > mod.py
def test(val):
        assert(val)
        print(val)
$ zip mod.zip mod.py
updating: mod.py (deflated 20%)
$ rm mod.py
$ python
Python 3.6.0b1 (v3.6.0b1:5b0ca4ed5e2f, Sep 12 2016, 09:24:46) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('mod.zip')
>>> import mod
>>> mod.test(False)
False
>>> mod.__loader__
<zipimporter object "mod.zip">
>>> sys.flags.optimize
0

I would have expected the call to mod.test to have raised an AssertionError, and on Python 3.5 it does.

I searched the what's new and didn't see anything advertising this change, so I suspect it's an unintentional regression.

I'm including Brett for his familiarity with importlib, but welcome re-assignment.

If I can do more to help, let me know.

[brettcannon]

I've added Greg and Thomas in case they have any ideas as they have looked at zipimport more recently than I have.

[gpshead]

This shouldn't be happening and makes no sense. It looks like the assert statement was removed at import code compilation time given the pdb trace with it from a zip file vs with it outside of a zip file:

>>> pdb.run('mod.test(False)')
> <string>(1)<module>()
(Pdb) n
False
--Return--
> <string>(1)<module>()->None
(Pdb) n
>>> pdb.run('mod.test(False)')
> <string>(1)<module>()->None
(Pdb) s
--Call--
> oss/cpython/build/mod.zip/mod.py(1)test()
-> def test(val):
(Pdb) s
> oss/cpython/build/mod.zip/mod.py(3)test()
-> print(val)
(Pdb) s
False
--Return--
> oss/cpython/build/mod.zip/mod.py(3)test()->None
-> print(val)
(Pdb) s
--Return--
> <string>(1)<module>()->None
(Pdb) s
> oss/cpython/default/Lib/bdb.py(435)run()
-> self.quitting = True
(Pdb) s
>>> 

vs no zip file:

>>> pdb.run('mod.test(False)')
> <string>(1)<module>()
(Pdb) s
--Call--
> oss/cpython/build/mod.py(1)test()
-> def test(val):
(Pdb) s
> oss/cpython/build/mod.py(2)test()
-> assert(val)
(Pdb) s
AssertionError
> oss/cpython/build/mod.py(2)test()
-> assert(val)
(Pdb) s
--Return--
> oss/cpython/build/mod.py(2)test()->None
-> assert(val)
(Pdb) s
AssertionError
> <string>(1)<module>()
(Pdb) s
--Return--
> <string>(1)<module>()->None
(Pdb) s
AssertionError
> oss/cpython/default/Lib/bdb.py(431)run()
-> exec(cmd, globals, locals)
(Pdb) s
> oss/cpython/default/Lib/bdb.py(432)run()
-> except BdbQuit:
(Pdb) s
> oss/cpython/default/Lib/bdb.py(435)run()
-> self.quitting = True
(Pdb) s
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "oss/cpython/default/Lib/pdb.py", line 1568, in run
    Pdb().run(statement, globals, locals)
  File "oss/cpython/default/Lib/bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "oss/cpython/build/mod.py", line 2, in test
    assert(val)
AssertionError

[gpshead]

from the zip:

>>> dis.dis(mod.test)
  3           0 LOAD_GLOBAL              0 (print)
              2 LOAD_FAST                0 (val)
              4 CALL_FUNCTION            1
              6 POP_TOP
              8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

from the filesystem:
>>> dis.dis(mod.test)
  2           0 LOAD_FAST                0 (val)
              2 POP_JUMP_IF_TRUE         8
              4 LOAD_GLOBAL              0 (AssertionError)
              6 RAISE_VARARGS            1

3 >> 8 LOAD_GLOBAL 1 (print)
10 LOAD_FAST 0 (val)
12 CALL_FUNCTION 1
14 POP_TOP
16 LOAD_CONST 0 (None)
18 RETURN_VALUE

[zhangyangyu]

This is introduced in 663a62bcf9c9. The compile optimize flag is opened in the change.

[zooba]

Bah, that's meant to be a -1 in that change, not 1. I must have typo'd when moving from the default branch to put it in 3.5.

Anyone's welcome to fix it, or I'll get it when I have time for CPython work.

[berkerpeksag]

Here is a patch with a test case.

[benjaminp]

lgtm

[python-dev]

New changeset 7bec326972f5 by Berker Peksag in branch '3.5':
Issue bpo-28131: Fix a regression in zipimport's compile_source()
https://hg.python.org/cpython/rev/7bec326972f5

New changeset 7a6c0c4e6072 by Berker Peksag in branch '3.6':
Issue bpo-28131: Merge from 3.5
https://hg.python.org/cpython/rev/7a6c0c4e6072

New changeset 873760b02024 by Berker Peksag in branch 'default':
Issue bpo-28131: Merge from 3.6
https://hg.python.org/cpython/rev/873760b02024

[berkerpeksag]

Thanks for the report and for the reviews!