Tests altered the execution environment in isolated mode

[serhiy-storchaka]

BPO	28655
Nosy	@vstinner, @serhiy-storchaka, @tirkarthi
PRs	[3.6] bpo-28655: Fix test_import.test_missing_source_legacy() #9589 [3.6] bpo-28655: Fix test bdb for isolate mode #10220
Dependencies	bpo-31837: ParseError in test_all_project_files()

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-29.07:14:57.314>
created_at = <Date 2016-11-10.08:46:58.206>
labels = ['3.7', 'type-bug', 'tests']
title = 'Tests altered the execution environment in isolated mode'
updated_at = <Date 2018-10-29.20:32:25.890>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-10-29.20:32:25.890>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-29.07:14:57.314>
closer = 'xtreak'
components = ['Tests']
creation = <Date 2016-11-10.08:46:58.206>
creator = 'serhiy.storchaka'
dependencies = ['31837']
files = []
hgrepos = []
issue_num = 28655
keywords = []
message_count = 12.0
messages = ['280497', '281274', '326459', '326466', '326468', '326474', '326475', '326477', '326479', '326480', '326670', '328866']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'xtreak']
pr_nums = ['9589', '10220']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28655'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[serhiy-storchaka]

Following tests altered the execution environment in isolated mode:

test_asyncio
test_ctypes
test_email
test_idle
test_import
test_importlib
test_json
test_lib2to3

$ ./python -I -S -m test.regrtest -vv test_asyncio test_ctypes test_email test_idle test_import test_importlib test_json test_lib2to3 >/dev/null
Warning -- sys.path was modified by test_asyncio
  Before: (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'])
  After:  (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/home/serhiy/py/cpython3.6-debug/Lib', '/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug']) 
Warning -- sys.path was modified by test_ctypes
  Before: (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'])
  After:  (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/home/serhiy/py/cpython3.6-debug/Lib', '/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug']) 
Warning -- sys.path was modified by test_email
  Before: (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'])
  After:  (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/home/serhiy/py/cpython3.6-debug/Lib', '/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug']) 
Warning -- sys.path was modified by test_idle
  Before: (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'])
  After:  (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/home/serhiy/py/cpython3.6-debug/Lib', '/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug']) 
Warning -- files was modified by test_import
  Before: []
  After:  ['@test_30748_tmp.pyc'] 
Warning -- sys.path was modified by test_importlib
  Before: (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'])
  After:  (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/home/serhiy/py/cpython3.6-debug/Lib', '/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug']) 
Warning -- sys.path was modified by test_json
  Before: (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'])
  After:  (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/home/serhiy/py/cpython3.6-debug/Lib', '/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug']) 
/home/serhiy/py/cpython3.6-debug/Lib/lib2to3/tests/test_parser.py:393: UserWarning: ParseError on file /home/serhiy/py/cpython3.6-debug/Lib/lib2to3/main.py (bad input: type=22, value='=', context=('', (130, 38)))
  warnings.warn('ParseError on file %s (%s)' % (filepath, err))
/home/serhiy/py/cpython3.6-debug/Lib/lib2to3/tests/test_parser.py:393: UserWarning: ParseError on file /home/serhiy/py/cpython3.6-debug/Lib/lib2to3/tests/pytree_idempotency.py (bad input: type=22, value='=', context=('', (49, 33)))
  warnings.warn('ParseError on file %s (%s)' % (filepath, err))
Warning -- sys.path was modified by test_lib2to3
  Before: (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'])
  After:  (3071075980, ['/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'], ['/home/serhiy/py/cpython3.6-debug/Lib', '/usr/local/lib/python36.zip', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/Lib/', '/home/serhiy/py/cpython3.6-debug/build/lib.linux-i686-3.6-pydebug'])

[serhiy-storchaka]

Most warnings are fixed in bpo-19398. The only warnings are left in test_import and test_lib2to3.

$ ./python -I -S -m test.regrtest -vv test_import test_lib2to3 >/dev/null
Warning -- files was modified by test_import
  Before: []
  After:  ['@test_15631_tmp.pyc'] 
/home/serhiy/py/cpython3.6/Lib/lib2to3/tests/test_parser.py:393: UserWarning: ParseError on file /home/serhiy/py/cpython3.6/Lib/lib2to3/main.py (bad input: type=22, value='=', context=('', (130, 38)))
  warnings.warn('ParseError on file %s (%s)' % (filepath, err))
/home/serhiy/py/cpython3.6/Lib/lib2to3/tests/test_parser.py:393: UserWarning: ParseError on file /home/serhiy/py/cpython3.6/Lib/lib2to3/tests/pytree_idempotency.py (bad input: type=22, value='=', context=('', (49, 33)))
  warnings.warn('ParseError on file %s (%s)' % (filepath, err))

[tirkarthi]

The dependency bpo-31837 dealing with test_lib2to3 failure was resolved by Victor in bpo-30117 and hence I marked it as a duplicate. The other issue regarding test_import is not reproducible on master as of 2aaf98c. test_import warning was reproducible with 14e976e and it seems to have been fixed later. The same issue with test_import was reported by Victor in bpo-30834 which was closed as outdated since it was not reproducible. I will try to bisect to the commit that fixed test_import. I propose closing this as outdated as part of triaging.

Thanks much Serhiy and Victor for the reports and fixes.

[vstinner]

I confirm that "./python -I -S -m test.regrtest -j0 --fail-env-changed test_asyncio test_ctypes test_email test_idle test_import test_importlib test_json test_lib2to3" now pass on master. I close the issue.

Thanks Karthikeyan ;-)

[serhiy-storchaka]

test_import still fails on 3.6.

[vstinner]

test_import still fails on 3.6.

Oh, I only tested master. I wrote PR 9589 to fix test_import on 3.6.

[tirkarthi]

Thanks Serhiy. Thanks Victor for the fix. I can confirm this on 3.6. Interestingly --fail-env-changed flag didn't catch this but running in verbose mode failed.

--fail-env-changed passes

➜ cpython git:(3707bcf) ./python.exe -I -S -m test.regrtest -j0 --fail-env-changed test_asyncio test_ctypes test_email test_idle test_import test_importlib test_json test_lib2to3
Run tests in parallel using 6 child processes
0:00:04 load avg: 3.51 [1/8] test_import passed
0:00:07 load avg: 3.51 [2/8] test_ctypes passed
0:00:09 load avg: 3.87 [3/8] test_idle passed
0:00:19 load avg: 6.23 [4/8] test_importlib passed
0:00:27 load avg: 6.05 [5/8] test_json passed
0:00:29 load avg: 5.72 [6/8] test_email passed
running: test_asyncio (59 sec 815 ms), test_lib2to3 (51 sec 972 ms)
0:01:06 load avg: 4.33 [7/8] test_asyncio passed (1 min 4 sec) -- running: test_lib2to3 (58 sec 276 ms)
0:01:09 load avg: 4.33 [8/8] test_lib2to3 passed (1 min 244 ms)

== Tests result: SUCCESS ==

All 8 tests OK.

# warnings are present

➜ cpython git:(3707bcf) ./python.exe -I -S -m test.regrtest -vv test_import test_lib2to3 >/dev/null
Warning -- files was modified by test_import
Before: []
After: ['@test_10093_tmp.pyc']

Total duration: 1 min 9 sec
Tests result: SUCCESS

[tirkarthi]

Verified the fix in the PR and there are no warnings

➜ cpython git:(pr_9589) ./python.exe
Python 3.6.6+ (heads/pr_9589:844abda318, Sep 26 2018, 20:59:26)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
➜ cpython git:(pr_9589) ./python.exe -I -S -m test.regrtest -vv test_import test_lib2to3 >/dev/null
➜ cpython git:(pr_9589) ./python.exe -I -S -m test.regrtest -j0 --fail-env-changed test_asyncio test_ctypes test_email test_idle test_import test_importlib test_json test_lib2to3
Run tests in parallel using 6 child processes
0:00:06 load avg: 3.19 [1/8] test_import passed
0:00:10 load avg: 3.73 [2/8] test_ctypes passed
0:00:12 load avg: 3.73 [3/8] test_idle passed
0:00:20 load avg: 4.08 [4/8] test_importlib passed
0:00:27 load avg: 4.07 [5/8] test_json passed
0:00:29 load avg: 4.15 [6/8] test_email passed
running: test_asyncio (59 sec 700 ms), test_lib2to3 (49 sec 60 ms)
0:01:04 load avg: 3.97 [7/8] test_asyncio passed (1 min 2 sec) -- running: test_lib2to3 (54 sec 18 ms)
0:01:07 load avg: 3.97 [8/8] test_lib2to3 passed (55 sec 380 ms)

== Tests result: SUCCESS ==

All 8 tests OK.

Total duration: 1 min 7 sec
Tests result: SUCCESS

Thanks

[vstinner]

I can confirm this on 3.6. Interestingly --fail-env-changed flag didn't catch this but running in verbose mode failed.

Oh... When using -j0, regrtest doesn't spawn worker processes using -I: the flag is simply omitted... As a side effect, forget() in test_import doesn't remove pyc_file from the current directory.

I created bpo-34812 to track this issue.

[vstinner]

New changeset 43500a5 by Victor Stinner in branch '3.6':
bpo-28655: Fix test_import.test_missing_source_legacy() (GH-9589)
43500a5