bpo-38470: compileall tests: Use a shorter path on Windows

[encukou]

I'd like to test compileall with a really deeply nested path everywhere, but I cannot reproduce the issue occuring on Windows.
This limits the maximum path depth to 20 on Windows. That's enough to test the compileall functionality.

cc @frenzymadness

https://bugs.python.org/issue38470

[frenzymadness]

Thank you for working on this!

[encukou]

The buildbot fails with this as well: https://buildbot.python.org/all/#/builders/42/builds/59
I have no idea what's wrong. Just this one bot is failing.
@vstinner, I think you mentioned you could reproduce this issue, is that right?

[vstinner]

The test still fails on my Windows 10 VM using my patch to disable long path support: https://bugs.python.org/issue38470#msg354636

vstinner@WIN C:\vstinner\python\master>python -m test test_compileall -v -m test_compile_dir_maxlevels
Running Debug|x64 interpreter...
== CPython 3.9.0a0 (heads/pr/16778-dirty:30f91ab592, Oct 14 2019, 17:09:51) [MSC v.1916 64 bit (AMD64)]
== Windows-10-10.0.18362-SP0 little-endian
== cwd: C:\vstinner\python\master\build\test_python_6600
== CPU count: 2
== encodings: locale=cp1252, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_compileall
test_compile_dir_maxlevels (test.test_compileall.CompileallTestsWithSourceEpoch) ... *** Error compiling 'C:\\Users\\vstinner\\AppData\\Local\\Temp\\tmpfwqi2hwl\\long\\dir_0_0\\dir_0_
1\\dir_0_2\\dir_0_3\\dir_0_4\\dir_0_5\\dir_0_6\\dir_0_7\\dir_0_8\\dir_0_9\\dir_1_0\\dir_1_1\\dir_1_2\\dir_1_3\\dir_1_4\\dir_1_5\\dir_1_6\\dir_1_7\\dir_1_8\\dir_1_9\\_test_long.py'... 
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\vstinner\\AppData\\Local\\Temp\\tmpfwqi2hwl\\long\\dir_0_0\\dir_0_1\\dir_0_2\\dir_0_3\\dir_0_4\\dir_0_5\\dir_0_6\\d
ir_0_7\\dir_0_8\\dir_0_9\\dir_1_0\\dir_1_1\\dir_1_2\\dir_1_3\\dir_1_4\\dir_1_5\\dir_1_6\\dir_1_7\\dir_1_8\\dir_1_9\\__pycache__\\_test_long.cpython-39.pyc.1893991391632'
FAIL
test_compile_dir_maxlevels (test.test_compileall.CompileallTestsWithoutSourceEpoch) ... *** Error compiling 'C:\\Users\\vstinner\\AppData\\Local\\Temp\\tmpvga6g_oc\\long\\dir_0_0\\dir
_0_1\\dir_0_2\\dir_0_3\\dir_0_4\\dir_0_5\\dir_0_6\\dir_0_7\\dir_0_8\\dir_0_9\\dir_1_0\\dir_1_1\\dir_1_2\\dir_1_3\\dir_1_4\\dir_1_5\\dir_1_6\\dir_1_7\\dir_1_8\\dir_1_9\\_test_long.py'.
..
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\vstinner\\AppData\\Local\\Temp\\tmpvga6g_oc\\long\\dir_0_0\\dir_0_1\\dir_0_2\\dir_0_3\\dir_0_4\\dir_0_5\\dir_0_6\\d
ir_0_7\\dir_0_8\\dir_0_9\\dir_1_0\\dir_1_1\\dir_1_2\\dir_1_3\\dir_1_4\\dir_1_5\\dir_1_6\\dir_1_7\\dir_1_8\\dir_1_9\\__pycache__\\_test_long.cpython-39.pyc.1893991707360'
FAIL

======================================================================
FAIL: test_compile_dir_maxlevels (test.test_compileall.CompileallTestsWithSourceEpoch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\vstinner\python\master\lib\test\test_py_compile.py", line 30, in wrapper
    return fxn(*args, **kwargs)
  File "C:\vstinner\python\master\lib\test\test_compileall.py", line 264, in test_compile_dir_maxlevels
    self.assertTrue(os.path.isfile(self.bc_path_long))
AssertionError: False is not true

======================================================================
FAIL: test_compile_dir_maxlevels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\vstinner\python\master\lib\test\test_py_compile.py", line 20, in wrapper
    return fxn(*args, **kwargs)
  File "C:\vstinner\python\master\lib\test\test_compileall.py", line 264, in test_compile_dir_maxlevels
    self.assertTrue(os.path.isfile(self.bc_path_long))
AssertionError: False is not true

----------------------------------------------------------------------

Ran 2 tests in 0.246s

FAILED (failures=2)
test test_compileall failed
test_compileall failed

== Tests result: FAILURE ==

1 test failed:
    test_compileall

Total duration: 704 ms
Tests result: FAILURE

[vstinner]

The path created by create_long_path() is smaller than the limit, but py_compile uses _write_atomic() of importlib._bootstrap_external which adds a ".xxx" suffix:

# id() is used to generate a pseudo-random filename.
path_tmp = '{}.{}'.format(path, id(path))

With this suffix, the temporary filename is too long: 261 characters on my VM.

[vstinner]

I proposed a different approach: PR #16789.

[encukou]

Aha! Thank you! I'll get on it tomorrow

…

On October 14, 2019 5:59:31 PM GMT+02:00, Victor Stinner ***@***.***> wrote: The path created by create_long_path() is smaller than the limit, but py_compile uses _write_atomic() of importlib._bootstrap_external which adds a ".xxx" suffix: ``` # id() is used to generate a pseudo-random filename. path_tmp = '{}.{}'.format(path, id(path)) ``` With this suffix, the temporary filename is too long: 261 characters on my VM. -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #16778 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[vstinner]

GH-16789 has been merged instead.