Compileall improvements

[encukou]

BPO	38112
Nosy	@gpshead, @vstinner, @encukou, @frenzymadness, @pablogsal
PRs	bpo-38112: Compileall improvements #16012 bpo-38112: compileall: Skip long path path on Windows if the path can't be created #16414 bpo-38112: compileall tests: Use shorter name for long_path test #16419 bpo-38112: Document that compileall.compile_[dir,file] also accept multiple opt levels #20174

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-10-15.13:40:03.440>
created_at = <Date 2019-09-11.14:01:19.510>
labels = ['expert-installation']
title = 'Compileall improvements'
updated_at = <Date 2020-05-18.13:23:44.140>
user = 'https://github.com/encukou'

bugs.python.org fields:

activity = <Date 2020-05-18.13:23:44.140>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-15.13:40:03.440>
closer = 'vstinner'
components = ['Installation']
creation = <Date 2019-09-11.14:01:19.510>
creator = 'petr.viktorin'
dependencies = []
files = []
hgrepos = []
issue_num = 38112
keywords = ['patch']
message_count = 11.0
messages = ['351887', '353266', '353271', '353272', '353275', '353286', '353319', '354610', '354728', '362783', '369215']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'vstinner', 'petr.viktorin', 'frenzy', 'pablogsal']
pr_nums = ['16012', '16414', '16419', '20174']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38112'
versions = []

[encukou]

Hello,
In Fedora, we've long used a Bash script to compile modules to bytecode, as the compileall module was historically unsuitable.
Recently, Lumír re-wrote our enhancements to compileall, and published the result as compileall2 on PyPI. We'd like to get the improvements back into the standard library so everyone can benefit.
The improvements are:

• Raise the default recursion limit
• Control of the path baked into the *.pyc file (-s to strip a prefix; -p to prepend a new one) to support different build/install locations.
• Compiling for multiple optimization levels at once (by specifying -o multiple times)

A PR should follow shortly.

[encukou]

New changeset 8e7bb99 by Petr Viktorin (Lumír 'Frenzy' Balhar) in branch 'master':
bpo-38112: Compileall improvements (GH-16012)
8e7bb99

[vstinner]

test_compileall failed on AMD64 Windows8.1 Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/12/builds/3257

Example:

ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_compileall.py", line 48, in setUp
    os.makedirs(self.long_path)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 18 more times]
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [WinError 206] The filename or extension is too long: 'D:\\temp\\tmpcgsk1z3q\\long\\1\\2\\3\\4\\5\\6\\7\\8\\9\\10\\11\\12\\13\\14\\15\\16\\17\\18\\19\\20\\21\\22\\23\\24\\25\\26\\27\\28\\29\\30\\31\\32\\33\\34\\35\\36\\37\\38\\39\\40\\41\\42\\43\\44\\45\\46\\47\\48\\49\\50\\51\\52\\53\\54\\55\\56\\57\\58\\59\\60\\61\\62\\63\\64\\65\\66\\67\\68\\69\\70\\71\\72\\73\\74\\75\\76\\77\\78'

Failures:

ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
FAIL: test_no_args_respects_force_flag (test.test_compileall.CommandLineTestsWithSourceEpoch)
Re-running failed tests in verbose mode
Re-running test_tarfile in verbose mode
Re-running test_compileall in verbose mode
ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)

pythoninfo says:

sys.windowsversion: sys.getwindowsversion(major=6, minor=3, build=9600, platform=2, service_pack='')
windows.RtlAreLongPathsEnabled: <function not available>

[vstinner]

Similar errors on AMD64 Windows10 3.x:
https://buildbot.python.org/all/#/builders/3/builds/3567

pythoninfo:

sys.windowsversion: sys.getwindowsversion(major=10, minor=0, build=17134, platform=2, service_pack='')
windows.RtlAreLongPathsEnabled: False

--

And AMD64 Windows7 SP1 3.x:
https://buildbot.python.org/all/#/builders/40/builds/3180

pythoninfo:

sys.windowsversion: sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1')
windows.RtlAreLongPathsEnabled: <function not available>

[encukou]

Thanks!
The buildbot complained in the pull request a while ago; I commented there that I'm looking into it.

I'd appreciate a review for the fix; I don't have a good way to test it other than offer it to the buildbots.

[encukou]

New changeset 4267c98 by Petr Viktorin in branch 'master':
bpo-38112: compileall: Skip long path path on Windows if the path can't be created (GH-16414)
4267c98

[vstinner]

FYI test_compileall.test_compile_dir_maxlevels() also fails on AMD64 FreeBSD CURRENT Shared 3.x:
https://buildbot.python.org/all/#builders/168/builds/1529

[pablogsal]

The AMD64 Windows7 SP1 3.x buildbot has been failing for a while since commit 4267c98, please take a look at https://bugs.python.org/issue38470.

[vstinner]

Should be fixed by:

New changeset eb1dda2 by Petr Viktorin (Victor Stinner) in branch 'master':
bpo-38470: Fix test_compileall.test_compile_dir_maxlevels() (GH-16789)
eb1dda2

[gpshead]

https://bugs.python.org/issue39769 may be of interest. compileall's ddir= does not behave properly in recursive subdirectories, it did the right thing in 2.7 which is similar to a combination of compile_dir(dir, prependdir=ddir, stripdir=dir) after this issue's feature.

[vstinner]

New changeset adc72bb by Lumír 'Frenzy' Balhar in branch 'master':
bpo-38112: Document that compileall.compile_[dir,file] also accept multiple opt levels (GH-20174)
adc72bb