Windows build reproducibility broken on master (`distlib` calls stdlib zipfile.writestr() which embeds time.time() into some files)

[SomberNight]

Currently on master the Windows builds are no longer reproducible.
I think I've managed to track down the root cause.

• we bumped the pyinstaller version used in the Windows builds from ~4.2 to ~4.10 recently in b5951ad
• pyinstaller made a change in 4.4 (Hook-utils: copy_metadata(): Preserve metadata folder name. pyinstaller/pyinstaller#5774) where they started including "dist-info" package metadata inside the final executable, e.g. wheel-0.37.1.dist-info/
• I've found that *.dist-info/RECORD files are not reproducible on Windows for packages that define entry_points/console_scripts that go into the Scripts/ folder when installed by pip. Details below.

---

For example, in wheel-0.37.1.dist-info/RECORD I have this line:

../../Scripts/wheel.exe,sha256=u9TbPw2XNs_F9uy7y2zwumuzAZDbOSB7BXjLHZ0tTHg,97103

Or in pyinstaller-4.10.dist-info/RECORD, I have these lines:

../../Scripts/pyi-archive_viewer.exe,sha256=nC-9muPlIhUC1qvFkXHpyKJyRQqXISXxbUPXQ1XVOiM,97133
../../Scripts/pyi-bindepend.exe,sha256=udFHiAdndPpSwaIqmhmLEy36IUs1cNNoNQznSEnLJQQ,97128
../../Scripts/pyi-grab_version.exe,sha256=3ET9E841tFWujFL99aG4frzgwlP9f9pAkMgE0k2UGK0,97131
../../Scripts/pyi-makespec.exe,sha256=dJkfmITdLJhyPngmqziqqj5tH9qqfeQc5BTubeoXWUs,97127
../../Scripts/pyi-set_version.exe,sha256=sWmcOVS93fUY-wbdoz6ixBCvjy1tC4Aaw30DMmrmo-0,97130
../../Scripts/pyinstaller.exe,sha256=haInbhH0pImJn24cW4v917oUZmzXZj8OE89KFh4MO2Y,97112

These exe files and hence the RECORD files themselves are not reproducible.
(note that pyinstaller is not packaging these executables inside the final distributable however it is packaging the dist-info/ folders that list the hashes)
Upon comparing multiple Scripts/wheel.exe files, I've found that the only difference is due to a timestamp embedded inside the exe (or rather, same timestamp embedded twice).

I have found where the exe files get created.
They are created by distlib, a library vendored by pip.
Here is a traceback with an artificial exception to illustrate the codepath:

(env) PS C:\tmp> pip install --no-build-isolation pyinstaller
Collecting pyinstaller
  Using cached pyinstaller-4.10-py3-none-win_amd64.whl (2.0 MB)
Requirement already satisfied: setuptools in c:\tmp\env\lib\site-packages (from pyinstaller) (61.0.0)
Requirement already satisfied: pyinstaller-hooks-contrib>=2020.6 in c:\tmp\env\lib\site-packages (from pyinstaller) (2022.3)
Requirement already satisfied: altgraph in c:\tmp\env\lib\site-packages (from pyinstaller) (0.17.2)
Requirement already satisfied: pefile>=2017.8.1 in c:\tmp\env\lib\site-packages (from pyinstaller) (2021.9.3)
Requirement already satisfied: pywin32-ctypes>=0.2.0 in c:\tmp\env\lib\site-packages (from pyinstaller) (0.2.0)
Requirement already satisfied: future in c:\tmp\env\lib\site-packages (from pefile>=2017.8.1->pyinstaller) (0.18.2)
Installing collected packages: pyinstaller
ERROR: Exception:
Traceback (most recent call last):
  File "C:\tmp\env\lib\site-packages\pip\_internal\cli\base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "C:\tmp\env\lib\site-packages\pip\_internal\cli\req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "C:\tmp\env\lib\site-packages\pip\_internal\commands\install.py", line 405, in run
    installed = install_given_reqs(
  File "C:\tmp\env\lib\site-packages\pip\_internal\req\__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "C:\tmp\env\lib\site-packages\pip\_internal\req\req_install.py", line 769, in install
    install_wheel(
  File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 729, in install_wheel
    _install_wheel(
  File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 646, in _install_wheel
    generated_console_scripts = maker.make_multiple(scripts_to_generate)
  File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 440, in make_multiple
    filenames.extend(self.make(specification, options))
  File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 427, in make
    return super().make(specification, options)
  File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 429, in make
    self._make_script(entry, filenames, options=options)
  File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 329, in _make_script
    self._write_script(scriptnames, shebang, script, filenames, ext)
  File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 263, in _write_script
    raise Exception(f"heyheyhey2. {sha256(launcher)=}. {sha256(shebang)=}. {sha256(zip_data)=}. " +
Exception: heyheyhey2. sha256(launcher)='a00a877acefc'. sha256(shebang)='58628e924f22'. sha256(zip_data)='a423496a0482'. ('SOURCE_DATE_EPOCH' in os.environ)=True

The interesting code is here:
https://github.com/pypa/distlib/blob/d0e3f49df5d1aeb9daeaaabf0391c9e13e4a6562/distlib/scripts.py#L251-L252
This calls into the cpython standard library, where time.time() gets written into the file:
https://github.com/python/cpython/blob/20e6e5636a06fe5e1472062918d0a302d82a71c3/Lib/zipfile.py#L1816-L1817

zinfo = ZipInfo(filename=zinfo_or_arcname,
                date_time=time.localtime(time.time())[:6])

Ideally, either distlib or the stdlib zipfile module should be changed to respect SOURCE_DATE_EPOCH.

[SomberNight]

We need a workaround for this for the next release.

[SomberNight]

Sort of related: https://bugs.python.org/issue45344 ("Have zipapp respect SOURCE_DATE_EPOCH")

[SomberNight]

I've opened issues with both distlib and cpython, and submitted patches:
pypa/distlib#164
pypa/distlib#165
https://bugs.python.org/issue47123