Running the Python test suite leaks .pem files in /tmp

[vstinner]

Running the Python test suite has two issues:

• A test leaks 3 .pem files in /tmp
• The test suite is not marked as failed when it leaks temporary files (in /tmp)

Moreover, test_tools enters an unlimited loop and fills the $TMPDIR directory if the $TMPDIR is a sub-directory of the Python source code directory. Example:

• /home/vstinner/python/main/ : Python source code
• /home/vstinner/python/main/TMP/ : Temporary directory ($TMPDIR)

Running test_freeze_simple_script() of test_tools copies TMP/ into TMP/TMP/ and then into TMP/TMP/TMP/, etc. Quickly, it fills TMP/ with a "loop" of files :-)

Linked PRs

• gh-93353: regrtest checks for leaked temporary files #93776

[vstinner]

The problem comes from pip bootstrap when running the test_with_pip() test of test_venv.

It seems like pip creates a .pem file in the temporary directory (like /tmp), but only when pip is imported from a ZIP file, which is the case when I run get-pip.py: https://bootstrap.pypa.io/get-pip.py

[vstinner]

I can reproduced the issue in the 3.9, 3.10, 3.11 and main branches (I didn't test older branches).

• In Python 3.9, I cannot reproduce the issue in Python 3.9 at commit 4b4d60f: pip 20.2.3.
• But I can reproduce the issue at commit d962b00: pip 21.1.

It seems to be a change between pip 20.2.3 and pip 21.1.

[vstinner]

In get-pip.py, from pip._internal.commands.install import InstallCommand is enough to create the .pem file.

[vstinner]

Shell script to reproduce the issue:

set -e -x

DIR=$PWD/TMP/
rm -rf $DIR
mkdir $DIR

rm -rf env
TMPDIR=$DIR TEMPDIR=$DIR ./python -m venv env --without-pip
TMPDIR=$DIR TEMPDIR=$DIR env/bin/python -m ensurepip -v

echo
echo
echo "=== TMPDIR ==="
find $DIR

Output:

(...)
=== TMPDIR ===
++ find /home/vstinner/python/main/TMP/
/home/vstinner/python/main/TMP/
/home/vstinner/python/main/TMP/tmpmb5ndxc7cacert.pem

[vstinner]

The PEM file is created indirectly by importlib.resources on import pip._vendor.requests.

It's created by this code:

>>> import sys; sys.path.insert(0, 'pip.zip')
>>> import pip._vendor.certifi
>>> pip._vendor.certifi.where()
pip.zip/pip/_vendor/certifi/core.py:50: DeprecationWarning: path is deprecated. Use files() instead. Refer to https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy for migration advice.
'/home/vstinner/python/main/TMP/tmp9osgb6wrcacert.pem'

Debugger:

Lib/importlib/resources/_common.py, line 84, in _tempfile (suffix='cacert.pem', fd=4, raw_path='/home/vstinner/python/main/TMP/tmpv7fwzaoocacert.pem')

If Python exits normally, the garbage collector deletes the temporary PEM file. When running ensurepip / get-pip.py, it's not deleted. I don't know why.

importlib.resources only creates a temporary file if pip is imported from a ZIP file.

[vstinner]

Workaround to manually delete the temporaryfile:

pip._vendor.certifi.core._CACERT_CTX = None

[vstinner]

OpenSSL requires a filename to a PEM file. certifi uses importlib.resources to create a temporary named file if pip._vendor.certifi is imported from a ZIP file. That's convenient.

The problem is more that the pip._vendor.certifi.core._CACERT_CTX context manager which keeps the temporary file alive until it's destroyed. Maybe sometimes _CACERT_CTX is part of a complex reference cycle, it's never destroyed, and so the temporary file is not deleted.

[serhiy-storchaka]

Nice. How did you find this?

[vstinner]

I used a shell script similar to #93353 (comment) and I ran manually a bisection on the test list (./python -m test --list-tests) using a script taking a random.sample() of half of tests. I repeated the operation until I found a single test file, then I looked at the code.

The leaked files was reported by more and more owners of buildbot workers.

Once the PEM issue is solved, my plan is to enhance test.regrtest to do something similar than the shell script: define TMPFILE env var. Maybe in the main process spawning test processes.

[vstinner]

For pip/certifi, I hacked get-pip.py to add breakpoint() and I followed the code flow. It's not easy to identify which part of pip created the file. It's not just an import. It's an import + call to where() function, done in the module body that I failed to find which module. Maybe requests. It would be nice to be able to postpone when where() is called to workaround the issue: only call it when a filename to cacert.pem is needed.

[vstinner]

Reproducer without pip nor certifi, attached run.py and bug.py scripts:

$ ./python run.py 
bug.py: temporary directory: ['tmp1wkab03ycacert.pem']
run.py: temporary directory: ['tmp1wkab03ycacert.pem']

The expected output is an empty list in run.py (parent process). bug.py never calls _tempfile() finally block.

I'm still bisecting the issue. It smells like logging prevents deleting an object somehow.

• bug.py.txt
• run.py.txt

[vstinner]

Ok, this problem is quite complicated:

• pip imports requests
• requests imports certifi
• requests calls certifi.where()
• certifi.where() uses importlib.resources to create a temporary file using a context manager.
• logging is imported by pip
• logging calls os.register_at_fork() which keeps the whole logging namespace alive
• somehow, the logging module keeps the whole pip package (!) and all of its sub-modules (!) alive until Python clears os.register_at_fork(): that happens VERY LATE, like the last function call before the last GC collection.
• Python clears os.register_at_fork() callbacks: the context manager finally: block is called as expected....
• Oops, os.remove is None at this point, the call fails.

Fixing importlib.resources is simple: keep a reference to os.remove() in importlib.resources implementation to make sure that we can remove the file very late during Python finalization.

This problem is tricky because:

• importlib.resources only creates a named temporary file if pip is imported from a ZIP file. When pip is installed on disk as a file, the bug is gone.
• the code is valid.
• certifi relies on the garbage collector to gently calls the context manager finalizer which removes the temporary file.
• Python doesn't log any error when a bug occurs very late during Python finalization.

I'm working on a fix for importlib.resources.