Recent importlib change breaks most recent certifi == 2020.4.5.2

[ned-deily]

BPO	40924
Nosy	@jaraco, @ned-deily, @ambv, @zooba, @dstufft
PRs	[3.9] bpo-40924: Revert "bpo-39791 native hooks for importlib.resources.files (GH-20576)" #20760 [3.9] bpo-40924: Remove protocol for supplying Traversable objects from loaders #20820 bpo-40924: Ensure importlib.resources.path returns an extant path #20857

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/jaraco'
closed_at = <Date 2020-06-29.21:10:55.215>
created_at = <Date 2020-06-09.05:59:48.615>
labels = ['type-bug', 'library', '3.9', '3.10', 'release-blocker']
title = 'Recent importlib change breaks most recent certifi == 2020.4.5.2'
updated_at = <Date 2020-06-29.21:10:55.214>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2020-06-29.21:10:55.214>
actor = 'lukasz.langa'
assignee = 'jaraco'
closed = True
closed_date = <Date 2020-06-29.21:10:55.215>
closer = 'lukasz.langa'
components = ['Library (Lib)']
creation = <Date 2020-06-09.05:59:48.615>
creator = 'ned.deily'
dependencies = []
files = []
hgrepos = []
issue_num = 40924
keywords = ['patch']
message_count = 13.0
messages = ['371073', '371084', '371094', '371095', '371132', '371138', '371143', '371144', '371252', '371256', '371502', '372631', '372634']
nosy_count = 5.0
nosy_names = ['jaraco', 'ned.deily', 'lukasz.langa', 'steve.dower', 'dstufft']
pr_nums = ['20760', '20820', '20857']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40924'
versions = ['Python 3.9', 'Python 3.10']

[ned-deily]

The very recent latest commits for bpo-39791, "New files() api from importlib_resources", have broken the popular certifi package, a package which provides a basic set of Root Certificates for TLS secure network connection verification. Among other users of it, the python.org macOS installers encourage its users to run a provided script to install certifi. Alas, we discovered just after v3.9.2b2 was tagged that that script is broken because certifi.where() is returning a bogus path toe the pem file, what appears to be a path in a deleted temp directory.

The culprit commits are 843c277 (master) and 9cf1be4 (3.9). This is now a critical problem because in its most recent release, certifi 2020.4.5.2, certifi was changed to try to use importlib.resources.path(), if available, to find the path to the installed cacert.pem file. (The previous release, 2020.4.5.1, appears not to use path() and so is not affected by this bug.)

certifi/python-certifi@3fc8fec

https://pypi.org/project/certifi/2020.4.5.2/

Without trying to debug the bug, I was able to bisect the branch and then reduce the problem seen in macOS installer testing to a fairly simple reproducible test case. The problem was reproduced on both Linux and macOS systems.

The test case:

# in a current cpython git repo, checkout the commit before the failing one
git checkout 843c277^
git log HEAD^..HEAD
git clean -fdxq
./configure --prefix=$PWD/root -q
make -j4
./python -m ensurepip
./python -m pip install --upgrade pip # not necessary to reproduce
./python -m pip install --force --no-binary certifi certifi==2020.4.5.2
./python -c 'import certifi;print(certifi.where())'

The output tail should be something like:
[...]
Successfully installed certifi-2020.4.5.2
/home/nad/cpython/root/lib/python3.10/site-packages/certifi/cacert.pem

Now checkout the failing commit and repeat all the other steps:

git checkout 843c277
git log HEAD^..HEAD
[...]

The output tail is now incorrect:
[...]
Successfully installed certifi-2020.4.5.2
/tmp/tmpqfjnbj5bcacert.pem

The cacert.pem is installed to the expected (same) location in either case; its just the output from importlib.resources.path that is incorrect:

./python
Python 3.10.0a0 (remotes/upstream/master:0a40849eb9, Jun  9 2020, 00:35:07)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.resources, os, os.path, certifi
>>> certifi.__file__
'/home/nad/cpython/root/lib/python3.10/site-packages/certifi/__init__.py'
>>> os.listdir(os.path.dirname(certifi.__file__))
['__pycache__', 'cacert.pem', '__init__.py', 'core.py', '__main__.py']
>>> with importlib.resources.path('certifi', 'cacert.pem') as f: print(f)
...
/tmp/tmpxsrjxb8lcacert.pem
>>> with importlib.resources.path('certifi', 'core.py') as f: print(f)
...
/tmp/tmpjq8h3si5core.py

No test suite failures were noted. Perhaps there should be a test case for this?

Presumably any other downstream users of importlib.resources.path() are affected.

Łukasz as 3.9 release manager is aware there is an issue but was awaiting the tracking down of the problem before making a decision about what to do for 3.9.0b2.

cc: Donald as the author of the certifi change.

[zooba]

That was a *big* patch to importlib.resources... And newly defined and standardised protocols? That seems like a lot to bring in without having gone through a PEP anyway, despite the minor bug that has drawn attention to it.

I don't think backports should be treated as incubators for standard modules, getting new features that then flow directly into the stdlib. It's fine to have a fork for incubation, but people need to rely on consistency in stdlib and backports.

All that to say, I think we should revert the importlib changes *anyway*, and also because there's something wrong with them right now.

[jaraco]

I understand the issue here and can supply more details soon (no later than this weekend).