Update bundled pip to 20.2.1 and setuptools to 49.2.1

[zooba]

BPO	41490
Nosy	@pfmoore, @jaraco, @ncoghlan, @merwok, @ambv, @ericsnowcurrently, @zooba, @dstufft, @pradyunsg, @JulienPalard, @pablogsal
PRs	bpo-41490: Update ensurepip to install pip 20.2.1 and setuptools 49.2.1 #21748 [3.9] bpo-41490: Update ensurepip to install pip 20.2.1 and setuptools 49.2.1 (GH-21748) #21774 [3.8] bpo-41490: Update ensurepip to install pip 20.2.1 and setuptools 49.2.1 (GH-21748) #21775 bpo-41490: Bump vendored pip to version 20.2.3 #22527 [3.9] bpo-41490: Bump vendored pip to version 20.2.3 (GH-22527). #22544 [3.8] bpo-41490: Bump vendored pip to version 20.2.3 (GH-22527). #22545 bpo-41490: Update ensurepip to install pip 20.2.3 and setuptools 49.2.1 #22779 bpo-41490: path and contents to aggressively close handles #22915

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2021-02-02.16:27:56.592>
created_at = <Date 2020-08-05.21:26:19.394>
labels = ['library', '3.10', 'release-blocker']
title = 'Update bundled pip to 20.2.1 and setuptools to 49.2.1'
updated_at = <Date 2021-02-02.16:27:56.591>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2021-02-02.16:27:56.591>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2021-02-02.16:27:56.592>
closer = 'steve.dower'
components = ['Distutils']
creation = <Date 2020-08-05.21:26:19.394>
creator = 'steve.dower'
dependencies = []
files = []
hgrepos = []
issue_num = 41490
keywords = ['patch']
message_count = 34.0
messages = ['374901', '374909', '375013', '375015', '375020', '375022', '375023', '375024', '376720', '376729', '376731', '376752', '377811', '377812', '377879', '377961', '377962', '377963', '377964', '378008', '378013', '378053', '378099', '378314', '379017', '379595', '384930', '385059', '386106', '386113', '386143', '386144', '386146', '386152']
nosy_count = 13.0
nosy_names = ['paul.moore', 'jaraco', 'ncoghlan', 'eric.araujo', 'lukasz.langa', 'eric.snow', 'steve.dower', 'dstufft', 'pradyunsg', 'Marcus.Smith', 'mdk', 'lkollar', 'pablogsal']
pr_nums = ['21748', '21774', '21775', '22527', '22544', '22545', '22779', '22915']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue41490'
versions = ['Python 3.10']

[zooba]

I'm doing the PR now, based on the latest versions available today:

https://pypi.org/project/pip/20.2.1/
https://pypi.org/project/setuptools/49.2.1/

If you're a maintainer and there's a reason to not update to to the latest, please let me know asap. All of our subsequent releases should be RC's, so I assume we won't take any changes bigger than targeted fixes before the next full releases.

[zooba]

Test failure on Windows. I'll take a look tomorrow.

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\test_venv.py", line 476, in do_test_with_pip
    self.run_with_capture(venv.create, self.env_dir,
  File "D:\a\cpython\cpython\lib\test\test_venv.py", line 76, in run_with_capture
    func(*args, **kwargs)
subprocess.CalledProcessError: Command '['C:\\Users\\runneradmin\\AppData\\Local\\Temp\\tmp3cz40z50\\Scripts\\python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\test_venv.py", line 536, in test_with_pip
    self.do_test_with_pip(False)
  File "D:\a\cpython\cpython\lib\test\test_venv.py", line 484, in do_test_with_pip
    self.fail(msg.format(exc, details))
AssertionError: Command '['C:\\Users\\runneradmin\\AppData\\Local\\Temp\\tmp3cz40z50\\Scripts\\python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

**Subprocess Output**
Looking in links: c:\Users\RUNNER~1\AppData\Local\Temp\tmped5jdzqn

Processing c:\users\runneradmin\appdata\local\temp\tmped5jdzqn\setuptools-49.2.1-py3-none-any.whl

Processing c:\users\runneradmin\appdata\local\temp\tmped5jdzqn\pip-20.2.1-py2.py3-none-any.whl

Installing collected packages: setuptools, pip

Successfully installed pip-20.2.1 setuptools-49.2.1

Traceback (most recent call last):

File "D:\a\cpython\cpython\lib\shutil.py", line 613, in _rmtree_unsafe

    os.unlink(fullname)

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmped5jdzqn\\pip-20.2.1-py2.py3-none-any.whl'

During handling of the above exception, another exception occurred:



Traceback (most recent call last):

File "D:\a\cpython\cpython\lib\tempfile.py", line 802, in onerror

    _os.unlink(path)

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmped5jdzqn\\pip-20.2.1-py2.py3-none-any.whl'

During handling of the above exception, another exception occurred:



Traceback (most recent call last):

File "D:\a\cpython\cpython\lib\runpy.py", line 197, in _run_module_as_main

    return _run_code(code, main_globals, None,

File "D:\a\cpython\cpython\lib\runpy.py", line 87, in _run_code

    exec(code, run_globals)

File "D:\a\cpython\cpython\lib\ensurepip\main.py", line 5, in <module>

    sys.exit(ensurepip._main())

File "D:\a\cpython\cpython\lib\ensurepip\init.py", line 213, in _main

    return _bootstrap(

File "D:\a\cpython\cpython\lib\ensurepip\init.py", line 132, in _bootstrap

return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)

File "D:\a\cpython\cpython\lib\tempfile.py", line 827, in __exit__

    self.cleanup()

File "D:\a\cpython\cpython\lib\tempfile.py", line 831, in cleanup

    self._rmtree(self.name)

File "D:\a\cpython\cpython\lib\tempfile.py", line 813, in _rmtree

    _shutil.rmtree(name, onerror=onerror)

File "D:\a\cpython\cpython\lib\shutil.py", line 737, in rmtree

    return _rmtree_unsafe(path, onerror)

File "D:\a\cpython\cpython\lib\shutil.py", line 615, in _rmtree_unsafe

    onerror(os.unlink, fullname, sys.exc_info())

File "D:\a\cpython\cpython\lib\tempfile.py", line 805, in onerror

    cls._rmtree(path)

File "D:\a\cpython\cpython\lib\tempfile.py", line 813, in _rmtree

    _shutil.rmtree(name, onerror=onerror)

File "D:\a\cpython\cpython\lib\shutil.py", line 737, in rmtree

    return _rmtree_unsafe(path, onerror)

File "D:\a\cpython\cpython\lib\shutil.py", line 596, in _rmtree_unsafe

    onerror(os.scandir, path, sys.exc_info())

File "D:\a\cpython\cpython\lib\shutil.py", line 593, in _rmtree_unsafe

    with os.scandir(path) as scandir_it:

NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmped5jdzqn\\pip-20.2.1-py2.py3-none-any.whl'

----------------------------------------------------------------------

[zooba]

The issue above doesn't appear to repro on 3.9, so I guess master has started leaking a file handle, presumably in zipimport.

I'll see what I can track down, but can't be sure I'll have enough time to get it done for RC, so if anyone else wants to help out feel free.

[zooba]

Okay, I've tracked it down to the new importlib.readers.ZipReader class keeping the zip file open, presumably until it gets GC'd. This is used by certifi to extract the CA certs from the whl when ensurepip is doing the self-install from the mounted wheel.

Jason is already on this bug, which is convenient :)

I haven't yet figured out whether there's a convenient way for the reader to not keep the ZIP open for as long as it exists, but I think that's going to be the safest fix.

We should definitely fix this one ourselves without forcing users to make changes to accommodate. As I mentioned above, it's only in 3.10 right now, but it's blocking updated pip and setuptools versions downlevel.

[zooba]

Added some test cases to the PR that directly trigger the issue, specifically this one:

    def test_entered_path_does_not_keep_open(self):
        # This is what certifi does on import to make its bundle
        # available for the process duration.
        c = resources.path('ziptestdata', 'binary.file').__enter__()
        self.zip_path.unlink()

All the tests I added pass on 3.9 (with minor tweaks for moved test utils).

To unblock the upcoming releases, I'm going to do the backports first and leave this as a release blocker for 3.10.