latest setuptools breaks virtualenvs due to unbundling dependencies

[stratakis]

BPO	29523
Nosy	@jaraco, @merwok, @dstufft, @stratakis
PRs	bpo-29523: Update setuptools to 34.2.0 and bundle its dependencies. #67
Files	bundle-setuptools-dependencies.patch bundle-setuptools-dependencies2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-06-21.11:05:18.205>
created_at = <Date 2017-02-10.10:14:21.568>
labels = ['3.7', 'library']
title = 'latest setuptools breaks virtualenvs due to unbundling dependencies'
updated_at = <Date 2017-06-21.11:05:18.204>
user = 'https://github.com/stratakis'

bugs.python.org fields:

activity = <Date 2017-06-21.11:05:18.204>
actor = 'cstratak'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-21.11:05:18.205>
closer = 'cstratak'
components = ['Distutils', 'Library (Lib)']
creation = <Date 2017-02-10.10:14:21.568>
creator = 'cstratak'
dependencies = []
files = ['46621', '46627']
hgrepos = []
issue_num = 29523
keywords = ['patch']
message_count = 10.0
messages = ['287504', '287535', '287536', '287538', '287543', '287546', '287552', '287567', '287692', '296545']
nosy_count = 4.0
nosy_names = ['jaraco', 'eric.araujo', 'dstufft', 'cstratak']
pr_nums = ['67']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue29523'
versions = ['Python 3.6', 'Python 3.7']

[stratakis]

The latest versions of setuptools stopped bundling its dependencies and instead starting requiring them [0]. This seems to break virtualenvs as those dependencies are not bundled with python.

In order to reproduce it, replace the setuptools-28.8.0-py2.py3-none-any.whl with setuptools-34.1.1-py2.py3-none-any.whl at Lib/ensurepip/_bundled/ and change

_SETUPTOOLS_VERSION = "28.8.0"

to

_SETUPTOOLS_VERSION = "34.1.1"

Then configure, make and make install:

This message will appear first:
Could not find a version that satisfies the requirement packaging>=16.8 (from setuptools) (from versions: )
No matching distribution found for packaging>=16.8 (from setuptools)

Which is one of the new dependencies.

Then:
./python3.7 -m test test_venv -v

Where the EnsurePipTest will fail due to this assertion [1] at the do_test_with_pip test case with "No module named pip\n' != ''".

One way to fix this would be to bundle the other wheels with python, not sure if that would be the ideal approach though.

Attaching a patch for Lib/ensurepip/init.py. With this patch and adding the other wheels [2][3][4][5] at the _bundled directory fixes the issue.

[0] https://github.com/pypa/setuptools/blob/master/CHANGES.rst#v3400
[1] https://github.com/python/cpython/blob/master/Lib/test/test_venv.py#L384
[2] https://pypi.python.org/pypi/six/1.10.0
[3] https://pypi.python.org/pypi/appdirs/1.4.0
[4] https://pypi.python.org/pypi/packaging/16.8
[5] https://pypi.python.org/pypi/pyparsing/2.1.10

[jaraco]

This approach, bundling the necessary dependencies to install setuptools, seems like exactly the right thing to do, assuming that ensurepip should install setuptools at all, and I don't want to challenge that assumption here.

Looking at the patch and ensurepip, I wonder if the patch is necessary at all. It looks to me like the find_links declaration should be sufficient to allow a pip install setuptools to succeed and install its dependencies as long as they're bundled. @cstratak, can you test a less aggressive approach that merely bumps the dependency on setuptools and bundles the dependencies?

[jaraco]

For convenience, here's a link to the rendered changelog in the docs, which provides hyperlinks to the relevant issues [0].

[0] https://setuptools.readthedocs.io/en/latest/history.html#v34-0-0

[dstufft]

I haven't updated the bundled copy of ensurepip yet because of this. Most likely the way this will get resolved is that the PEP-518 PR will land in pip and we will no longer require setuptools to be pre-installed at all and ensurepip will go to only bundling pip itself.

[stratakis]

@jason.coombs

Already tried to just bump the setuptools version and bundle the other wheels but the result is still the same

[jaraco]

Aah, yes. I see now. Ensure_pip uses _PROJECTS for three purposes: for defining additional_projects, extracting those packages to the tmpdir, and for declaring the installation of those projects. Only that last declaration is unnecessary.

This patch looks good to me.

[stratakis]

Will send a pull request which includes the extra wheels.

[stratakis]

Second version of the patch:

Adjusted the test_ensurepip test cases to account for the new modules

[stratakis]

Pull Request has been sent: #67

[stratakis]

Closing this per the discussion at the PR.