Upgrade to pip 10 for Python 3.7

[ncoghlan]

BPO	33131
Nosy	@pfmoore, @ncoghlan, @ned-deily, @serhiy-storchaka, @dstufft, @miss-islington
PRs	bpo-33131: Upgrade ensurepip to pip 10.0.1 #6546 [3.7] bpo-33131: Upgrade ensurepip to bundle pip 10.0.1 (GH-6546) #6550 [2.7] bpo-33131: Upgrade ensurepip to bundle pip 10.0.1 (GH-6546) #6551 [3.6] bpo-33131: Upgrade ensurepip to bundle pip 10.0.1 (GH-6546) #6553

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pfmoore'
closed_at = <Date 2018-04-21.04:58:22.940>
created_at = <Date 2018-03-24.07:27:40.947>
labels = ['3.8', 'deferred-blocker', 'type-feature', '3.7']
title = 'Upgrade to pip 10 for Python 3.7'
updated_at = <Date 2018-04-21.04:58:22.939>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2018-04-21.04:58:22.939>
actor = 'ncoghlan'
assignee = 'paul.moore'
closed = True
closed_date = <Date 2018-04-21.04:58:22.940>
closer = 'ncoghlan'
components = []
creation = <Date 2018-03-24.07:27:40.947>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 33131
keywords = ['patch']
message_count = 16.0
messages = ['314360', '314361', '314362', '314577', '314622', '314629', '314634', '315314', '315315', '315316', '315321', '315532', '315533', '315537', '315538', '315541']
nosy_count = 7.0
nosy_names = ['paul.moore', 'ncoghlan', 'ned.deily', 'serhiy.storchaka', 'dstufft', 'Marcus.Smith', 'miss-islington']
pr_nums = ['6546', '6550', '6551', '6553']
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue33131'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[ncoghlan]

Paul brought up recently [1] that with pip 10.0.0 due for release next month [2], we'd really prefer to ship that in Python 3.7.0 (such that 3.7 launches with PEP 518/517 pyproject.toml support), rather than shipping with 9.0.x and then upgrading to 10.0.0 in Python 3.7.1.

The timing is such that 10.0.0 won't quite be ready for 3.7.0b3, but it should be released before 3.7.0b4 at the end of April.

[1] pypa/packaging-problems#127 (comment)
[2] https://mail.python.org/pipermail/distutils-sig/2018-March/032047.html

[serhiy-storchaka]

Currently the repository contains blobs for pip and setuptools (2 MB total) which are updated with every release of pip and setuptools. This increases the size of the repository by 2 MB several times per year. Wouldn't be better to put them in a separate repository like Tcl/Tk?

[ncoghlan]

Possibly - we implemented ensurepip the way we did just because it was the easiest option, not because we closely considered all the available approaches.

That would be a separate process improvement issue, though :)

[ned-deily]

BTW, when updating the bundled wheels, please remember to use an issue (like this one) and include a NEWS entry using blurb; in other words, don't use the "skip news" label on PRs. That way the change will be documented in the Misc/NEWS and docs changelog.html for the release(s). See, for example, bpo-33163, which added NEWS entries after the fact. Thanks!

[dstufft]

Oh, I had never written NEWS entries for upgrading pip/setuptools, with the idea that the specific version being installed is considered an implementation detail. However if folks want it, I can.

[ncoghlan]

It's a useful data point for folks trying to diagnose post-maintenance-update failures when we upgrade pip on their behalf, rather than them explicitly doing it themselves.

[ned-deily]

The changelog for a release is our primary notification to users about what has changed in a Python release. With 3.6.5, we just had a case where it seemed important to update pip between the release candidate and the final, something we normally would not do. In the past, we could manually munge the changelog to include this info but it's much more difficult to do so now with the NEWS entries managed via blurb. In fact, IIRC, it was a design decision to require a b.p.o issue for any item meriting a NEWS item. So, it's a bit of extra work for you all but it does make life a lot easier for the release team as it's a lot easier to add the NEWS item at the time when the updates are checked in rather than having to go back and do it after the fact. Thanks!

https://docs.python.org/3.6/whatsnew/changelog.html#python-3-6-5-final

[pfmoore]

Pip 10 has now been released. I'll be getting a PR to put it into 3.7 in the next few days. One question on workflow - is the right approach to create a PR for master and request a backport to the 3.7 branch? Or is there something else I should do?

Also, can I just check the timing implications should we need to do a 10.0.1 followup release? Would we need to have that released in time for b4, or would it be acceptable to update to 10.0.1 between b4 and rc1? (I'm not trying to push for any particular decision here - we may not even need a 10.0.1 - just want to be sure of the rules).

[ncoghlan]

Merge to master and set the "needs backport to 3.7" label for the initial PR - the backport bot will then take care of creating the 3.7 PR. For the backport, all you then need to do is leave an Approve review, and the bot will automatically merge it once the CI completes.

I'll defer to Ned regarding post-b4 updates, but I'd expect them to be fine if needed - otherwise we wouldn't be able to respond to any pip bugs reported against b4.

[pfmoore]

Thanks Nick, that's what I thought for the workflow. And agreed with your understanding of the post-b4 question, but as you say Ned's the boss on that one :-)

[ned-deily]

The ensurepip blob is a bit of a special case regarding updates, that is, I think we can be a little more flexible about them since they are getting tested and a lot of exposure external to Python releases. Accepting a bugfix update for pip between 3.7.0b4 and 3.7.0rc1 should be OK although the closer it is to the b4 date, rather than the rc1 date, the happier I would be :)