Decouple ensurepip from pip's internals using runpy

[pradyunsg]

BPO	38662
Nosy	@brettcannon, @pfmoore, @ncoghlan, @vstinner, @ned-deily, @dstufft, @pradyunsg, @hroncok, @miss-islington
PRs	bpo-38662: Invoke pip via runpy, in ensurepip #17029 bpo-38662: Invoke pip via runpy, in ensurepip #18901 [3.8] bpo-38662: ensurepip invokes pip via runpy (GH-18901) #18906 [3.7] bpo-38662: ensurepip invokes pip via runpy (GH-18901) #18907

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-10.21:40:24.083>
created_at = <Date 2019-11-01.17:24:24.199>
labels = ['3.8', '3.7', 'library', '3.9']
title = "Decouple ensurepip from pip's internals using runpy"
updated_at = <Date 2020-03-10.21:40:24.083>
user = 'https://github.com/pradyunsg'

bugs.python.org fields:

activity = <Date 2020-03-10.21:40:24.083>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-10.21:40:24.083>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2019-11-01.17:24:24.199>
creator = 'pradyunsg'
dependencies = []
files = []
hgrepos = []
issue_num = 38662
keywords = ['patch']
message_count = 11.0
messages = ['355826', '355828', '356021', '356031', '363846', '363847', '363848', '363849', '363851', '363852', '363854']
nosy_count = 10.0
nosy_names = ['brett.cannon', 'paul.moore', 'ncoghlan', 'vstinner', 'ned.deily', 'dstufft', 'pradyunsg', 'Marcus.Smith', 'hroncok', 'miss-islington']
pr_nums = ['17029', '18901', '18906', '18907']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38662'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[pradyunsg]

Inspired by #16782 (comment)

This changes ensurepip's current assumption that pip would expose a "main" function, which can be invoked to run pip.

I've selected all actively supported Python versions for this ticket, since this patch should be backported to all Python versions, to make it robust to internal changes within pip.

[pradyunsg]

Note that the patch uses the fact that:

def foo():
    try:
        raise Exception("returned")
    except Exception as e:
        print("except")
        return e.args[0]
    finally:
        print("finally")


print(foo())

will print:

except
finally
returned

[ned-deily]

Can you explain how this would be rolled out? Does this mean as of a planned release of pip the current ensurepip would break? If so, when? Is 2.7 still supported with that version of pip and, if so, have you tested the PR with it?

[pradyunsg]

On Tue, 5 Nov 2019 at 4:30 PM, Ned Deily <report@bugs.python.org> wrote:

Ned Deily <nad@python.org> added the comment:

Can you explain how this would be rolled out?

This, as in? If you're referring to this specific change, I don't think
there's anything special needed for this change (it's backwards compatible
with past pip releases, but that doesn't matter).

ensurepip only installs the bundled wheel it already has, into the
environment. Thus, existing ensurepip releases won't be broken.

However, when we bump up to pip 19.3 [1], we'd need to be updating the
invocation since it's using a pip internal and pip's internals can change
without notice even in a patch version.

This change ensures that ensurepip is robust to changes in the location of
pip's main function, thus making those updates as simple as a bundled wheel
change.

[1]: IIUC we try to keep CPython's bundled wheel in sync with latest)

Does this mean as of a planned release of pip the current ensurepip would

break?

Yes. pip 19.3 made a change and as can be seen in the PR for that (sorry,
it's a GitHub PR and my internet is ridiculously slow right now).

If so, when?

That's already been released. 🙃

Is 2.7 still supported with that version of pip and, if so, have you tested

the PR with it?

Yes, pip will support 2.7 at least until CPython does (and likely longer).

----------
nosy: +ned.deily

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38662\>

---

[vstinner]

New changeset 88f82b2 by Miro Hrončok in branch 'master':
bpo-38662: ensurepip invokes pip via runpy (GH-18901)
88f82b2

[vstinner]

While it's not a major issue in CPython, I can say that being tightly coupled to pip internals is a major in Fedora which updates pip frequently and tries to support multiple pip versions.

[vstinner]

Miro Hrončok: "I wasn't even hoping for a backport. Thanks!" ( #18901 (comment) )

pip wheel files are still updated in stable 3.7 and 3.8 branches. Example in August 2018 in 3.7:

• commit c1c1a33
• bpo-37664

[vstinner]

Yes, pip will support 2.7 at least until CPython does (and likely longer).

2.7 is no longer supported in CPython. So I will not backport the fix to 2.7. And I removed the 2.7 from versions of this issue.

[miss-islington]

New changeset 8d5c958 by Miss Islington (bot) in branch '3.7':
bpo-38662: ensurepip invokes pip via runpy (GH-18901)
8d5c958

[miss-islington]

New changeset 212acf5 by Miss Islington (bot) in branch '3.8':
bpo-38662: ensurepip invokes pip via runpy (GH-18901)
212acf5

[vstinner]

Thanks Pradyun Gedam and Miro Hrončok for the fix.