Move path-based bootstrap code to a separate frozen file.

[ericsnowcurrently]

BPO	23911
Nosy	@brettcannon, @ncoghlan, @larryhastings, @ned-deily, @ericsnowcurrently, @raulcd
Files	path-based-importlib.diff path-based-importlib.diff path-based-importlib.diff path-based-importlib.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2015-05-06.03:31:10.031>
created_at = <Date 2015-04-11.01:29:38.032>
labels = ['interpreter-core', 'type-feature']
title = 'Move path-based bootstrap code to a separate frozen file.'
updated_at = <Date 2015-05-13.05:35:07.502>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2015-05-13.05:35:07.502>
actor = 'python-dev'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2015-05-06.03:31:10.031>
closer = 'eric.snow'
components = ['Interpreter Core']
creation = <Date 2015-04-11.01:29:38.032>
creator = 'eric.snow'
dependencies = []
files = ['38893', '39138', '39272', '39275']
hgrepos = []
issue_num = 23911
keywords = ['patch']
message_count = 18.0
messages = ['240457', '241592', '241593', '242001', '242160', '242433', '242434', '242435', '242437', '242440', '242441', '242444', '242557', '242565', '242635', '242639', '242640', '243040']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'ncoghlan', 'larry', 'ned.deily', 'python-dev', 'eric.snow', 'raulcd']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23911'
versions = ['Python 3.5']

[ericsnowcurrently]

The bootstrap code has a clear division between the core import functionality and the path-based import machinery. The attached patch makes that division explicit by moving the latter into its own module. The module is also frozen, necessarily. In addition to clearly distinguishing the two parts, this division will help with some later work that I'd like to do later with an encapsulated import system abstraction.

The patch uses the name "pathy" throughout, which I'll change to something more descriptive later. I'll also add the news entry then.

[ericsnowcurrently]

Here's an updated patch, with the PEP-489 changes merged in. Only one test isn't passing and it is due to something in the pip that is bundled into ensurepip. I'll work on fixing that when I have some time. I'm sure there's documentation near the bundle that explains how to update it.

[ericsnowcurrently]

s/PEP 489/PEP 488/

[ncoghlan]

Nice, this ties directly into one of the thornier problems in the PEP-432 bootstrapping work, which needs to set up the core import system in Py_BeginInitialization, but delay setting up the rest of it until Py_EndInitialization.

Your patch achieves this by moving everything that belongs in the latter part of the operation to the "import _frozen_path_importer" step.

As far as naming goes, I'd suggest referring to the two import subsystems as "internal imports" and "external imports". The key aspect of builtin and frozen modules is that they're an inherent part of the interpreter itself - if you have an interpreter, you have all of its buitin and frozen modules natively available. By contrast, setting up the external import machinery requires that the interpreter first be configured to appropriately communicate with the host system.

[ericsnowcurrently]

Glad to hear the patch is conceptually consistent with other components. :) And the "internal"/"external" suggestion is a good one. I'll update the patch when I have a minute.

[ericsnowcurrently]

Here's an updated patch with "_pathy.py" changed to "_bootstrap_external.py" (and similar changes with freezing). The patch does not include fixing the venv test (i.e. the bundled pip).

Also, I'll be adding a note to NEWS.

[ericsnowcurrently]

As I mentioned, I'm pretty sure that the failing venv test is due to the bundled pip. Here's the test output:

test test_venv failed -- Traceback (most recent call last):
  File "/home/esnow/projects/cpython/Lib/test/test_venv.py", line 356, in test_with_pip
    with_pip=True)
subprocess.CalledProcessError: Command '['/tmp/tmphxh1zztf/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/esnow/projects/cpython/Lib/test/test_venv.py", line 362, in test_with_pip
    self.fail(msg.format(exc, details))
AssertionError: Command '['/tmp/tmphxh1zztf/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

**Subprocess Output**
Traceback (most recent call last):
  File "/home/esnow/projects/cpython/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/esnow/projects/cpython/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/esnow/projects/cpython/Lib/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/home/esnow/projects/cpython/Lib/ensurepip/__init__.py", line 209, in _main
    default_pip=args.default_pip,
  File "/home/esnow/projects/cpython/Lib/ensurepip/__init__.py", line 116, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/home/esnow/projects/cpython/Lib/ensurepip/__init__.py", line 40, in _run_pip
    import pip
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load
  File "<frozen importlib._bootstrap>", line 947, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 625, in _load_backward_compatible
  File "/tmp/tmp57j9vkrt/pip-6.1.1-py2.py3-none-any.whl/pip/__init__.py", line 13, in <module>
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load
  File "<frozen importlib._bootstrap>", line 947, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 625, in _load_backward_compatible
  File "/tmp/tmp57j9vkrt/pip-6.1.1-py2.py3-none-any.whl/pip/utils/__init__.py", line 22, in <module>
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load
  File "<frozen importlib._bootstrap>", line 947, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 625, in _load_backward_compatible
  File "/tmp/tmp57j9vkrt/pip-6.1.1-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py", line 1712, in <module>
AttributeError: module 'importlib._bootstrap' has no attribute 'SourceFileLoader'

[ericsnowcurrently]

Looks like setuptool's pkg_resources is directly importing importlib._bootstrap. I've filed a bug: https://bitbucket.org/pypa/setuptools/issue/378. In the meantime, what are our options for getting that test passing?

[ericsnowcurrently]

Here's the correct patch.

[ncoghlan]

As a compatibility hack for setuptools versions with the issue, I'd suggest
making importlib._bootstrap.setup alias SourceFileLoader back into
importlib._bootstrap, with an explanation and link to back to this issue in
a comment.

[ericsnowcurrently]

Gah. I had tried exactly that but did it in the wrong spot. Here's an updated patch which fixes the test.

[python-dev]

New changeset 02e3bf65b2f8 by Eric Snow in branch 'default':
Issue bpo-23911: Move path-based bootstrap code to a separate frozen module.
https://hg.python.org/cpython/rev/02e3bf65b2f8