Create a lazy import loader mixin

[brettcannon]

BPO	17621
Nosy	@Yhg1s, @warsaw, @brettcannon, @ericvsmith, @jwilk, @ericsnowcurrently
Files	test_lazy_loader.py: tests lazy_mixin.py: Solution using a mixin lazy_proxy.py: Solution using a proxy lazy_test.py lazy_loader.diff lazy_loader.diff lazy_loader.diff lazy_loader.diff: Detect sys.modules swap

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2014-04-04.17:55:06.543>
created_at = <Date 2013-04-02.18:00:47.615>
labels = ['type-feature', 'library']
title = 'Create a lazy import loader mixin'
updated_at = <Date 2014-04-04.22:10:33.832>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2014-04-04.22:10:33.832>
actor = 'eric.snow'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2014-04-04.17:55:06.543>
closer = 'brett.cannon'
components = ['Library (Lib)']
creation = <Date 2013-04-02.18:00:47.615>
creator = 'brett.cannon'
dependencies = []
files = ['30670', '30671', '30672', '33136', '33947', '34556', '34576', '34591']
hgrepos = []
issue_num = 17621
keywords = ['patch']
message_count = 26.0
messages = ['185852', '191551', '191616', '191623', '191651', '191660', '191667', '191675', '191772', '191776', '191779', '201930', '202445', '206205', '206211', '210410', '214410', '214508', '214536', '214627', '214921', '214922', '214954', '215542', '215543', '215572']
nosy_count = 10.0
nosy_names = ['twouters', 'barry', 'brett.cannon', 'eric.smith', 'jwilk', 'durin42', 'Arfrever', 'python-dev', 'sbt', 'eric.snow']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17621'
versions = ['Python 3.5']

[brettcannon]

People keep asking and I keep promising to get a lazy loader into Python 3.4. This issue is for me to track what I have done towards meeting that promise.

To start, I have something working at https://code.google.com/p/importers/, but I need to make sure that the code copies any newly assigned objects post-import but before completing an attribute read::

  import lazy_mod
  lazy_mod.attr = True  # Does not have to trigger import, although there is nothing wrong if it did.
  lazy_mod.func()  # Since it might depend on 'attr', don't return attr until after import and 'attr' with a value of True has been set.

Also need to see if anything can be done about isinstance/issubclass checks as super() is used for assigning to __loader__ and thus might break checks for ABCs. Maybe create a class from scratch w/o the mixin somehow (which I don't see from looking at http://docs.python.org/3.4/library/types.html#module-types w/o re-initializing everything)? Somehow get __subclasscheck__() on the super() class? Some other crazy solution that avoids having to call __init__() a second time?

[brettcannon]

One problem with the importers code is it doesn't properly clean up after the module if the load failed and it wasn't a reload. Should store an attribute on the module signaling whether it was a reload or not to know whether an exception raised during loading should lead to the module being removed from sys.modules.

[brettcannon]

While seing if it was worth making isinstance work with super(), I came up with this at Antoine's suggestion of using a proxy instead of a mixin:

class LazyProxy(importlib.abc.Loader):

  def __init__(self, loader):
    self.loader = loader

  def __call__(self, *args, **kwargs):
    self.args = args
    self.kwargs = kwargs
    return self

  def load_module(self, fullname):
    # XXX ignoring sys.modules details, e.g. if module already existed.
    lazy_module = LazyModule(fullname, proxy=self, name=fullname)
    sys.modules[fullname] = lazy_module
    return lazy_module

class LazyModule(types.ModuleType):

    def __init__(*args, proxy, name, **kwargs):
      self.__proxy = proxy
      self.__name = name
      super().__init__(*args, **kwargs)

    def __getattribute__(self, attr):
      self.__class__ = Module
      state = self.__dict__.copy()
      loader = self.__proxy.loader(*self.proxy.args, **self.proxy.kwargs)
      # XXX ignoring sys.modules details, e.g. removing module on load
failure.
      loader.load_module(self.__name)
      self.__dict__.update(state)
      return getattr(module, attr)

[brettcannon]

I think the first step for this bug, now that I have two possible approaches, is to write the unit tests. That way both approaches can equally be validated based on their merits of complexity, etc. while verifying the work properly.

[sbt]

Apologies for being dense, but how would you actually use such a loader?

Would you need to install something in sys.meta_path/sys.path_hooks? Would it make all imports lazy or only imports of specified modules?

[brettcannon]

So the approaches I have been using make a loader lazy, so what you have to change in terms of sys.meta_path, sys.path_hooks, etc. would very from loader to loader.

I have realized one tricky thing with all of this is that importlib itself inspects modules post-import to verify that __loader__ and __package__ have been set. That typically triggers an immediate load and so might need to be special-cased.

[brettcannon]

I have attached the test suite and two versions: one using a mixin and one using a proxy. Neither solve the issue of import touching the module in any way to check __loader__ and __package__. The mixin version is also failing one test which could quite possibly be a pain to fix and so it might cause me to prefer the proxy solution.

[sbt]

Shouldn't the import lock be held to make it threadsafe?

[brettcannon]

Import manages the lock, not loaders.

[sbt]

I was thinking about the line

      self.__dict__.update(state)

overwriting new data with stale data.

[brettcannon]

It still falls under the purview of import to manage that lock. It's just the design of the API from PEP-302. Otherwise it's just like any other reload.

[brettcannon]

Won't work until PEP-451 code lands and lets us drop __package__/loader checking/setting post-load.