importlib/__init__.py can not be loaded without __file__ - breaks cxFreeze

[JurkoGospodneti]

BPO	20884
Nosy	@warsaw, @brettcannon, @ncoghlan, @larryhastings, @ned-deily, @bitdancer, @ericsnowcurrently, @takluyver
Files	cx_Freeze.7z: cx_Freeze source code checkout for reproducing the issue frozen_file.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2014-03-21.15:03:33.794>
created_at = <Date 2014-03-10.16:13:50.128>
labels = ['interpreter-core', 'type-bug', 'library']
title = 'importlib/__init__.py can not be loaded without __file__ - breaks cxFreeze'
updated_at = <Date 2014-03-21.20:34:25.338>
user = 'https://bugs.python.org/JurkoGospodneti'

bugs.python.org fields:

activity = <Date 2014-03-21.20:34:25.338>
actor = 'Arfrever'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2014-03-21.15:03:33.794>
closer = 'brett.cannon'
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2014-03-10.16:13:50.128>
creator = 'Jurko.Gospodneti\xc4\x87'
dependencies = []
files = ['34353', '34416']
hgrepos = []
issue_num = 20884
keywords = ['patch', '3.4regression']
message_count = 32.0
messages = ['213047', '213048', '213049', '213051', '213085', '213088', '213089', '213137', '213138', '213142', '213230', '213255', '213258', '213260', '213263', '213265', '213280', '213289', '213291', '213300', '213306', '213342', '213357', '213393', '213550', '213551', '213556', '213717', '213718', '213720', '214374', '214375']
nosy_count = 11.0
nosy_names = ['barry', 'brett.cannon', 'ncoghlan', 'larry', 'ned.deily', 'Arfrever', 'r.david.murray', 'Jurko.Gospodneti\xc4\x87', 'python-dev', 'eric.snow', 'takluyver']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20884'
versions = ['Python 3.4', 'Python 3.5']

[JurkoGospodneti]

Python 3.4 introduced a change to Lib/importlib/init.py that added the following code to it:

else:
# importlib._bootstrap is the built-in import, ensure we don't create
# a second copy of the module.
_bootstrap.__name__ = 'importlib._bootstrap'
_bootstrap.__package__ = 'importlib'
_bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py')
sys.modules['importlib._bootstrap'] = _bootstrap

When attempting to use cxFreeze on a project, using Python 3.4. we ran into a problem with '__file__' identifier not being defined.

I believe this is a bug since the code expects the module loaded from importlib/init.py to always have a __file__ identifier set.

Python 3.4.0rc3 documentation explicitly states:

__file__ is optional. If set, this attribute’s value must be a
string. The import system may opt to leave __file__ unset if it
has no semantic meaning (e.g. a module loaded from a database).

and:

Ultimately, the loader is what makes use of __file__ and/or
__cached__.

A possible fix would be to skip setting the _bootstrap.__file__ attribute if the current module does not have a __file__ attribute set.

I hope this can be resolved before the 3.4 final release or it will not be possible to use cxFreeze with Python 3.4 without additional workarounds in cxFreeze.

Best regards,
Jurko Gospodnetić

[bitdancer]

So you aren't running into problems with bpo-20778? (That issue exists in 3.3 as well).

[brettcannon]

Not having __file__ defined is an odd issue as importlib.__init__ always comes from a file in CPython. I have no issue making the setting of the attribute conditional, though.

[JurkoGospodneti]

David: Nope, the issue does not exist with 3.3.

[ned-deily]

"I hope this can be resolved before the 3.4 final release or it will not be possible to use cxFreeze with Python 3.4 without additional workarounds in cxFreeze."

It's too late for this to go into 3.4.0 unless someone can make a really good case to the release manager that this is critical enough to be a "release blocker". My guess is that it isn't but I don't use cxFreeze. Anyone? If not, then one should ask if this should be a "release blocker" for 3.4.1. If so, the priority should be set to "deferred blocker".

[JurkoGospodneti]

I'll try to set up and post a reproducible use case tomorrow. Then you can decide. It could turn out that the use case we ran into has an easy workaround.

[ncoghlan]

It's at least a deferred blocker, and that's my instinct as to the right setting (especially since we're assuming that 3.4.1 will happen a month or so after the PyCon US sprints).

I'll dig into a bit more to see if I can provoke the misbehaviour (e.g. we may be avoiding it in the test suite by always unloading the frozen one before loading from source, so cx_freeze may be able to just skip loading the bootstrap module from source entirely in both 3.3 and 3.4 with no ill effects).

[ncoghlan]

Reading https://bitbucket.org/anthony_tuininga/cx_freeze/src/default/cx_Freeze/finder.py I have a suspicion that the import system reimplementation in there is going to need updates to handle PEP-451 (I suspect there are even latent defects in relation to Python 3.3 changes).

Setting this to pending for the moment - it needs someone that is more familiar than I am with cx-freeze internals to go over:

The import related changes in http://docs.python.org/3/whatsnew/3.3.html#porting-to-python-3-3
The import related changes in http://docs.python.org/dev/whatsnew/3.4.html#changes-in-the-python-api
The import system update in http://www.python.org/dev/peps/pep-0451/

In particular, bpo-18065 (path now being empty in frozen packages) seems like it might have the potential to cause trouble.

[ncoghlan]

Another relevant link: http://docs.python.org/3.4/reference/import.html

Just to clarify the reason for the pending status:

At the moment, I suspect this is a matter of either 3.4 revealing a latent defect in the cx-freeze import emulation (by being less tolerant of deviations from the language spec), or else that emulation needs to be updated to address one or more of the existing porting notes.

In the first case, we'll just need a new porting note, in the latter, no change at all.

However, an inadvertent backwards incompatible change as a result of the PEP-451 changes is also a possibility. cx-freeze digs much deeper into the import system than most tools, so it's entirely plausible that it might hit an edge case that isn't covered by our test suite.

Note that Python source and bytecode files, whether imported from the filesystem or from inside a zipfile *should* have __file__ set (as should extension modules). Frozen modules won't have it set, but that shouldn't apply in the case of importlib/init.py.

[JurkoGospodneti]

Here's a recipe I can use to reproduce the problem on my PC.

Environment:

• Windows 7 SP1 x64
• Python 3.4.0rc3
• cx_Freeze checkout from its current HEAD
  • repository: https://bitbucket.org/anthony_tuininga/cx_freeze
  • HEAD commit: 52b63b3296843cd612cfbe047a9f6529df4c0444
  • I'm attaching a compressed checkout.
• TortoiseHg 2.11 - just in case you're having problems using
  the cx_Freeze Hg sandbox I attached to this comment and want
  to know if it is Hg version related.
• Microsoft Visual Studio 2010 C/C++ compiler.
  • Exact cl.exe version information: "Microsoft (R) 32-bit
    C/C++ Optimizing Compiler Version 16.00.40219.01 for 80x86"

-- 1. prepare to build cx_Freeze --
To build the given cx_Freeze project using Python 3.4 the
following tweak must first be done in Python's
Lib\distutils\msvc9compiler.py module:

Around line 649 replace the line:
if mfinfo is not None:
with:
if mfinfo is not None and target_desc != "executable":

This is an unrelated issue that still needs to be researched.

-- 2. build & install cx_Freeze --
Standard Python package installation - go to your cx_Freeze
sandbox project folder and run "py34 setup.py install".
This should result in "cx_Freeze-4.3.2-py3.4-win-amd64.egg"
folder being added under your Python 3.4 installation's
site-packages folder.

-- 3. prepare a test project --
Create a new folder and add the following files to it:

hello_world.py:
print("Hello world")
import asyncio
import sys
print(sys.version_info)

setup.py:
from cx_Freeze import setup, Executable
setup(name="HelloWorld",
version="0.1",
description="HelloWorld",
executables=[Executable("hello_world.py", base="Console")])

Note that only the 'import asyncio' line is necessary to
reproduce the issue. The rest is just useful output for the
user/tester.

-- 4. prepare a 'frozen' test project executable --
Go to the test project folder and run the following command:

py34 setup.py build_exe

This should create a new executable under:

build\exe.win-amd64-3.4\hello_world.exe

-- 5. run the prepared executable --

You should get output looking something like:

Hello world
Traceback (most recent call last):
  File "C:\Program Files\Python\Python34rc3\lib\site-packages\cx_freeze-4.3.2-py3.4-win-amd64.egg\cx_Freeze\initscripts\Cons
    exec(code, m.__dict__)
  File "hello_world.py", line 8, in <module>
    import asyncio
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\asyncio\__init__.py", line 23, in <module>
    from .locks import *
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\asyncio\locks.py", line 9, in <module>
    from . import tasks
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2261, in _handle_fromlist
    _call_with_frames_removed(import_, from_name)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 321, in _call_with_frames_removed
    return f(*args, **kwds)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\asyncio\tasks.py", line 12, in <module>
    import inspect
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\inspect.py", line 35, in <module>
    import importlib.machinery
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2189, in _find_and_load_unlocked
    _call_with_frames_removed(import_, parent)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 321, in _call_with_frames_removed
    return f(*args, **kwds)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1200, in _load_unlocked
    self._exec(module)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1129, in _exec
    self.spec.loader.exec_module(module)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1336, in exec_module
    exec(code, module.__dict__)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\__init__.py", line 26, in <module>
    _bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py')
NameError: name '__file__' is not defined

Hope this helps.

Best regards,
Jurko Gospodnetić

[JurkoGospodneti]

As mentioned by Anthony Tuininga at:

https://bitbucket.org/anthony_tuininga/cx_freeze/issue/61/python-340-nameerror-name-__file__-is-not#comment-9077661

The value of __file__ in the problematic importlib/init.py
module with Python 3.3. is '<frozen>' while with Python 3.4 it is
not defined.

And this '<frozen>' string seems to be something coming from Python
and not cx_Freeze (I see it in 'Python/import.c' in both Python
3.3.5 & 3.4).

Hope this helps.

Best regards,
Jurko Gospodnetić