Tools/freeze no longer works in Python 3

[malemburg]

BPO	16047
Nosy	@malemburg, @loewis, @warsaw, @brettcannon, @jcea, @ericvsmith, @jkloth, @bitdancer, @Trundle, @meadori, @ericsnowcurrently
Files	issue16047-0.patch issue16047-1.patch site.py: manually patches version from Python 3.4.0

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/meadori'
closed_at = <Date 2014-03-30.19:36:46.475>
created_at = <Date 2012-09-25.16:27:39.527>
labels = ['type-bug']
title = 'Tools/freeze no longer works in Python 3'
updated_at = <Date 2014-03-31.17:46:59.853>
user = 'https://github.com/malemburg'

bugs.python.org fields:

activity = <Date 2014-03-31.17:46:59.853>
actor = 'meador.inge'
assignee = 'meador.inge'
closed = True
closed_date = <Date 2014-03-30.19:36:46.475>
closer = 'loewis'
components = ['Demos and Tools']
creation = <Date 2012-09-25.16:27:39.527>
creator = 'lemburg'
dependencies = []
files = ['28530', '28537', '34547']
hgrepos = []
issue_num = 16047
keywords = ['patch', '3.3regression']
message_count = 37.0
messages = ['171295', '175492', '178771', '178819', '178821', '178823', '178824', '178825', '178877', '214352', '214400', '214427', '214794', '214797', '214802', '214805', '214806', '214808', '214817', '214824', '214826', '214829', '214831', '214832', '214839', '214840', '214842', '214843', '214844', '214849', '214851', '214939', '214940', '214944', '215196', '215197', '215255']
nosy_count = 14.0
nosy_names = ['lemburg', 'loewis', 'barry', 'brett.cannon', 'jcea', 'eric.smith', 'jkloth', 'Arfrever', 'r.david.murray', 'Trundle', 'meador.inge', 'python-dev', 'eric.snow', 'chba']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16047'
versions = ['Python 3.4', 'Python 3.5']

[malemburg]

The freeze tool used for compiling Python binaries with frozen modules no longer works with Python 3.x.

It looks like it was never updated to the various path and symbols changes introduced with PEP-3149 (ABI tags) in Python 3.2.

Even with lots of symlinks to restore the non-ABI flagged names, freezing fails with a linker error in Python 3.3:

Tools/freeze> python3 freeze.py hello.py
Tools/freeze> make
config.o:(.data+0x38): undefined reference to `PyInit__imp'
collect2: ld returned 1 exit status
make: *** [hello] Error 1

[Trundle]

See also issue bpo-11824 for the ABI tags changes.

[meadori]

As mentioned, the ABI issues are being handled in bpo-11824. I believe the linking problems in this issue have to do with the changes that were made in 3.3 to bootstrap importlib into Python. I will look into it more.

[meadori]

After applying the fix for bpo-11824 to tip there ended up being three more
issues:

1. makeconfig.py should have been updated in d777f854a66e when changing
   imp to _imp. Without this change a reference to 'PyInit__imp'
   was being generated, but that function does not actually exist.

2. After fixing (1) I hit an issue where '_frozen_importlib' can
   not be found while bringing up the interpreter.

3. After fixing (2) I hit the following error when running the
   simple frozen hello app:

         $ ./hello 
         Traceback (most recent call last):
      
         ...

       File "/Users/meadori/Code/src/cpython/Lib/site.py", line 578, in main
         setcopyright()
       File "/Users/meadori/Code/src/cpython/Lib/site.py", line 438, in setcopyright
         here = os.path.dirname(os.__file__)
     AttributeError: 'module' object has no attribute '__file__'

  This happens during 'initsite' and is because of a change made
  to remove the __file__ attribute from frozen modules in 702009f3c0b1.

The attached patch fixes (1) by filtering '_imp' instead of 'imp' in
makeconfig.py, (2) by making sure '_frozen_importlib' is in the frozen
module table created by freeze.py, and (3) by removing the code to delete
__file__ from frozen modules.

I am a little unsure about (3) since I am not sure why the __file__ attribute
is being removed to begin with. eric.smith?

[warsaw]

On Jan 02, 2013, at 04:06 PM, Meador Inge wrote:

I am a little unsure about (3) since I am not sure why the __file__ attribute
is being removed to begin with. eric.smith?

This is related to PEP-420, which relaxes the requirement that all modules
have a __file__ attribute. Now they only need it if it makes sense. At the
time, we didn't think it made sense for frozen modules to have an __file__.
FrozenImporter.module_repr() provides a reasonable repr in the absence of
__file__.

I'm not sure what to do. I still think it doesn't make a lot of sense for
frozen modules to have an __file__, but perhaps practicality beats purity
here.

[ericvsmith]

What Barry said. :)

I haven't had time to check yet: but why does site.py need the __file__ attribute? Maybe that's the actual problem.

[warsaw]

On Jan 02, 2013, at 04:29 PM, Eric V. Smith wrote:

I haven't had time to check yet: but why does site.py need the __file__
attribute? Maybe that's the actual problem.

It uses it to find the license text when you type license() at the interactive
prompt. I think setcopyright() should just ignore that if os.__file__ isn't
defined. Why would you be printing the license at a prompt in a frozen
Python?

[brettcannon]

I agree that it makes no sense to define __file__ for frozen modules. Originally they did because that was the only way to tell if a module was a builtin module or not, but with the imp module's API on top of sys.builtin_module_names, there is no need to maintain this invariant.

As for site.py requiring the module to have a __file__ on os, I think that is somewhat bogus as well and should be optional so issue (3) for Meador should be to patch site.py to not flat-out require os.__file__ exist.

[meadori]

Thanks for the feedback everyone -- that helped. Here is a new patch
which addresses issue (3) by not requiring os.__file__ to exist.

The patch from bpo-11824 fixes freeze for 3.2 completely. The patch
from bpo-11824 plus this patch fixes freeze for 3.3 and 3.4. I tested
both make install and Python source tree versions (i.e. freeze.py -p).
A frozen "Hello, world" app works fine in all cases.

I also verified that 'license' works from a frozen app. You will see
something like the following instead of the full licensing text on stdout:

See http://www.python.org/3.4/license.html

[chba]

Thanks for the patches. After applying bpo-11824-0.patch and then isse16047-1.patch I am successfully able to freeze a hello world python script under ubuntu 14.04 with python 3.4 rc3. I have attached my new site.py file, since the automatical patch application did not run through.

However, in my productive project I need the psycopg2 library for database access. Whenever using 'import psycopg2' in my code the binary shows a runtime error that it cannot find the module psycopg2._psycopg. A one line script suffices for reproduction. Executing the script without freezing it works like a charm. In Python 3.2 it helped to create in the working dir a psycopg2 subdir with a symlink _psycopg.so -> /usr/lib/python3/dist-packages/psycopg2/_psycopg.cpython-32mu.so . This seems no longer help. Is this a bug/feature of freeze or psycopg2? Thanks for your interest and help.

[brettcannon]

Did you want to update your patch for Python 3.4 and 3.5, Meador?