"Support PyInstaller" broke Pyinstaller support

[yagebu]

PyInstaller has supported Babel for ages (it has had a hook for Babel since 2009). #505 actually broke PyInstaller support AFAICS (tested on Python 3.6 with PyInstaller dev version and Python 3.5 with the latest PyInstaller release). PyInstaller will automatically bundle the data files (babel/locale-data) using the hook and when the executable is run, these files are extracted to a temporary folder, say $TMP/_MEI123456/babel/locale-data. However the new code will look for it in {sys._MEIPASS}/locale-data, i.e., $TMP/_MEI123456/locale-data and fail.

PyInstaller actually sets __file__ for "frozen" modules to a path relative to the temp folder that the data files get extracted to (see pyinstaller/pyinstaller#1598 (comment)). So the old was already doing the right thing - if it didn't work in some setup, that was really a PyInstaller bug.

[akx]

@wodo: Can you chime in on this?

[akx]

Hi @wodo :) Can you tell us about the setup you had (#146, #500, #505) that had troubles with PyInstaller?

[wodo]

The start of our troubles was try to use babel and a lot of other packages (e.g. Sphinx) together in one pyinstaller generated exe. It seems the locale-data directory is searched from different packages at different places. One package call it localedata the other locale-data and so on... On clean way to solve this was to place it in a common folder and let the path calculated by babel directly (calculating basedir) and not by some hooks . But at the moment after reading this thread this idea seems not so clever.
The setup starts with Python 2.7.12, Sphinx 1.4.8, babel 2.3.4, pyinstaller 2.1, ...

[yagebu]

These other packages (your packages?) tried to access Babel's locale-data directly? So if you update them to access {sys._MEIPASS}/babel/locale-data instead of {sys._MEIPASS}/locale-data the old code should work, right?

I've submitted a PR that restores the old code.

[akx]

@yagebu Can you explain how PyInstaller does its thing?
There's a hook file https://github.com/pyinstaller/pyinstaller/blob/3c87c135abdfaf435ad3bbaf481a13be211c10a9/PyInstaller/hooks/hook-babel.py ... does collect_data_files collect the .dats, then drop them back into some temporary directory during run of a PyInstallered app, or..?

I can also find a commit from 2015 that removes a runtime hook that touched these paths (pyinstaller/pyinstaller@a7b29d8)...

[yagebu]

does collect_data_files collect the .dats, then drop them back into some temporary directory during run of a PyInstallered app, or..?

Yes pretty much. As far as I can tell collect_data_files simply collects all non-Python files it can find in the directory of the Babel package (so global.dat and locale-date/*.dat) and zips them up. When the bundled executable is run, these files get extracted to a temporary directory (sys._MEIPASS). The Python files don't get extracted but directly imported from the bundled archive. Their __file__ attribute will be set by PyInstaller on import so that relative paths to data files work. So the __file__ attribute of babel/localedata.py will be something like /tmp/_MEIE4nhbc/babel/localedata.pyc. That file doesn't exist - it's just to make the relative paths work.

I can also find a commit from 2015 that removes a runtime hook that touched these paths

Maybe that hook was needed before PyInstaller set __file__

[yagebu]

@akx: Do you have any more questions about this or is there anything else I can do to get the PR merged and this issue fixed?

[akx]

Hey @yagebu!
I'm so so sorry I dropped the ball on this one. I verified this and you're absolutely correct, 2.5.x is broken with PyInstaller. (Not that I doubted you, I just wanted to verify, and then, well, forgot.)

Here's how I tested things out:

te.py

import babel
import babel.numbers  # (see below!)
from babel.dates import format_datetime
import datetime
import argparse

ap = argparse.ArgumentParser()
ap.add_argument('locale')
args = ap.parse_args()

print(format_datetime(datetime.datetime.now(), locale=babel.Locale.parse(args.locale)))

rm -r build dist && pyinstaller te.py was used to build ./dist/te/te.

With Babel 2.4 ./dist/te/te fi prints the expected "15.1.2018 klo 12.34.14".
With Babel 2.5.x, ./dist/te/te fi crashes with

Traceback (most recent call last):
  File "te.py", line 11, in <module>
    print(format_datetime(datetime.datetime.now(), locale=babel.Locale.parse(args.locale)))
  File "site-packages/babel/core.py", line 288, in parse
  File "site-packages/babel/core.py", line 273, in _try_load
  File "site-packages/babel/core.py", line 167, in __init__
  File "site-packages/babel/localedata.py", line 64, in exists
  File "site-packages/babel/localedata.py", line 47, in normalize_locale
  File "site-packages/babel/localedata.py", line 76, in locale_identifiers
FileNotFoundError: [Errno 2] No such file or directory: '/Users/akx/build/babel-test/dist/te/locale-data'
[12174] Failed to execute script te

As an aside, there seems to be a minor bug in PyInstaller, making it require a dummy import of babel.numbers, likely because there's some pickled data that refers to that module. Without the babel.numbers import, the built executable crashes (on Babel 2.4, didn't test with 2.5) with

Traceback (most recent call last):
  File "te.py", line 10, in <module>
  File "site-packages/babel/dates.py", line 718, in format_datetime
  File "site-packages/babel/dates.py", line 383, in get_datetime_format
  File "site-packages/babel/core.py", line 800, in datetime_formats
  File "site-packages/babel/core.py", line 363, in _data
  File "site-packages/babel/localedata.py", line 107, in load
  File "site-packages/babel/localedata.py", line 113, in load
ModuleNotFoundError: No module named 'babel.numbers'
[11903] Failed to execute script te

[yagebu]

Perfect @akx, thanks for the merge!

[warrenstephens]

currently getting

ModuleNotFoundError: No module named 'babel.numbers'

with pyinstaller 3.4 and python 3.6.4 on OSX

I am importing tkcalendar, but not babel directly.

[akx]

@warrenstephens Can you open another issue with more information and repro steps, please?