CPython fails to build modules with LLVM LTO on Mac OS X

[Sjlver]

BPO	21122
Nosy	@brettcannon, @ronaldoussoren, @vstinner, @ned-deily, @koobs
Files	issue21122.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-06.23:22:25.643>
created_at = <Date 2014-04-01.12:11:00.925>
labels = ['OS-mac', 'build']
title = 'CPython fails to build modules with LLVM LTO on Mac OS X'
updated_at = <Date 2016-09-06.23:22:25.642>
user = 'https://bugs.python.org/Sjlver'

bugs.python.org fields:

activity = <Date 2016-09-06.23:22:25.642>
actor = 'brett.cannon'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-06.23:22:25.643>
closer = 'brett.cannon'
components = ['Build', 'macOS']
creation = <Date 2014-04-01.12:11:00.925>
creator = 'Sjlver'
dependencies = []
files = ['44352']
hgrepos = []
issue_num = 21122
keywords = ['patch']
message_count = 12.0
messages = ['215307', '215311', '215314', '215344', '215362', '215364', '215366', '215367', '215522', '226645', '274278', '274639']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'ronaldoussoren', 'vstinner', 'ned.deily', 'python-dev', 'koobs', 'Sjlver']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue21122'
versions = ['Python 3.5']

[Sjlver]

CPython fails to build with LLVM's link-time optimization (LTO) in Mac OS. Very similar commands work on Linux.

I'm currently configuring CPython as follows:

on Linux:
RANLIB="ar -s --plugin=/path/to/llvm/lib/LLVMgold.so" CC=/path/to/llvm/bin/clang CXX=/path/to/llvm/bin/clang++ CFLAGS="-g -flto" LDFLAGS="-flto" ./configure

on Mac OS:
CC=/path/to/llvm/bin/clang CXX=/path/to/llvm/bin/clang++ CFLAGS="-g -flto" LDFLAGS="-flto" ./configure

The RANLIB variable should not be needed on Mac, because its toolchain does not need a GOLD plugin.

On Linux, the above builds correctly and passes most of the test suite. On Mac, I receive the following error (similar for other extensions):

building '_scproxy' extension
/usr/bin/clang -Wno-unused-result -Werror=declaration-after-statement -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -g -flto -I../cpython/Include -I. -I/usr/local/include -I../cpython/Include -I../cpython-lto-build -c ../cpython/Modules/_scproxy.c -o build/temp.macosx-10.9-x86_64-3.5/path/to/cpython/Modules/_scproxy.o
/usr/bin/clang -bundle -undefined dynamic_lookup -flto build/temp.macosx-10.9-x86_64-3.5/path/to/cpython/Modules/_scproxy.o -L/usr/local/lib -o build/lib.macosx-10.9-x86_64-3.5/_scproxy.so -framework SystemConfiguration -framework CoreFoundation
*** WARNING: renaming "_scproxy" since importing it failed: dlopen(build/lib.macosx-10.9-x86_64-3.5/_scproxy.so, 2): Symbol not found: __Py_NoneStruct
Referenced from: build/lib.macosx-10.9-x86_64-3.5/_scproxy.so
Expected in: flat namespace
in build/lib.macosx-10.9-x86_64-3.5/_scproxy.so

[vstinner]

What is your clang version? See also issue bpo-20767.

[Sjlver]

I am indeed using Clang 3.4 (both the one that ships with Mac OS, and a version compiled from the sources).

However, the errors I get are rather different than bpo-20767. In particular, Clang finishes successfully and does produce shared object files; they just don't seem to be loadable.

[ned-deily]

Just as an experiment (using the 3.4 branch and the Xcode 5.1 clang), the list of unique symbols not found during the test dlopen in setup.py when using -flto:

_PyArg_ParseTuple
_PyArg_ParseTupleAndKeywords
_PyBaseObject_Type
_PyBool_Type
_PyByteArray_Type
_PyBytes_Type
_PyCFunction_Type
_PyExc_AssertionError
_PyExc_BufferError
_PyExc_EOFError
_PyExc_IndexError
_PyExc_IOError
_PyExc_KeyError
_PyExc_LookupError
_PyExc_MemoryError
_PyExc_NotImplementedError
_PyExc_OSError
_PyExc_OverflowError
_PyExc_RuntimeError
_PyExc_TypeError
_PyExc_ValueError
_PyModule_Create2
__Py_NoneStruct

Anyone see a pattern there?

Do we know if anyone has tried to use LTO with a Python build previously? I've never tried it myself and there certainly could be ld and/or dyld differences on OS X. Also, some thought would need to go into and tests developed to see what the performance trade-offs are. For example, I could imagine that LTO might be have more impact if the standard library extension modules were statically linked, e.g. via Modules/Setup*. And there are at least three separate current build configurations to consider on OS X: unshared, --enable-shared, --enable-framework. One would need to look at things like what effect these all have on memory and shared memory footprints as well as cpu resources and real time, with and without LTO and/or other optimizations. It certainly would be an interesting project for someone with the interest and time.

Potentially supporting LTO seems to me to be more of a feature than a bug so I think should be considered a 3.5 issue, at least initially.

[Sjlver]

Thanks Ned, this is interesting!

I don't know about Mac OS, but on Ubuntu, LTO and PGO apparently make Python around 10% faster (see bpo-17781). However, that data point refers to GCC's LTO, not LLVM's.

Personally I'm interested in LTO because I want to obtain whole-program LLVM bitcode files (for use in a research project about instrumentation). However, if there is something I can help (e.g., running benchmarks with different compilation settings), let me know.

[ronaldoussoren]

I've used -O4 for extensions in the past (which until recently implied LTO) and that worked fine.

I'm pretty sure that I haven't used LTO for python itself, apart from a some tests with an early version llvm-gcc where using LTO for building python used to crash the compiler :-)

BTW. There's no clear pattern in the missing symbols. The missing symbols for global functions could be due to aggressive inlining (and then deciding that the standalone function isn't needed anymore), but that is fairly unlikely and wouldn't explain the missing data symbols.

[ronaldoussoren]

Have you tried the -export_dynamic option for ld(1):

-export_dynamic
Preserves all global symbols in main executables during LTO. Without this option, Link Time Optimization is allowed to inline and remove global functions. This
option is used when a main executable may load a plug-in which requires certain symbols from the main executable.

[ronaldoussoren]

This works for me (with a separate build directory):

  CC=clang CXX=clang++ CFLAGS="-g -flto" LDFLAGS="-flto -Wl,-export_dynamic" ../configure

This is on OSX 10.9.2, with Xcode 5.1, and clang --version says:

$ clang --version
Apple LLVM version 5.1 (clang-503.0.38) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.1.0
Thread model: posix

Tests are still running, but so far there are no unexpected failures.

[Sjlver]

I confirm that this also works with my self-compiled Clang 3.4. -export_dynamic was the missing option.

Is a good place to document this?

Otherwise, I think this issue can be closed. Thanks a lot for the help!