Fixing a bug related to LTO only build

[octaviansoldea]

BPO	31354
Nosy	@gpshead, @vstinner, @ned-deily, @stratakis, @octaviansoldea
PRs	bpo-31354: Fixing a bug related to LTO only build #3110 [3.6] bpo-31354: Let configure --with-lto work on all builds #10261

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2018-12-09.09:08:30.867>
created_at = <Date 2017-09-05.21:54:13.298>
labels = ['3.8', 'type-feature', '3.7', 'build']
title = 'Fixing a bug related to LTO only build'
updated_at = <Date 2018-12-09.09:08:30.865>
user = 'https://github.com/octaviansoldea'

bugs.python.org fields:

activity = <Date 2018-12-09.09:08:30.865>
actor = 'ned.deily'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2018-12-09.09:08:30.867>
closer = 'ned.deily'
components = ['Build']
creation = <Date 2017-09-05.21:54:13.298>
creator = 'octavian.soldea'
dependencies = []
files = []
hgrepos = []
issue_num = 31354
keywords = []
message_count = 11.0
messages = ['301381', '301385', '301722', '329444', '329844', '329876', '329897', '329925', '331346', '331351', '331418']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'vstinner', 'ned.deily', 'cstratak', 'octavian.soldea']
pr_nums = ['3110', '10261']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31354'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[octaviansoldea]

Hi All,

This is Octavian Soldea from the Scripting Languages Optimization team at Intel Corporation. I would like to submit a patch to fix an LTO build issue. With "./configure --with-lto" followed by "make", CPython should be built with LTO or Link-Time-Optimization. However, currently, no impact is made to the build process, meaning the flag is not effective. This patch has been created out of a python-dev discussion thread: http://www.mail-archive.com/python-dev@python.org/msg96469.html. The LTO flag was originally introduced by my team member Alecsandru Patrascu back in 2015: https://bugs.python.org/issue25702.

Before the change:

./configure --with-lto
make
gcc command line output example:
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I. -I./Include -DPy_BUILD_CORE -o Python/ceval.o Python/ceval.c
Note no "lto" flag was present. This is the same result as the default build without the LTO flag.

After changes:

./configure --with-lto
make
expected gcc command line output example =>
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I. -I./Include -DPy_BUILD_CORE -o Python/ceval.o Python/ceval.c

Thanks,
Octavian

[vstinner]

Would you mind to create a pull request?

[gpshead]

New changeset 4c81401 by Gregory P. Smith (octaviansoldea) in branch 'master':
bpo-31354: Let configure --with-lto work on all builds
4c81401

[vstinner]

New changeset f6b9459 by Victor Stinner (stratakis) in branch '3.6':
[3.6] bpo-31354: Let configure --with-lto work on all builds (GH-10261)
f6b9459

[stratakis]

This change exports -flto in cflags.

You can check it with python3-config --cflags after this commit.

Which means that every c extension which will use those cflags will also utilize -flto which wasn't happening before, thus I'd say it's a regression.

[gpshead]

Is it an actual problem to compile extension modules with -flto? (I realize as an extension module there isn't a huge benefit to the concept unless it happens to have multiple source files or link against a non-shared library)

[stratakis]

I'd say yes. In general python may have been compiled with -flto, but it's still a bit buggy from the compilers' side. It doesn't work well always or at all depending on the toolchain, and even if python was compiled with -flto, that doesn't mean that all c extensions compiled using that python build, would work properly with -flto.

In my opinion that should be a flag reserved for python only, if used for its compilation, and not having it propagated to c extensions, similarly to the PGO related flags.

[ned-deily]

This does sound like a regression bug that should be fixed in all branches. Another issue is that, if it is propagated to third-party extension module builds through distutils, there is no guarantee that the extension module is being built with the same compiler as Python itself was.