--with-lto builds segfault in many situations

[gpshead]

BPO	28032
Nosy	@gpshead, @pitrou, @larryhastings, @benjaminp, @ned-deily, @yaseppochi, @koobs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2016-11-21.08:19:52.627>
created_at = <Date 2016-09-08.21:32:32.434>
labels = ['invalid', 'type-crash']
title = '--with-lto builds segfault in many situations'
updated_at = <Date 2016-11-21.08:19:52.626>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2016-11-21.08:19:52.626>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2016-11-21.08:19:52.627>
closer = 'gregory.p.smith'
components = []
creation = <Date 2016-09-08.21:32:32.434>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 28032
keywords = []
message_count = 12.0
messages = ['275171', '275177', '275182', '275194', '275198', '280630', '280631', '281300', '281313', '281327', '281330', '281332']
nosy_count = 9.0
nosy_names = ['gregory.p.smith', 'pitrou', 'larry', 'benjamin.peterson', 'ned.deily', 'sjt', 'python-dev', 'koobs', 'alecsandru.patrascu']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue28032'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[gpshead]

configure --with-optimizations is failing on the Debian --with-optimizations buildbot:

http://buildbot.python.org/all/builders/AMD64%20Debian%20PGO%203.x/builds/2448/steps/compile/logs/stdio

The compiled python is segfaulting when it goes to run the interpreter to build modules via setup.py.

this passes manually on ubuntu 16.04. debian jessie 8.5 is failing here. older gcc 4.9.2 toolchain.

i suspect --with-lto is not a good thing to enable by default everywhere.

configure.ac should be updated to exclude --with-lto in more situations than just Darwin. we also know it doesn't work with clang as our configure and makefile are today. so start with a conditional on gcc version being at least 5.4 (which is what ubuntu 16.04 ships with)? or is it related to the binutils (ld) version when gcc is present? or both. LTO is a maze of twisty passages.

To lower this from "release blocker" we need to make --with-optimizations not enable --with-lto anywhere it is likely to fail.

[pitrou]

Can we just split --with-optimizations into --with-pgo and --with-lto, and let users choose whichever combination they want?

[gpshead]

That is effectively what this devolves to. i'm going to try "gcc and gcc > 5.4" implies --with-lto as part of --with-optimizations for now. but if that causes problems, i'll just remove the --with-lto implication from --with-optimizations entirely.

[python-dev]

New changeset b77465544f28 by Gregory P. Smith in branch '3.5':
bpo-28032: never imply --with-lto as part of --with-optimizations.
https://hg.python.org/cpython/rev/b77465544f28

New changeset 4a50d87808a6 by Gregory P. Smith in branch 'default':
bpo-28032: never imply --with-lto as part of --with-optimizations.
https://hg.python.org/cpython/rev/4a50d87808a6

[python-dev]

New changeset 7e89469e4342 by Gregory P. Smith in branch '2.7':
bpo-28032: never imply --with-lto as part of --with-optimizations.
https://hg.python.org/cpython/rev/7e89469e4342

[koobs]

Could we please:

a) Rename 'optimizations' to 'pgo' or 'profiled'
b) Use enable/disable instead of with/without. The former is used for toggling program features, the latter is used for external dependencies [1][2][3]

Yes Python's configure / autofoo isn't the paragon of conventions, consistency or elegance, but it would be fantastic to make the job of bringing it to (or arguing for) a more standards-oriented state moving forward easier rather than harder.

[1] https://www.gnu.org/software/autoconf/manual/autoconf-2.66/html_node/Package-Options.html
[2] https://www.gnu.org/software/autoconf/manual/autoconf-2.66/html_node/External-Software.html
[3] https://autotools.io/autoconf/arguments.html

[koobs]

I also invoke PEP-20 (explicit > implicit) in favour of renaming the option :]

[gpshead]

The configure flag has been renamed to --enable-optimizations in the following commits for 3,5, 3.6, default, & 2.7 branches (everywhere it exists):

remote: notified python-checkins@python.org of incoming changeset c0ea81315fb6
remote: notified python-checkins@python.org of incoming changeset 58c1a49a10b4
remote: notified python-checkins@python.org of incoming changeset 11cacf9f9a33
remote: notified python-checkins@python.org of incoming changeset 0d2b42344ae5
remote: notified python-checkins@python.org of incoming changeset 2d1d70b53376
remote: notified python-checkins@python.org of incoming changeset d5ff5a2f33fd

[yaseppochi]

FWIW, XEmacs has used a bit of m4 magic to make --with-* and --enable-* equivalent for 15 years, and nobody has ever complained. The autotools convention is a distinction without a difference, and confuses users when a program feature depends on an external library (especially where there are alternative implementations).

[ned-deily]

For 3.6 at least, there are still mentions of --with-optimizations in Doc/whatsnew/3.6.rst and README.