AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge

[vstinner]

BPO	31963
Nosy	@nascheme, @gpshead, @vstinner, @albertchristianto1994

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-02-10.23:56:08.615>
created_at = <Date 2017-11-06.21:57:41.142>
labels = ['build', '3.7']
title = 'AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge'
updated_at = <Date 2020-05-24.14:31:35.818>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-05-24.14:31:35.818>
actor = 'albertchristianto1994'
assignee = 'none'
closed = True
closed_date = <Date 2020-02-10.23:56:08.615>
closer = 'vstinner'
components = ['Build']
creation = <Date 2017-11-06.21:57:41.142>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 31963
keywords = []
message_count = 14.0
messages = ['305682', '305687', '305707', '305708', '305906', '306157', '306177', '306237', '309643', '309680', '361756', '369328', '369351', '369804']
nosy_count = 4.0
nosy_names = ['nascheme', 'gregory.p.smith', 'vstinner', 'albertchristianto1994']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue31963'
versions = ['Python 3.7']

[vstinner]

http://buildbot.python.org/all/#/builders/47/builds/127

(...)
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction -I. -I./Include -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c
Objects/setobject.c: In function ‘PySet_Clear’:
Objects/setobject.c:2579:1: internal compiler error: in create_edge, at cgraph.c:850
};
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
Makefile:1584: recipe for target 'Objects/setobject.o' failed
make[1]: Leaving directory '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build'
make[1]: *** [Objects/setobject.o] Error 1

pythoninfo from the previous build:

• GCC 6.3.0 20170516
• os.uname: posix.uname_result(sysname='Linux', nodename='enable-optimizations-bot', release='4.9.0-3-amd64', version='Support "bpo-" in Misc/NEWS #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)', machine='x86_64')
• platform.platform: Linux-4.9.0-3-amd64-x86_64-with-debian-9.1

http://buildbot.python.org/all/#/builders/47/builds/126/steps/3/logs/stdio

[nascheme]

Hi Victor,

My first guess is that the build bot is not cleaning the fprofile
information after updating the source tree. A few options:

• remove the profile-run-stamp file after checking out new code

• call "make profile-removal" after checkout of new code

• I can try to modify the makefile so that profile-run-stamp depends
  on the source files (.h, .c, etc). I think that would make it
  re-generate automatically if the sources change.

On 2017-11-06, STINNER Victor wrote:

New submission from STINNER Victor <victor.stinner@gmail.com>:

http://buildbot.python.org/all/#/builders/47/builds/127

(...)
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction -I. -I./Include -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c
Objects/setobject.c: In function ‘PySet_Clear’:
Objects/setobject.c:2579:1: internal compiler error: in create_edge, at cgraph.c:850
};
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
Makefile:1584: recipe for target 'Objects/setobject.o' failed
make[1]: Leaving directory '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build'
make[1]: *** [Objects/setobject.o] Error 1

pythoninfo from the previous build:

• GCC 6.3.0 20170516
• os.uname: posix.uname_result(sysname='Linux', nodename='enable-optimizations-bot', release='4.9.0-3-amd64', version='Support "bpo-" in Misc/NEWS #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)', machine='x86_64')
• platform.platform: Linux-4.9.0-3-amd64-x86_64-with-debian-9.1

http://buildbot.python.org/all/#/builders/47/builds/126/steps/3/logs/stdio

----------
components: Build
messages: 305682
nosy: gregory.p.smith, haypo, nascheme
priority: normal
severity: normal
status: open
title: AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge
type: performance
versions: Python 3.7

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue31963\>

---

[gpshead]

On Mon, Nov 6, 2017 at 3:11 PM Neil Schemenauer <report@bugs.python.org>
wrote:

Neil Schemenauer <nas-python@arctrix.com> added the comment:

Hi Victor,

My first guess is that the build bot is not cleaning the fprofile
information after updating the source tree. A few options:

But it is. I actually looked at what the buildbots did before the PR to
change the Makefile.pre.in for profile_opt builds went in.

Here's what happens:

http://buildbot.python.org/all/#/builders/47/builds/127/steps/2/logs/stdio

Notice the "make clean profile-removal" on line 17.

-gps

[nascheme]

Oh I see. I'm at a loss then as to why the build is failing. A possible clue is the errors like:

profiling:/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build/Objects/setobject.gcda:Merge mismatch for function 10

A Google search brings up this Stackoverflow page:

https://stackoverflow.com/questions/2590794/gcov-warning-merge-mismatch-for-summaries

To me, that implies that there are compiler outputs (.o, .a, etc) that were build from a previous version source code. However, the first part of the log shows that the build does "make clean" so I don't see how that is possible.

Is there a way to run "git show 4e38d71 | git apply -R" on the built host and then try to re-build?

[nascheme]

The current "master" branch seems to be building successfully on "AMD64 Debian PGO 3.x". Can we close this issue?

[vstinner]

Same bug, again, AMD64 Debian PGO 3.x, so I reopen the issue:

http://buildbot.python.org/all/#/builders/47/builds/159

Python/compile.c: In function ‘compiler_sync_comprehension_generator’:
Python/compile.c:5457:1: internal compiler error: in create_edge, at cgraph.c:850
}
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.

[gpshead]

Given this is gcc exploding, I'm not sure there is anything we can do about it. I _believe_ we're doing everything right.

[nascheme]

I'm willing to put some time into trying to fix this, in the case that it is caused by my Makefile changes. However, it would be very helpful if I could login to the build-bot and try running with the Makefile change backed out. Is that possible? Having to setup a VM to match the compiler of the build-bot would take quite a bit a of extra time.

Looking at the "stdout" log file, I can't see that we are doing anything wrong. Specifically, the ".gc??" files are being removed and so I think that rules out the theory of stale profile information files messing up GCC.