AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge

@nascheme

BPO	31963
Nosy	@nascheme, @gpshead, @vstinner, @albertchristianto1994

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-02-10.23:56:08.615>
created_at = <Date 2017-11-06.21:57:41.142>
labels = ['build', '3.7']
title = 'AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge'
updated_at = <Date 2020-05-24.14:31:35.818>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-05-24.14:31:35.818>
actor = 'albertchristianto1994'
assignee = 'none'
closed = True
closed_date = <Date 2020-02-10.23:56:08.615>
closer = 'vstinner'
components = ['Build']
creation = <Date 2017-11-06.21:57:41.142>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 31963
keywords = []
message_count = 14.0
messages = ['305682', '305687', '305707', '305708', '305906', '306157', '306177', '306237', '309643', '309680', '361756', '369328', '369351', '369804']
nosy_count = 4.0
nosy_names = ['nascheme', 'gregory.p.smith', 'vstinner', 'albertchristianto1994']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue31963'
versions = ['Python 3.7']

http://buildbot.python.org/all/#/builders/47/builds/127

(...)
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction -I. -I./Include -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c
Objects/setobject.c: In function ‘PySet_Clear’:
Objects/setobject.c:2579:1: internal compiler error: in create_edge, at cgraph.c:850
};
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
Makefile:1584: recipe for target 'Objects/setobject.o' failed
make[1]: Leaving directory '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build'
make[1]: *** [Objects/setobject.o] Error 1

pythoninfo from the previous build:

GCC 6.3.0 20170516
os.uname: posix.uname_result(sysname='Linux', nodename='enable-optimizations-bot', release='4.9.0-3-amd64', version='Support "bpo-" in Misc/NEWS #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)', machine='x86_64')
platform.platform: Linux-4.9.0-3-amd64-x86_64-with-debian-9.1

http://buildbot.python.org/all/#/builders/47/builds/126/steps/3/logs/stdio

Hi Victor,

My first guess is that the build bot is not cleaning the fprofile
information after updating the source tree. A few options:

remove the profile-run-stamp file after checking out new code
call "make profile-removal" after checkout of new code
I can try to modify the makefile so that profile-run-stamp depends
on the source files (.h, .c, etc). I think that would make it
re-generate automatically if the sources change.

On 2017-11-06, STINNER Victor wrote:

New submission from STINNER Victor <victor.stinner@gmail.com>:

http://buildbot.python.org/all/#/builders/47/builds/127

(...)
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction -I. -I./Include -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c
Objects/setobject.c: In function ‘PySet_Clear’:
Objects/setobject.c:2579:1: internal compiler error: in create_edge, at cgraph.c:850
};
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
Makefile:1584: recipe for target 'Objects/setobject.o' failed
make[1]: Leaving directory '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build'
make[1]: *** [Objects/setobject.o] Error 1

pythoninfo from the previous build:

GCC 6.3.0 20170516

os.uname: posix.uname_result(sysname='Linux', nodename='enable-optimizations-bot', release='4.9.0-3-amd64', version='Support "bpo-" in Misc/NEWS #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)', machine='x86_64')

platform.platform: Linux-4.9.0-3-amd64-x86_64-with-debian-9.1

http://buildbot.python.org/all/#/builders/47/builds/126/steps/3/logs/stdio

----------
components: Build
messages: 305682
nosy: gregory.p.smith, haypo, nascheme
priority: normal
severity: normal
status: open
title: AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge
type: performance
versions: Python 3.7

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue31963\>

On Mon, Nov 6, 2017 at 3:11 PM Neil Schemenauer <report@bugs.python.org>
wrote:

Neil Schemenauer <nas-python@arctrix.com> added the comment:

Hi Victor,

My first guess is that the build bot is not cleaning the fprofile
information after updating the source tree. A few options:

But it is. I actually looked at what the buildbots did before the PR to
change the Makefile.pre.in for profile_opt builds went in.

Here's what happens:

http://buildbot.python.org/all/#/builders/47/builds/127/steps/2/logs/stdio

Notice the "make clean profile-removal" on line 17.

-gps

Oh I see. I'm at a loss then as to why the build is failing. A possible clue is the errors like:

profiling:/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build/Objects/setobject.gcda:Merge mismatch for function 10

A Google search brings up this Stackoverflow page:

https://stackoverflow.com/questions/2590794/gcov-warning-merge-mismatch-for-summaries

To me, that implies that there are compiler outputs (.o, .a, etc) that were build from a previous version source code. However, the first part of the log shows that the build does "make clean" so I don't see how that is possible.

Is there a way to run "git show 4e38d71 | git apply -R" on the built host and then try to re-build?

The current "master" branch seems to be building successfully on "AMD64 Debian PGO 3.x". Can we close this issue?

Same bug, again, AMD64 Debian PGO 3.x, so I reopen the issue:

http://buildbot.python.org/all/#/builders/47/builds/159

Python/compile.c: In function ‘compiler_sync_comprehension_generator’:
Python/compile.c:5457:1: internal compiler error: in create_edge, at cgraph.c:850
}
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.

Given this is gcc exploding, I'm not sure there is anything we can do about it. I _believe_ we're doing everything right.

I'm willing to put some time into trying to fix this, in the case that it is caused by my Makefile changes. However, it would be very helpful if I could login to the build-bot and try running with the Makefile change backed out. Is that possible? Having to setup a VM to match the compiler of the build-bot would take quite a bit a of extra time.

Looking at the "stdout" log file, I can't see that we are doing anything wrong. Specifically, the ".gc??" files are being removed and so I think that rules out the theory of stale profile information files messing up GCC.

Different error:

http://buildbot.python.org/all/#/builders/47/builds/428

Objects/bytearrayobject.c: In function ‘bytearray_startswith’:
Objects/bytearrayobject.c:2429:1: error: the control flow of function ‘bytearray_startswith’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
}
^
cc1: some warnings being treated as errors

That's mysterious. I reviewed the 'stdout' log from the buildbot. The removal of profile data is run early in the build:

find . -name '*.gc??' -exec rm -f {} ';'

Later, the bytearrayobject is compiled with profile generation enabled:

gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate -I. -I./Include -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c

When the unit tests run, we get errors:

profiling:/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build/Objects/bytearrayobject.gcda:Merge mismatch for function 35

The second build (with profile use) fails because of the missing profile information:

gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction -I. -I./Include -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c
Objects/bytearrayobject.c: In function ‘bytearray_startswith’:
Objects/bytearrayobject.c:2429:1: error: the control flow of function ‘bytearray_startswith’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
}
^
cc1: some warnings being treated as errors

Seems like a compiler bug to me. Wild theory, could it be something to do with address randomization and forking? Looks like maybe the merge mismatch errors come when the 'python' executable forks.

I didn't see this failure recently, I close the issue.

Hi admins,

I need help for compiling and building Python-3.7.7.
My system is Ubuntu 16.04 LTS, gcc 5.4.0 20160609

this is the configuration cmd for compling python
./configure --enable-optimizations --enable-shared

when i compiled it, i get these similar error

Parser/tokenizer.c: In function ‘PyTokenizer_FindEncodingFilename’:
Parser/tokenizer.c:1909:1: error: the control flow of function ‘PyTokenizer_FindEncodingFilename’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
 }
 ^
Parser/tokenizer.c:1909:1: error: the control flow of function ‘PyTokenizer_FindEncodingFilename’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Parser/tokenizer.c: In function ‘tok_get’:
Parser/tokenizer.c:1909:1: error: the control flow of function ‘tok_get’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Parser/tokenizer.c:1909:1: error: the control flow of function ‘tok_get’ does not match its profile data (counter ‘single’) [-Werror=coverage-mismatch]
Parser/tokenizer.c:1909:1: error: the control flow of function ‘tok_get’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include   -fPIC -DPy_BUILD_CORE -o Objects/bytes_methods.o Objects/bytes_methods.c
Objects/boolobject.c: In function ‘bool_xor’:
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_xor’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
 };
 ^
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_xor’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_xor’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/boolobject.c: In function ‘bool_or’:
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_or’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_or’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_or’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/boolobject.c: In function ‘bool_and’:
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_and’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_and’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_and’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/boolobject.c: In function ‘bool_new’:
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_new’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_new’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include   -fPIC -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c
cc1: some warnings being treated as errors
Makefile:1652: recipe for target 'Objects/boolobject.o' failed
make[1]: *** [Objects/boolobject.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Objects/abstract.c: In function ‘PyObject_GetIter’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_GetIter’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
 }
 ^
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_GetIter’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_GetIter’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PySequence_InPlaceRepeat’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceRepeat’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceRepeat’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceRepeat’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PySequence_InPlaceConcat’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceConcat’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceConcat’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceConcat’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PySequence_Repeat’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Repeat’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Repeat’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Repeat’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PySequence_Concat’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Concat’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Concat’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Concat’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PyObject_CopyData’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_CopyData’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_CopyData’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PyObject_LengthHint’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_LengthHint’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_LengthHint’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_LengthHint’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
cc1: some warnings being treated as errors
Makefile:1652: recipe for target 'Parser/tokenizer.o' failed
make[1]: *** [Parser/tokenizer.o] Error 1
cc1: some warnings being treated as errors
Makefile:1652: recipe for target 'Objects/abstract.o' failed
make[1]: *** [Objects/abstract.o] Error 1
make[1]: Leaving directory '/home/ion/Documents/Python-3.7.7'
Makefile:534: recipe for target 'profile-opt' failed
make: *** [profile-opt] Error 2

what should i do to fix this? thank you very much.

best regards,
Albert Christianto

I need help for compiling and building Python-3.7.7. My system is Ubuntu 16.04 LTS, gcc 5.4.0 20160609

You have different options:

Upgrade Ubuntu to the newer LTS to get a more recent C compiler
Don't use PGO

Also, you might use a pre-built package rather than build Python yourself? See https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa

Sorry for my late response.
Well, thank you very much for your fast response to help me.
Actually, I have solved the problem in 3 hours after I posted my question. hehehe.
I found this tip about uncleaned files after executing "make clean" (https://bugs.python.org/msg346553). So, I decided to wipe out all python3.7 files from my computer and repeat the installation process. It worked!!
Anyway, thank you again for your response and your tip. It really made my day like I am not alone while doing some python program debugging. (^v^)
best regards,
Albert Christianto

Navigation Menu

Repository navigation

AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge #76144

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge #76144

Description

Activity

vstinner commented on Nov 6, 2017

nascheme commented on Nov 6, 2017

gpshead commented on Nov 7, 2017

nascheme commented on Nov 7, 2017

nascheme commented on Nov 8, 2017

vstinner commented on Nov 13, 2017

gpshead commented on Nov 13, 2017

nascheme commented on Nov 14, 2017

vstinner commented on Jan 7, 2018

nascheme commented on Jan 8, 2018

vstinner commented on Feb 10, 2020

albertchristianto1994 commented on May 19, 2020

vstinner commented on May 19, 2020

albertchristianto1994 commented on May 24, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions