Reland "Compute GUIDs once and store in metadata" (#184065) #201849
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
ad94f68 to
a1a789e
Compare
🐧 Linux x64 Test Results
✅ The build succeeded and all tests passed. |
8cd8eb5 to
f053482
Compare
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
f053482 to
9926393
Compare
🪟 Windows x64 Test Results
✅ The build succeeded and all tests passed. |
13d3ad0 to
f8a1ec9
Compare
854a6f9 to
6dc97cf
Compare
|
✅ With the latest revision this PR passed the LLVM ABI annotation checker. |
6f02138 to
c7ff207
Compare
c7ff207 to
525046e
Compare
fa7961a to
1817d11
Compare
1817d11 to
2829ab3
Compare
|
Does this also address the compile-time regressions from the previous implementation? |
| } | ||
|
|
||
| if (!ShouldCloneDefinition(&I)) { | ||
| CopyMD(); |
There was a problem hiding this comment.
Doesn't this line create a blowup of unnecessary copies?
There was a problem hiding this comment.
good catch - fixed. Thanks!
|
I looked more into it but didn't update the other thread. IIRC the remaining surprise was the clang build regressing by 0.15%, because for the single-TU compilations, the regression was attributable to modules with lots of global values but not so much IR, where the bitcode serialization of the extra GUID table would start showing up. I'm actually not able to repro the clang build - I figured in the meantime that the "instructions" link in the clang-build part of the report gives me a per-module compilation change, and I found a module ( What's the cmake you use for building the host clang, and what's the cmake for the build target? |
a6fdde1 to
4da2c32
Compare
|
Turns out the clang build regression was in large due to some Fixed and re-ran the compile time tracker, the largest clang-build regression is now 1%. |
4da2c32 to
4bf5379
Compare
|
The stage1-ReleaseThinLTO results still don't look great and it seems like the geomean for the clang build is about the same? |
4bf5379 to
1cbfe8b
Compare
Note that some compile time performance effect in thinlto builds should be expected: we serialize/deserialize more. I think the main concern was "is there something else". re. stage1 - yes, see my earlier response ("for the single-TU compilations, the regression was attributable to modules with lots of global values but not so much IR, where the bitcode serialization of the extra GUID table would start showing up.") Re. clang-build, there's little change in the overall build timing (I don't think it's a geomean) because the individual compilation outliers like ModuleLinker were few and their negative contribution likely lost, before, in the overall parallel compilation part of the build. The rest is very likely for the same reason (more serialization) seen in the benchmarks builds, as at most we see compilation regressions within the same relative ranges for individual modules. |
|
To clarify, the regression I was concerned about is the one on bin/clang-23, i.e. the one during thin linking, not the one on individual files (pre-link). |
|
Ah, ok. About that one: Looking at the profile itself, ( Before: After: This lines up with the "we serialize more" (and so we deserialize during linking). Not sure what to make of the |
1cbfe8b to
55df97c
Compare
55df97c to
e432443
Compare
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/196/builds/20771 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/232/builds/3124 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/174/builds/37887 Here is the relevant piece of the build log for the reference |
This reverts #201194, thus relanding @orodley's PR #184065 (and #200323): > This allows us to keep GUIDs consistent across compilation phases which may change the name or linkage type. > See https://discourse.llvm.org/t/rfc-keep-globalvalue-guids-stable/84801 The CFI issues that triggered the original revert are fixed by #201370, together with the addressing of the TODOs in `LowerTypeTests.cpp` left in the latter. The [graphite diff](https://app.graphite.com/github/pr/llvm/llvm-project/201849/Reland-%23184065) between this change's V1 and V2 shows what's been added: - the `TODO`s from #201370 are done - in LowerTypeTests.cpp, passing `!guid` when creating a new declaration and when converting a definition to a declaration. - `llvm/test/Transforms/LowerTypeTests/export-icall.ll` tests also the above def->decl conversion - removed `test/Analysis/CtxProfAnalysis/flatten-prethinlink-requires-guid-metadata.ll` introduced in #194383 (this was between the revert and this PR), as now the general expectation is that GUID assignment happens appropriately and all passes use `getGUID`, so there's no reason for `CtxProfAnalysis` to do something different. Currently, we reassign GUIDs when CFI promotes internal linkage symbols, which is counter to the goal of the RFC. This is addressed in PR #203171. The reason for this split fix can be explained on `compiler-rt/test/cfi/icall/wrong-signature-mixed-lto.c`. Here, a module with the exact same source path is compiled twice, under different conditional compilation, to produce 2 objects. Each object defines an internal linkage symbol with the same name (this is `install_trap_loop_detection` from `compiler-rt/test/cfi/trap_loop_signal_handler.inc` which is `-include`\-d by both - see how `%clang_cfi` is defined). The ThinLTO GUID of this symbol will be the same. Its name won't be - because CFI promotes it and renames it using a hash that is based on the IR Module content (rather than the source path). During thinlink, `LTO::addThinLTO`will mark each of the 2 exported symbols as prevailing in their corresponding modules. But that is done by associating their GUID to the module. So whichever comes last wins. The other symbol will be marked available externally and its body DCEd later in backend. But each module will refer to its copy of `install_trap_loop_detection`, and so we end up with a linker error. As mentioned, the fix is in PR #203171, and this relanding PR just maintains the existing ThinLTO behavior by rewriting the GUIDs. Since we haven't yet leveraged the GUID mechanics for e.g. simplifying PGO, this aspect of this change is essentially NFC. Co-authored-by: Owen Rodley <orodley@google.com>

This reverts #201194, thus relanding @orodley's PR #184065 (and #200323):
The CFI issues that triggered the original revert are fixed by #201370, together with the addressing of the TODOs in
LowerTypeTests.cppleft in the latter. The graphite diff between this change's V1 and V2 shows what's been added:TODOs from [CFI][ThinLTO] Remove the need for CFI calculating ThinLTO GUIDs #201370 are done!guidwhen creating a new declaration and when converting a definition to a declaration.llvm/test/Transforms/LowerTypeTests/export-icall.lltests also the above def->decl conversiontest/Analysis/CtxProfAnalysis/flatten-prethinlink-requires-guid-metadata.llintroduced in [CtxProf] emit fatal usage error when flatten-prethinlink runs without guid metadata #194383 (this was between the revert and this PR), as now the general expectation is that GUID assignment happens appropriately and all passes usegetGUID, so there's no reason forCtxProfAnalysisto do something different.Currently, we reassign GUIDs when CFI promotes internal linkage symbols, which is counter to the goal of the RFC. This is addressed in PR #203171. The reason for this split fix can be explained on
compiler-rt/test/cfi/icall/wrong-signature-mixed-lto.c. Here, a module with the exact same source path is compiled twice, under different conditional compilation, to produce 2 objects. Each object defines an internal linkage symbol with the same name (this isinstall_trap_loop_detectionfromcompiler-rt/test/cfi/trap_loop_signal_handler.incwhich is-include-d by both - see how%clang_cfiis defined). The ThinLTO GUID of this symbol will be the same. Its name won't be - because CFI promotes it and renames it using a hash that is based on the IR Module content (rather than the source path). During thinlink,LTO::addThinLTOwill mark each of the 2 exported symbols as prevailing in their corresponding modules. But that is done by associating their GUID to the module. So whichever comes last wins. The other symbol will be marked available externally and its body DCEd later in backend. But each module will refer to its copy ofinstall_trap_loop_detection, and so we end up with a linker error.As mentioned, the fix is in PR #203171, and this relanding PR just maintains the existing ThinLTO behavior by rewriting the GUIDs. Since we haven't yet leveraged the GUID mechanics for e.g. simplifying PGO, this aspect of this change is essentially NFC.
Co-authored-by: Owen Rodley orodley@google.com