Combining LTO + PGO + lib/cdylib crashes on LLVM assertion

[Kobzol]

When compiling a cargo project with:

• LTO enabled (LTO=thin/fat)
• PGO (-Cprofile-use)
• Both lib and dylib targets for its library

Reproducer

// src/main.rs
use clap::Command;

fn main() {
    Command::new("a").about("a");
}

# Cargo.toml
[package]
name = "hello"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
clap = { version = "4.3.24" , default-features = false, features = ["std"]}

[profile.release]
lto = "thin"

[lib]
crate-type = ["lib", "cdylib"]

There also needs to be an empty src/lib.rs file. Here is a reproducer script for PGO:

#!/bin/bash

set -e

VERSION=${1:-1.71}

rustup install ${VERSION}
rustup component add llvm-tools-preview --toolchain ${VERSION}
rm -rf target
rm -rf /tmp/pgo
RUSTFLAGS="-Cprofile-generate=/tmp/pgo" cargo +${VERSION} build --release
./target/release/hello /tmp
`rustc +${VERSION} --print=sysroot`/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata merge -o merged-${VERSION}.profdata /tmp/pgo/*
RUSTFLAGS="-Cprofile-use=${PWD}/merged-${VERSION}.profdata" cargo +${VERSION} build --release

I expected to see this happen: the project compiles.

Instead, compilation fails with the following error:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!

This was originally found in #115344, which contains many examples of this problem in the wild. Here I sent a minimal example, I wasn't able to minimize it further (removing LTO, PGO, clap or lib/dylib) fixes the problem.

Meta

rustc --version --verbose:

rustc 1.73.0 (cc66ad468 2023-10-03)
binary: rustc
commit-hash: cc66ad468955717ab92600c770da8c1601a4ff33
commit-date: 2023-10-03
host: x86_64-unknown-linux-gnu
release: 1.73.0
LLVM version: 17.0.2

I managed to reproduce this all the way back to 1.60.0 (although this specific example requires at least 1.70.0 due to clap).

[Jules-Bertholet]

@rustbot label T-compiler A-lto A-llvm I-crash

[dianqk]

I can reproduce the issue either with the Cargo.toml or using the lock file below.

# Cargo.toml
[package]
name = "hello"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
clap = { version = "=4.3.24" , default-features = false, features = ["std"]}

[profile.release]
lto = "thin"

[lib]
crate-type = ["lib", "cdylib"]

Cargo.lock

# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3

[[package]]
name = "anstyle"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "038dfcf04a5feb68e9c60b21c9625a54c2c0616e79b72b0fd87075a056ae1d1b"

[[package]]
name = "clap"
version = "4.3.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb690e81c7840c0d7aade59f242ea3b41b9bc27bcd5997890e7702ae4b32e487"
dependencies = [
 "clap_builder",
]

[[package]]
name = "clap_builder"
version = "4.3.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ed2e96bc16d8d740f6f48d663eddf4b8a0983e79210fd55479b7bcd0a69860e"
dependencies = [
 "anstyle",
 "clap_lex",
]

[[package]]
name = "clap_lex"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cd7cc57abe963c6d3b9d8be5b06ba7c8957a930305ca90304f24ef040aa6f961"

[[package]]
name = "hello"
version = "0.1.0"
dependencies = [
 "clap",
]

[dianqk]

Minimal Reproduction

First, let me provide a minimal reproduction:

# Cargo.toml
[package]
name = "hello"
version = "0.1.0"
edition = "2021"

[profile.release]
lto = "fat"  # "thin" is also ok.

[lib]
crate-type = ["lib", "cdylib"]

// src/lib.rs
pub fn foo() {
    let mut v = Vec::new();
    v.push(1);
}

// src/main.rs
fn main() {
    hello::foo();
}

Next, run the PGO script similar to the one described in the issue.

Why

What LLVM Did

The LLVM error indicates that we created two !"CG Profile".

This happens because every time we effectively run cg-profile, it creates a flag:
https://github.com/llvm/llvm-project/blob/602634d70cba2c51f6177740c4a98a377d10ab6a/llvm/lib/Transforms/Instrumentation/CGProfile.cpp#L42-L44

A simple example: https://llvm.godbolt.org/z/5bzeh6e77.

For me, running it multiple times might lead to unexpected behavior, and I'm not sure if the data can be merged or ignored here. cc @nikic

What rustc/cargo Did

The key point now is understanding why we are running CGProfile twice. The key parameters for the commands executed by cargo are as follows:

rustc src/lib.rs --crate-type lib --crate-type cdylib -Cprofile-use=merged.profdata
rustc src/main.rs --crate-type bin -C lto=fat --extern hello=/path/libhello.rlib -Cprofile-use=merged.profdata

We can remove --crate-type cdylib.

Because we did not pass -Clinker-plugin-lto, PGO data was applied when building libhello.rlib.
Finally, when building the bin, LTO added libhello.rlib that has the "CG Profile" flag.

How to Fix

I believe the key is that crate-type = ["lib", "cdylib"] is causing cargo to omit the -Clinker-plugin-lto and -Clto=fat.

Yes, there's also the issue that cdylib doesn't apply -Clto=fat.

Perhaps we should split this into two separate rustc commands? Or find a way to correctly pass these two parameters to different crate-type? cc @weihanglo

Discovery Process

Some may be interested in this.

A brief description:

1. Use -Csave-temps to get the LTO input IR and reproduce the issue using opt -O2.
2. (At this point, I wasn't yet aware of CGProfile) Use llvm-reduce to minimize this IR, and read the CGProfile code. I believe we executed this pass multiple times.
3. Use -Csave-temps and llvm-dis to find that the clap_builder IR includes the !"CG Profile".
4. I tried removing some code in clap_builder to reproduce the issue and realized that PGO only handles runtime path. I guessed this was the key, so I remained the calling code, and finally arrived at a minimal reproduction.

[insilications]

It seems to be happening with sccache

Using: cargo pgo optimize build -- --release --release --no-default-features

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `sccache` (bin "sccache");
PGO optimized build has failed.
Cargo finished with an error (101)

Rust version: 1.81.0-nightly (6b0f4b5 2024-06-24)
sccache version: main branch d5a1787a8582bd99d3f2fc27f4c3191bd8cc76d9

[weihanglo]

I believe the key is that crate-type = ["lib", "cdylib"] is causing cargo to omit the -Clinker-plugin-lto and -Clto=fat.

Yes. That is rust-lang/cargo#9672.

[dianqk]

I believe the key is that crate-type = ["lib", "cdylib"] is causing cargo to omit the -Clinker-plugin-lto and -Clto=fat.

Yes. That is rust-lang/cargo#9672.

So do we have to execute multiple crate types in the one command? What about compile time? Can we split it into multiple commands when encountering unsupported parameter scenarios?

[weihanglo]

That might require non-trivial works in Cargo internals (e.g. change Unit-to-CrateType relationship from 1:N to 1:1). For now, a workaround is removing the lib.crate-type from Cargo.toml and instead running cargo rustc --crate-type cdylib to override crate-type on-the-fly via CLI option.

[dianqk]

It seems to be happening with sccache

Using: cargo pgo optimize build -- --release --release --no-default-features

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `sccache` (bin "sccache");
PGO optimized build has failed.
Cargo finished with an error (101)

Rust version: 1.81.0-nightly (6b0f4b5 2024-06-24) sccache version: main branch d5a1787a8582bd99d3f2fc27f4c3191bd8cc76d9

I can't reproduce it with the following command. Do you have more detailed reproduction steps?

rustup install nightly-2024-06-25
rustup +nightly-2024-06-25 component add llvm-tools
cargo +nightly-2024-06-25 pgo instrument build -- --no-default-features
./target/x86_64-unknown-linux-gnu/release/sccache --show-stats
cargo +nightly-2024-06-25 pgo optimize build -- --no-default-features