Include self-contained -L in native-static-libs

[alyssais]

View all comments

(I suggest reviewing with whitespace ignored.)

rustc uses heuristics to determine whether the self-contained path should be on the linker search path. It's important to get these right, because if self-contained is improperly missing from the search path, linking will fail (as seen here), but if it's improperly present, a broken executable can be produced due to e.g. linking two different libcs (as seen here and here). All of this works reasonably well when rustc is doing the linking, but when linking is done externally, there's a problem: there's no way for a build tool to know whether rustc would include the self-contained path or not. Without a way to find this out from rustc, the only way for a build tool to correctly link with a staticlib produced by rustc is to exactly reimplement rustc's heuristics. Not ideal.

The existing rustc --print native-static-libs option exists to enable build tools to find out what linker flags should be used when linking a rustc-produced staticlib. Without including the argument to add the self-contained directory to the search path when necessary, though, it doesn't meet its intended purpose — build tools relying on native-static-libs will still sometimes do the wrong thing. Fixing this will make it possible for build tools to link the correct self-contained libraries when necessary, without each build tool having to match rustc's heuristics itself. This will enable fixing real problems encountered with Meson, where either self-contained libraries were not found, or self-contained libraries were incorrectly linked.

[rustbot]

r? @TaKO8Ki

rustbot has assigned @TaKO8Ki.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: codegen, compiler
• codegen, compiler expanded to 75 candidates
• Random selection from 18 candidates

[bjorn3]

If you are building a staticlib, I wouldn't expect it to contain self-contained artifacts. Whatever executable/dylib you link it into already has to either provide those artifacts for itself or get them from the system even if the rust staticlib wasn't linked in. The self-contained artifacts consists of a linker, libraries and object files (think crt*.o) a regular C toolchain for the target would ship. The self-contained artifacts allow you to link without needing a C toolchain for the target. For Meson I would expect a C toolchain to already be available for the C code that gets link too. Or is the use case for this building pure Rust applications with Meson?

[alyssais]

The staticlib in this case does not contain/bundle artifacts, but it might expect to link with e.g. rustc's self-contained libunwind. If I build a staticlib with rustc's +crt-static x86_64-unknown-linux-musl target, it's expected to be ultimately linked with libunwind even though the musl C toolchain probably doesn't provide that by default. In a mixed Rust+C program, if rustc does the final link, it will link with the self-contained libunwind. Presumably, since the libunwind requirement is coming from the Rust part, that same libunwind (from rustc's self-contained directory) should be used when the C toolchain happens to be doing the final link.

[alyssais]

(Building pure Rust applications with Meson works fine, because then rustc does the final link.)

[bjorn3]

The standard library needs either libunwind or libgcc (libgcc_eh.a or libgcc_s.so) as unwinder. It is important that only one unwinder ends up getting linked into the entire program to ensure dynamic registration of unwind tables will always be done against the same unwinder as the one that does the actual unwinding. If your C toolchain comes with libgcc instead of libunwind you should probably link against that instead. If your C toolchain provides neither, is it not capable of compiling C++ code?

[alyssais]

I believe it provides libgcc_eh.a, but per bootstrap.example.toml, rustc will only ever use libunwind for musl +crt-static, not libgcc_eh.

rust/bootstrap.example.toml

Lines 1075 to 1077 in f0b782b

	# the behavior is depend on the libc. On musl target, 'no' and 'in-tree' both
	# means static link to the in-tree build of llvm libunwind, and 'system' means
	# static link to `libunwind.a` provided by system. Due to the limitation of glibc,

[alyssais]

(And if I am supposed to link against libgcc_eh anyway, that means I can't trust the output of native-static-libs, which includes -lunwind, even though it may not exist outside of self-contained.)

[TaKO8Ki]

r? compiler

[JohnTitor]

@rustbot reroll

[chenyukang]

@rustbot reroll

[adwinwhite]

@rustbot reroll

[mu001999]

@rustbot reroll

[saethlin]

r? me
cc @bjorn3 you're the better reviewer but I'm willing to give this a go to move it forward

[mati865]

(And if I am supposed to link against libgcc_eh anyway, that means I can't trust the output of native-static-libs, which includes -lunwind, even though it may not exist outside of self-contained.)

Wouldn't the solution here be to link libunwind only if self-contained mode is active?
+crt-static by itself shouldn't dictate the unwind library.

[bjorn3]

Wouldn't the solution here be to link libunwind only if self-contained mode is active?

If we didn't explicitly link libunwind, no unwinder would be linked at all.

[mati865]

If we are not in self-contained mode the user has to supply Musl objects and libs, either by linking with {gcc,clang}-musl or {gcc,musl} on Musl based distro. In that case, linkable libgcc should be present in the system.
For self-contained mode, we always link libunwind as that's what we provide.

I forgot that we handle unwind library linking via library/unwind/src/lib.rs which complicates this approach a lot.

[bjorn3]

If we are not in self-contained mode the user has to supply Musl objects and libs, either by linking with {gcc,clang}-musl or {gcc,musl} on Musl based distro. In that case, linkable libgcc should be present in the system.

If the user statically links the executable, linking libgcc is not enough. libgcc_eh also needs to be linked. And even if that may be the default of gcc and clang, we explicitly disable linking to default libraries on most targets.

[mati865]

If the user statically links the executable, linking libgcc is not enough. libgcc_eh also needs to be linked.

My bad there, I thought of libgcc unwind library (libgcc{_eh,_s}) when writing libgcc.

And even if that may be the default of gcc and clang, we explicitly disable linking to default libraries on most targets.

Yeah, but then we explicitly link the unwind library.

So, ideally for musl we'd link:

	+crt-static	-crt-static
link-self-contained=on	-Lself-contained/ -lunwind	-Lself-contained/ -lunwind
link-self-contained=off	-lgcc_eh	-lgcc_s

[bjorn3]

We don't know if the host toolchain uses libgcc or compiler-rt, right? In the latter case libunwind would still be used I think.

[mati865]

We don't know if the host toolchain uses libgcc or compiler-rt, right? In the latter case libunwind would still be used I think.

Builtins and unwinder are two distinct things. First one is handled by --rtlib in Clang (GCC supports only libgcc), second one is handled by --unwindlib in Clang (again, GCC supports only libgcc_). But yeah, we don't know what the host toolchain uses, although libgcc_ is de facto dominant standard on Linux.

Perhaps we could use target-feature akin to system-llvm-libunwind std feature to control the unwinding library...

[bjorn3]

Builtins and unwinder are two distinct things.

libgcc_s contains both. So you can't use libunwind when dynamically linking libgcc.

[mati865]

libgcc_s contains both.

Well, yes it's a linker script that links libgcc_s.so.1 and libgcc.a. However, you can still use compiler-rt builtins by putting them first in the linker invocation. Different builtins don't conflict each other.

So you can't use libunwind when dynamically linking libgcc.

I don't understand. You cannot use libuwind when linking either libgcc unwinder, be it libgcc_s or libgcc_eh. Still you can use LLVM's libunwind.a together with libgcc.a.

EDIT: fixed typo.

[bjorn3]

libgcc_s.so.1 for me does export part of the symbols of libgcc.a, so that is why I assumed they would conflict.

So you can't use libunwind when dynamically linking libgcc.

I don't understand. You cannot use libuwind when linker either libgcc unwinder, be it libgcc_s or libgcc_eh. Still you can use LLVM's libunwind.a together with libgcc.a.

If you statically link libgcc.a you have the option to use libunwind instead of libgcc_eh. If you dynamically link libgcc_s.so (which I assumed would be providing the compiler builtins instead of libgcc.a), you don't have the option to use libunwind.

[mati865]

I think I see where the confusion is coming from. You could say that symbols from libgcc.a (or compiler-rt) are always linked statically, no matter what you pass to the linker because there is no .dynsym there.

So, to rephrase it. You (almost) always need the unwinder (shared: libgcc_s.so, libunwind.so or static: libgcc_eh.a, libunwind.a) and the builtins (libgcc.a or libclang_rt.builtins-<arch>.a). You can use combine them as you want as long as only a single unwinder is used. Although, there are edge cases like libgcc_s.

If we had target modifier to control the unwinder, we could let the user decide which one to use (unless I'm missing something). Rather than forcing one whether linking Musl statically or dynamically. But that's a much more complex solution.

Coming back to the solution proposed by this PR, native-static-libs predates link-self-contained a lot: #43067 So, it's easy to say why it doesn't include search dirs for it.
The idea isn't bad, but my concern is whether we are fine with exposing our self-contained contents (perhaps we are?).

Regarding https://bugs.gentoo.org/967728 I suppose self-contained makes little sense when building Rust as a system package. We need to include libunwind.a because of how we always link it for Musl when linking statically (again that problem would be gone if we could change unwinder library), but crt objects and Musl library should be rather linked from the system.

I think mesonbuild/meson#15216 is an unfortunate workaround for our shortcomings in this area, because AFAIK the files layout is an implementation detail and therefore not guaranteed.

[alyssais]

So, ideally for musl we'd link:
+crt-static -crt-static
link-self-contained=on -Lself-contained/ -lunwind -Lself-contained/ -lunwind
link-self-contained=off -lgcc_eh -lgcc_s

Alas it is more complicated than this! My toolchain does not have libgcc_eh.a, because libgcc is built with --disable-shared. In this case, there's no need for separate static/non-static unwinding implementations, so unwinding just gets built into libgcc.a. This means the right thing for rustc to do in this case would be to link -lgcc.

It's not clear to me how this should be accomodated in rustc. Whether to link libgcc_eh.a or libgcc.a depends on how the toolchain was built. GCC knows which to link, but that doesn't help us given we tell it not to include default libraries. We could try detecting whether libgcc_eh.a exists before invoking the linker? Maybe we could do something with a linker script? I'm not sure there's a better solution.

Coming back to the solution proposed by this PR, native-static-libs predates link-self-contained a lot: #43067 So, it's easy to say why it doesn't include search dirs for it. The idea isn't bad, but my concern is whether we are fine with exposing our self-contained contents (perhaps we are?).

Regarding https://bugs.gentoo.org/967728 I suppose self-contained makes little sense when building Rust as a system package. We need to include libunwind.a because of how we always link it for Musl when linking statically (again that problem would be gone if we could change unwinder library), but crt objects and Musl library should be rather linked from the system.

I think mesonbuild/meson#15216 is an unfortunate workaround for our shortcomings in this area, because AFAIK the files layout is an implementation detail and therefore not guaranteed.

It certainly seems that this isn't the right fix for my problem, since it turns out that I shouldn't even be linking libunwind! I plan to propose a revert of my Meson PR, since it's clearly not right and is causing people problems.

[bjorn3]

Maybe we should stop using -nodefaultlibs when -Clink-self-contained=-libc,-unwind? And then only explicitly link against system libraries using #[link] when -Clink-self-contained=+libc,+unwind? This would also allow getting rid of a fair bit of linker arguments from target specs for all targets that don't support self contained linking.

[alyssais]

That would certainly make me happy!

[mati865]

Alas it is more complicated than this! My toolchain does not have libgcc_eh.a, because libgcc is built with --disable-shared. In this case, there's no need for separate static/non-static unwinding implementations, so unwinding just gets built into libgcc.a. This means the right thing for rustc to do in this case would be to link -lgcc.

Gotta love when decisions made 20 years ago still haunt us today...
I don't know if this is an excuse for us, but Clang also didn't consider the case where libgcc.a is an amalgamation: https://github.com/llvm/llvm-project/blob/3e8f824e6e37d12c5c741c73a96dee7f6f0e4eb1/clang/lib/Driver/ToolChains/CommonArgs.cpp#L2516
Unless GCC still creates empty libgcc_eh.a with that option, in that case we and Clang would be fine by linking -lgcc_eh -lgcc.

It certainly seems that this isn't the right fix for my problem, since it turns out that I shouldn't even be linking libunwind!

It's fine to link libunwind if you know that's the only unwinder. But indeed you often don't know it upfront.

Maybe we should stop using -nodefaultlibs when -Clink-self-contained=-libc,-unwind? And then only explicitly link against system libraries using #[link] when -Clink-self-contained=+libc,+unwind? This would also allow getting rid of a fair bit of linker arguments from target specs for all targets that don't support self contained linking.

Is it just me, or does this start drifting into a total makeover of how we handle "system" libs linking?
Currently we link unwind libs via either:

• libc out-of-tree crate: https://github.com/rust-lang/libc/blob/75b2850150d0c50fa012ce965b7466d46da8e4e3/src/unix/mod.rs#L532
• libstd
• target spec (Windows and L4Re)
  Libc is handled by either said libc crate, or target spec (Windows only I think).

I'm afraid -Clink-self-contained=-libc,-unwind would be annoying to implement. While there is -nolibc to disable C library linkage you cannot disable unwinder unless you use -nodefaultlibs to disable almost everything, or -nostdlib to get rid of everything.
Unless I misunderstood, and your idea is to not use -nodefaultlibs by default. Instead, use it only for self-contained targets, when self-contained mode is enabled. That would probably make life easier for almost everyone. Still not sure how you'd implement the granularity for cases like -Clink-self-contained=-libc,+unwind or -Clink-self-contained=+libc,-unwind. Guess it'd still need to rely on -nodefaultlibs` and whole that mess for finding the right libraries. Or don't allow the granularity: either link entire self-contained or nothing from it.

---

Frankly, I believe the least hairy way for musl hosts (and some others like MSYS2) there is right now, is to patch Rust compiler to just get rid of self-contained.

[saethlin]

r? bjorn3

I think you have a better understanding of what's happening here than I do.