rust-lld fails with "dyld: Library not loaded: @rpath/libLLVM.dylib" on 1.30.0-nightly (02cb8f2a4 2018-08-29)

[parasyte]

Short description: I am building for a no_std environment using cargo-xbuild, and the embedded lld as my linker of choice. The linker fails to load since it depends on a dynamic library that doesn't actually exist.

$ rustc --version
rustc 1.30.0-nightly (02cb8f2a4 2018-08-29)

$ rustc --print sysroot
/Users/parasyte/.rustup/toolchains/nightly-x86_64-apple-darwin

$ /Users/parasyte/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin/rust-lld --help
dyld: Library not loaded: @rpath/libLLVM.dylib
  Referenced from: /Users/parasyte/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin/rust-lld
  Reason: image not found
Abort trap: 6

$ otool -L /Users/parasyte/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin/rust-lld
/Users/parasyte/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin/rust-lld:
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 7.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)

Rolling back to nightly-2018-08-27-x86_64-apple-darwin works

$ rustup override set nightly-2018-08-27-x86_64-apple-darwin
info: using existing install for 'nightly-2018-08-27-x86_64-apple-darwin'
info: override toolchain for '/Users/parasyte/projects/technek' set to 'nightly-2018-08-27-x86_64-apple-darwin'

  nightly-2018-08-27-x86_64-apple-darwin unchanged - rustc 1.30.0-nightly (721913067 2018-08-26)

$ rustc --version
rustc 1.30.0-nightly (721913067 2018-08-26)

$ rustc --print sysroot
/Users/parasyte/.rustup/toolchains/nightly-2018-08-27-x86_64-apple-darwin

$ /Users/parasyte/.rustup/toolchains/nightly-2018-08-27-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin/rust-lld --help
lld is a generic driver.
Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-lld (WebAssembly) instead

$ otool -L /Users/parasyte/.rustup/toolchains/nightly-2018-08-27-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin/rust-lld
/Users/parasyte/.rustup/toolchains/nightly-2018-08-27-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin/rust-lld:
	/usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version 10.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)

[parasyte]

Searching the source tree for rpath turns up some humorous comments...

rust/src/bootstrap/bin/rustc.rs

Lines 199 to 232 in 4f1e235

	// Dealing with rpath here is a little special, so let's go into some
	// detail. First off, `-rpath` is a linker option on Unix platforms
	// which adds to the runtime dynamic loader path when looking for
	// dynamic libraries. We use this by default on Unix platforms to ensure
	// that our nightlies behave the same on Windows, that is they work out
	// of the box. This can be disabled, of course, but basically that's why
	// we're gated on RUSTC_RPATH here.
	//
	// Ok, so the astute might be wondering "why isn't `-C rpath` used
	// here?" and that is indeed a good question to task. This codegen
	// option is the compiler's current interface to generating an rpath.
	// Unfortunately it doesn't quite suffice for us. The flag currently
	// takes no value as an argument, so the compiler calculates what it
	// should pass to the linker as `-rpath`. This unfortunately is based on
	// the **compile time** directory structure which when building with
	// Cargo will be very different than the runtime directory structure.
	//
	// All that's a really long winded way of saying that if we use
	// `-Crpath` then the executables generated have the wrong rpath of
	// something like `$ORIGIN/deps` when in fact the way we distribute
	// rustc requires the rpath to be `$ORIGIN/../lib`.
	//
	// So, all in all, to set up the correct rpath we pass the linker
	// argument manually via `-C link-args=-Wl,-rpath,...`. Plus isn't it
	// fun to pass a flag to a tool to pass a flag to pass a flag to a tool
	// to change a flag in a binary?
	if env::var("RUSTC_RPATH") == Ok("true".to_string()) {
	let rpath = if target.contains("apple") {
	// Note that we need to take one extra step on macOS to also pass
	// `-Wl,-instal_name,@rpath/...` to get things to work right. To
	// do that we pass a weird flag to the compiler to get it to do
	// so. Note that this is definitely a hack, and we should likely
	// flesh out rpath support more fully in the future.

rust/src/librustc_codegen_llvm/back/linker.rs

Lines 368 to 371 in 9d69e81

	// Note that the `osx_rpath_install_name` option here is a hack
	// purely to support rustbuild right now, we should get a more
	// principled solution at some point to force the compiler to pass
	// the right `-Wl,-install_name` with an `@rpath` in it.

[TejasQ]

I have this issue too!

[GabrielMajeri]

I believe this issue is caused by 3cf6f0d: "Link LLVM tools dynamically...".

It only enables dynamic linking libllvm on Linux and Mac OS, and since no Windows users complained so far I can only assume it's the probable cause.

---

Speaking of which, it might be a good idea to add testing the built LLVM tools / LLD to Rust's CI, to ensure issues like this are caught earlier on.

[japaric]

@alexcrichton @michaelwoerister this may have been caused by #53245.

Also, I don't see this issue on Linux:

$ ldd $(rustc --print sysroot)/lib/rustlib/x86_64-unknown-linux-gnu/bin/rust-lld
        linux-vdso.so.1 (0x00007ffe1cd74000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fab8c935000)
        libLLVM-7.so => /home/japaric/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-7.so (0x00007fab88ecb000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007fab88d46000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fab88d2c000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007fab88b68000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fab8c984000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007fab88b5c000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fab88b57000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fab889c8000)

$ $(rustc --print sysroot)/lib/rustlib/x86_64-unknown-linux-gnu/bin/rust-lld -flavor gnu -version
LLD 7.0.0 (compatible with GNU linkers)

$ rustc -V
rustc 1.30.0-nightly (02cb8f2a4 2018-08-29)

[GabrielMajeri]

@japaric I keep my toolchain updated with rustup, yet it seems I don't have libllvm.so...?

$ ldd $(rustc --print sysroot)/lib/rustlib/x86_64-unknown-linux-gnu/bin/rust-lld
        linux-vdso.so.1 (0x00007ffe30fa4000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007ff1b4c40000)
        libLLVM-7.so => not found
        libm.so.6 => /usr/lib/libm.so.6 (0x00007ff1b4abb000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007ff1b4aa1000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007ff1b48dd000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007ff1b4c9d000)

rustup update output:

info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: checking for self-updates

  nightly-x86_64-unknown-linux-gnu unchanged - rustc 1.30.0-nightly (02cb8f2a4 2018-08-29)

[japaric]

@GabrielMajeri it seems that libLLVM.so is provided by the llvm-tools-preview component which I have installed. Try running rustup component add llvm-tools-preview and see if that fixes LLD.

@alexcrichton @michaelwoerister it sounds like libLLVM.so should always by shipped with the compiler since rust-lld depends on it.

[GabrielMajeri]

@japaric I installed them, but it didn't help. The LLVM tools do seem to work, and I now have the libLLVM.so library in the <rust-sysroot>/lib directory.

However, checking rust-lld's RPATH shows $ORIGIN/../lib,
i.e. it points to <rust-sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/lib/.

Doing

ln -s $(rustc --print sysroot)/lib/libLLVM-7.so $(rustc --print sysroot)/lib/rustlib/x86_64-unknown-linux-gnu/lib/

does (temporarily) fix the problem.

[alexcrichton]

I'm working on a fix for this!

[alexcrichton]

@japaric's diagnosis is spot on and this should be fixed in #53828

To recap, interim fixes include:

• rustup component add llvm-tools-preview --toolchain nightly
• Downgrading to an older nightly temporarily

[TejasQ]

Thanks, @alexcrichton!