make check-stage1 is broken for run-make/bootstrap-from-c-with-green

[pnkfelix]

There is something funky going on when I currently try to use make check-stage1 on a clean checkout of rust (commit e01e78f)

https://gist.github.com/pnkfelix/11260219

Summary of how the story in that gist ends:

% rm -rf x86_64-apple-darwin/stage1/lib/stamp.green
% rm -rf x86_64-apple-darwin/stage1/lib/stamp.rustuv
% rm -rf x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green/libboot-c9bec0e5-0.1.dylib
% remake --trace check-stage1
...
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
/Users/fklock/Dev/Mozilla/rust.git/objdir-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green/main
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

#0 all at /Users/fklock/Dev/Mozilla/rust.git/src/test/run-make/bootstrap-from-c-with-green/Makefile:3
remake[1]: Leaving directory `/Users/fklock/Dev/Mozilla/rust.git/src/test/run-make/bootstrap-from-c-with-green'
Command-line invocation:
"/Users/fklock/opt/remake/bin/remake -C /Users/fklock/Dev/Mozilla/rust.git/src/test/run-make/bootstrap-from-c-with-green/"

------ stderr ---------------------------------------------
dyld: Symbol not found: __ZN10event_loop20h1197db882d44291cDOf9v0.11.preE
Referenced from: /Users/fklock/Dev/Mozilla/rust.git/objdir-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green/libboot-c9bec0e5-0.1.dylib
Expected in: /Users/fklock/Dev/Mozilla/rust.git/objdir-dbgopt/x86_64-apple-darwin/stage1/lib/librustuv-30cdf6fe-0.11-pre.dylib
in /Users/fklock/Dev/Mozilla/rust.git/objdir-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green/libboot-c9bec0e5-0.1.dylib
Makefile:3: *** [all] Trace/BPT trap: 5

------ ---------------------------------------------

/Users/fklock/Dev/Mozilla/rust.git/mk/tests.mk:922: *** [x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green-1-T-x86_64-apple-darwin-H-x86_64-apple-darwin.ok] Error 2

#0 x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green-1-T-x86_64-apple-darwin-H-x86_64-apple-darwin.ok at /Users/fklock/Dev/Mozilla/rust.git/mk/tests.mk:922
#1 tmp/check-stage1-T-x86_64-apple-darwin-H-x86_64-apple-darwin-rmake.ok at /Users/fklock/Dev/Mozilla/rust.git/mk/tests.mk:923
#2 check-stage1-T-x86_64-apple-darwin-H-x86_64-apple-darwin-rmake-exec at /Users/fklock/Dev/Mozilla/rust.git/mk/tests.mk:923
#3 check-stage1-T-x86_64-apple-darwin-H-x86_64-apple-darwin-exec at /Users/fklock/Dev/Mozilla/rust.git/mk/tests.mk:332
#4 check-stage1-T-x86_64-apple-darwin-H-x86_64-apple-darwin at /Users/fklock/Dev/Mozilla/rust.git/mk/tests.mk:807
#5 check-stage1-H-x86_64-apple-darwin at /Users/fklock/Dev/Mozilla/rust.git/mk/tests.mk:847
#6 check-stage1 at /Users/fklock/Dev/Mozilla/rust.git/mk/tests.mk:828
Command-line invocation:
"/Users/fklock/opt/remake/bin/remake --trace check-stage1"
%

@alexcrichton says he often just runs make check-stage1-rpass, so that is sort of a workaround for the near term.

[pnkfelix]

see also, further irc discussion/debugging between myself and alexcrichton here: https://botbot.me/mozilla/rust-internals/msg/13776568/

[pnkfelix]

I resolved where the rpath into my $HOME/opt/rust-dbg/lib/rustlib/x86_64-apple-darwin/lib was coming from: the config.mk file in my build tree has this setting

CFG_LIBDIR           := /Users/fklock/opt/rust-dbg/lib

(I am still a bit confused because this particular build directory almost always doesn't use that particular --prefix) (update: no, it was correct, I shouldn't have been confused at all. I forgot I keep the non-optimized build in rust-dbg-nopt)

Anyway, so this is getting installed as the fallback rpath, as indicated here in a verbose log:

relative rpaths:
    @loader_path/../../../stage1/lib/rustlib/x86_64-apple-darwin/lib
    @loader_path/../../../stage1/lib/rustlib/x86_64-apple-darwin/lib
    @loader_path/../../../stage1/lib/rustlib/x86_64-apple-darwin/lib
    @loader_path/../../../stage1/lib/rustlib/x86_64-apple-darwin/lib
    @loader_path/../../../stage1/lib/rustlib/x86_64-apple-darwin/lib
fallback rpaths:
    /Users/fklock/opt/rust-dbg/lib/rustlib/x86_64-apple-darwin/lib

[pnkfelix]

Looking at the code in librustc/back/rpath.rs, we hard-code a reference to the install prefix into rustc itself.

I don't have a problem with that, but we probably should have a compiler flag to either remove the usage of that fallback rpath, or override the path that rustc uses for the fallback. Then the Makefile could actually run rustc but have it referencing its own build tree for the fallback by using the overriding option.

[pnkfelix]

Okay, so I hacked up a patch to add a --install-prefix option to rustc that overrides the embedded one in the librustc library.

After doing that and using it in the run-make/bootstrap-from-c-with-green/Makefile, I still hit the problem.

I did some more digging and determined that there is some funky effect being introduced by the setting of the DYLD_LIBRARY_PATH environment variable. (I added a call to env to the start of the Makefile rule to print out the environment before running the recipe, that helped a lot).

From the settings of the DYLD_LIBRARY_PATH and LD_LIBRARY_PATH environment variables, I determined that you need at least one of these environment variables to be set if you want this test to run at all; if both are unset, you get:

% /Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green/main
dyld: Library not loaded: @rpath/libboot-c9bec0e5-0.1.dylib
  Referenced from: /Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green/main
  Reason: image not found
Trace/BPT trap: 5
% 

So I transcribed the settings that the Makefiles are current establishing for the two environment variables at the time that this recipe is run (by the aforementioned env printout), and found the following distinct behaviors.

% DYLD_LIBRARY_PATH=/Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green::/Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/stage1/lib LD_LIBRARY_PATH=/Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green: /Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green/main
dyld: Symbol not found: __ZN10event_loop20h1197db882d44291cDOf9v0.11.preE
  Referenced from: /Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green/libboot-c9bec0e5-0.1.dylib
  Expected in: /Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/stage1/lib/librustuv-30cdf6fe-0.11-pre.dylib
 in /Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green/libboot-c9bec0e5-0.1.dylib
Trace/BPT trap: 5
% LD_LIBRARY_PATH=/Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green: /Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/test/run-make/bootstrap-from-c-with-green/main
hello
% 

It seems like the DYLD_LIBRARY_PATH is causing my problem here at this point. In particular the entry for /Users/fklock/Dev/Mozilla/rust.git/objdir-check1-dbgopt/x86_64-apple-darwin/stage1/lib is causing problems. Should that be pointing instead to x86_64-apple-darwin/stage1/lib/rustlib/x86_64-apple-darwin/lib ?

[pnkfelix]

attempting to fix, work in progress, see PR #13753

[pnkfelix]

(closed #13753 in favor of PR #14000)

[pnkfelix]

part of #8058