Unable to build nightly from tarball due to miri (git repository expected)

[semarie]

The latest update of miri (#84050) makes build from tarball to fail. In nigthly, it isn't really a big problem (as tarball is unusual build method), but when it would reach stable (1.53.0), it will hurt downstream distributions.

If I correctly understood the problem, miri tries to extract informations at build time from expected git repository (sha + timestamp). See src/tools/miri/cargo-miri/build.rs. But when the build is from tarball, such git repository doesn't exist, and the build fails with Unable to generate vergen keys!.

The full error when building 1.53.0-nightly (a866124 2021-04-11) is the following:

   Compiling cargo-miri v0.1.0 (/data/semarie/build-rust/build_dir/rustc-nightly-src/src/tools/miri/cargo-miri)
error: failed to run custom build command for `cargo-miri v0.1.0 (/data/semarie/build-rust/build_dir/rustc-nightly-src/src/tools/miri/cargo-miri)`

Caused by:
  process didn't exit successfully: `/data/semarie/build-rust/build_dir/build/x86_64-unknown-openbsd/stage1-tools/release/build/cargo-miri-a3c3912099a18c0b/build-script-build` (exit code: 101)
  --- stdout
  cargo:rerun-if-changed=build.rs

  --- stderr
  thread 'main' panicked at 'Unable to generate vergen keys!: could not find repository from '/data/semarie/build-rust/build_dir/rustc-nightly-src/src/tools/miri/cargo-miri'; class=Repository (6); code=NotFound (-3)

  Stack backtrace:
     0: <unknown>
     1: <unknown>
     2: <unknown>
     3: <unknown>
     4: <unknown>
     5: <unknown>
     6: <unknown>
     7: <unknown>
     8: <unknown>
     9: <unknown>
    10: <unknown>
    11: <unknown>
    12: <unknown>', src/tools/miri/cargo-miri/build.rs:10:24
  stack backtrace:
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
command did not execute successfully: "/data/semarie/build-rust/install_dir/beta/bin/cargo.bin" "build" "--target" "x86_64-unknown-openbsd" "-Zbinary-dep-depinfo" "-j" "4" "--release" "--frozen" "--manifest-path"
"/data/semarie/build-rust/build_dir/rustc-nightly-src/src/tools/miri/cargo-miri/Cargo.toml" "--message-format" "json-render-diagnostics"
expected success, got: exit code: 101
      < ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-openbsd", file: None } }, target: TargetSelection { triple: "x86_64-unknown-openbsd", file: None }, tool: "cargo-miri", path:
"src/tools/miri/cargo-miri", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: [] }
    < CargoMiri { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-openbsd", file: None } }, target: TargetSelection { triple: "x86_64-unknown-openbsd", file: None }, extra_features: [] }
thread 'main' panicked at 'Unable to build cargo miri', src/bootstrap/dist.rs:44:9
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Traceback (most recent call last):
  File "/data/semarie/build-rust/build_dir/rustc-nightly-src/x.py", line 27, in <module>
    bootstrap.main()
  File "/data/semarie/build-rust/build_dir/rustc-nightly-src/src/bootstrap/bootstrap.py", line 1191, in main
    bootstrap(help_triggered)
  File "/data/semarie/build-rust/build_dir/rustc-nightly-src/src/bootstrap/bootstrap.py", line 1177, in bootstrap
    run(args, env=env, verbose=build.verbose)
  File "/data/semarie/build-rust/build_dir/rustc-nightly-src/src/bootstrap/bootstrap.py", line 153, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /data/semarie/build-rust/build_dir/build/bootstrap/debug/bootstrap dist --jobs=4

[semarie]

cc @RalfJung

[RalfJung]

This is strange -- that Miri PR you cite does not even touch the vergen code! Are you sure this worked before that PR? FWIW vergen in Miri was recently updated (not in the PR you cite though), but vergen has been used in Miri already for a long time.

Why are you even trying to build Miri? Miri is a nightly-only tool, and not supposed to be built on beta/stable, so distros should not be affected. In particular, we make no guarantee that the Miri shipped in a beta/stable tarball builds at all against that rustc (and in the past indeed it has occasionally not built). Cc #74709

[semarie]

Yes it was working before: I am running a build daily, and the last successfully build version was 1.53.0-nightly (673d0db 2021-03-23). In previous days, the build failed due to some other error (if I recall correctly some unsync in nightly code between tools and rustc), and it is the first build which fail with this particular error.

Thanks to note I pointed this PR because it was the latest change in miri, but I don't specifically tested the build against it. Sorry to not have mentioned that. Maybe some others commit unbreak the unsync, and the build fail later on miri. I will check that to properly identify the commit causing the problem.

Why are you even trying to build Miri?

I am building -nightly (from tarball) with a extended = true in config.toml, so it built miri.

The purpose of the build is to catch regressions on OpenBSD platform (which is tier-3 on Rust side). So it is basically a buildbot.

[RalfJung]

#83922 changed the version of vergen we used, so that could have caused a change in behavior.

However, Miri toolstate is "green" since 2021-03-29, so there should not have been any build (or test) failures for that entire time.

[RalfJung]

(if I recall correctly some unsync in nightly code between tools and rustc)

We didn't have one of those for Miri since 2021-03-29. So if builds failed in April, even before #83922, something else seems to have gone wrong. (Some other tools could have failed, like rustfmt, but Miri should not.)

[semarie]

Between the two rustc build, src/tools/miri passed from 12dac5c to b9b2af9, and it includes the vergen update

Still investigating

[RalfJung]

In particular it would be good to figure out what caused the Miri build to fail e.g. on 2021-04-01. That was before the vergen update but the toolstate was green so everything should have built.

[semarie]

if the build failed previously due to rustfmt, depending the build order, it could don't have tested miri at all.
(I will change my config.toml to use missing-tools = true and don't fail immediately on tools build error)