Toolstate tracking is broken

[RalfJung]

./x.py test --stage 0 src/tools/miri fails currently due to #73272. However, the toolstate is green nevertheless. Looks like somehow toolstate tracking is broken.

Due to this, we currently ship a broken Miri via rustup.

Cc @rust-lang/infra @Mark-Simulacrum @kennytm

[oli-obk]

I can't make heads or tails of the log in https://dev.azure.com/rust-lang/rust/_build/results?buildId=31744&view=logs&j=396dc680-0b3a-5910-2395-0c692e01f85b&t=2ca6b3c4-04bf-58c0-3d82-f09735e43e8e It looks to me like tests weren't run at all?

[RalfJung]

Yeah that's expected, the dist builders only build the tools and do not test them. The tools builder tests them.

Manifest creation (when the final toolchain is added to rustup, as part of auto -> master promotion) reads the toolstate JSON files and omits Miri if its tests failed. (So far, no other tool opted into using that system, as far as I know.)

[RalfJung]

But for some reason, the tools builder doesn't even seem to build Miri?!?

https://dev.azure.com/rust-lang/rust/_build/results?buildId=31744&view=logs&j=e50ab380-190b-535c-8a18-25c988f7577b&t=4a9c8583-1de4-50df-1657-629e71b187fc

[oli-obk]

It runs the miri tests here and shows that they fail. The toolstate processing after that is just not verbose enough to tell us what's up

[RalfJung]

It says

2020-06-11T20:43:41.5188357Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --no-fail-fast src/doc/book src/doc/nomicon src/doc/reference src/doc/rust-by-example src/doc/embedded-book src/doc/edition-guide src/tools/clippy src/tools/rls src/tools/rustfmt src/tools/miri

But that did not even get to Miri. It stopped after another test failure:

2020-06-11T20:43:00.3764050Z test [ui] ui/empty_line_after_outer_attribute.rs ... FAILED

I think that's a clippy test. So what likely happened is that a clippy test failed, but for some reason, instead of aborting CI, that lead to all tools being marked as test-pass...

[RalfJung]

It runs the miri tests here and shows that they fail.

That line says Verifying status of book...? And above some tests fail, but those are not Miri's tests.

[oli-obk]

Oh... right... maybe clippy tests failing cause all other tools not to get tested at all, thus leading to "successful" test results

[RalfJung]

And, notably, clippy tests failing do not block the PR from landing!

[oli-obk]

oh... right 🤦

so... let's do this PR and figure out things in another try?

@bors rollup-

[RalfJung]

@bors rollup-

You did this in an issue, not sure what that is going to accomplish. :D

[oli-obk]

oops. I thought we were on the PR