Tracking issue for rustbuild, the alternate build system for rustc

[alexcrichton]

I'm opening this up as a tracking issue for all work that needs to happen to get rustbuild, our new build system, turned on by default. The end goal here is to completely jettison our thousands of lines of makefiles and configure scripts for tons and tons of Rust code! Initial support has landed in #31123, but we've still got quite aways to go!

Here's a list of items left that need to be implemented, and help is always appreciated when tackling these! If you want any clarifications or any help on any of these, feel free to reach out to me (acrichto) on IRC.

Documentation

work in progress

• nomicon - rustbuild: Add rustbook/standalone doc support #31713
• rustbook - rustbuild: Add rustbook/standalone doc support #31713
• error index - Add a link validator to rustbuild #32133
• crate docs - Add a link validator to rustbuild #32133
• compiler docs - Add a link validator to rustbuild #32133
• standalone docs - rustbuild: Add rustbook/standalone doc support #31713
• style guide - rustbuild: Add rustbook/standalone doc support #31713

Testing

• crate tests - rustbuild: Add support for crate tests + doctests #33282
• crate doctests - rustbuild: Add support for crate tests + doctests #33282
• rustbook/nomicon tests - rustbuild: Run all markdown documentation tests #33092
• standalone documentation tests - rustbuild: Run all markdown documentation tests #33092
• error-index tests - rustbuild: Run all markdown documentation tests #33092
• run-pass - rustbuild: Add support for compiletest test suites #32755
• run-fail - rustbuild: Add support for compiletest test suites #32755
• compile-fail - rustbuild: Add support for compiletest test suites #32755
• parse-fail - rustbuild: Add support for compiletest test suites #32755
• codegen - rustbuild: Add support for compiletest test suites #32755
• debuginfo - rustbuild: Add support for compiletest test suites #32755
• codegen-units - rustbuild: Add support for compiletest test suites #32755
• pretty - rustbuild: Add support for compiletest test suites #32755
• run-pass-fulldeps - rustbuild: Add support for compiletest test suites #32755
• compile-fail-fulldeps - rustbuild: Add support for compiletest test suites #32755
• run-pass-valgrind - rustbuild: Add support for compiletest test suites #32755
• rustdoc - rustbuild: Add support for compiletest test suites #32755
• run-make -- move maketest.py to rust as well - test: Move run-make tests into compiletest #33093
• Android testing - ship things to the emulator - rustbuild: Implement testing for Android #34567
• multi-platform testing, handle multiple hosts and multiple targets - rustbuild: Implement testing for Android #34567

Porting

These should all be verified to work and then ideally we'd add a builder verifying they continue to work!

• Android - rustbuild: Implement testing for Android #34567
• iOS
• various cross compiled linuxes
• musl-x86_64

configure script

• Migrate compiler version detection to Rust (maybe optional?)
• Ensure all tool detection is moved into Rust

Misc

• Add build automation to pass --enable-rustbuild, but not run tests - Add auto builders for rustbuild build system rust-lang-deprecated/rust-buildbot#61
• Verify that multi-host builds work - Enable bootstrapping non-build-machine targets #31884
• Verify sha256 of downloaded nightly rustc and nightly cargo - rustbuild: Verify sha256 of downloaded tarballs #32926
• Migrate tidy to rust - rustbuild: Migrate tidy checks to Rust #32590
  • binary check WIP
• Migrate licenseck to rust - rustbuild: Migrate tidy checks to Rust #32590
  • work in progress
• Migrate featureck to rust - rustbuild: Migrate tidy checks to Rust #32590
  • work in progress
• Migrate errock to rust - rustbuild: Migrate tidy checks to Rust #32590
  • work in progress
• make clean - Add make clean support to rustbuild #31591

make install + make dist

These are pretty huge portions of the make files, they may take quite some time to migrate over, and it's probably fine to keep them around for now. @brson's plans with multirust-rs and reusable library components may become relevant here as well.

This may be subdivided into a bunch of smaller tasks over time!

• make dist - rustbuild: Implement make dist #32237

Far future (not required)

• Support src/bootstrap/bootstrap.py being called directly (e.g. on MSVC where make isn't available)
• verify gpg signatures of downloaded nightlies - Validate gpg signatures of downloaded nightlies #34598
• Support building rustc for architectures that can themselves not be run. For example a x86_64 host should be able to build an arm compiler (e.g. a full installation tarball) - Enable bootstrapping non-build-machine targets #31884
• Move building LLVM out of rustbuild (not actually sure if this is possible)
• Move building compiler-rt out of rustbuild (eases cross-compiling libstd) - Move compiler-rt build into a crate dependency of libcore #34400
• migrate htmldockck to Rust

[alexcrichton]

Also as the most bike-sheddy thing, I'm not a huge fan of the term "rustbuild", so I'd also be fine calling this something cooler like "rust laser build"

[steveklabnik]

This is a very simple thing, but I hit control-C while it was downloading stage0 and because the file wasn't valid, it just bailed out and I had to remove it manually. I'm not sure if deleting it and retrying is the right thing, or if this is just going to be a papercut generally.

[alexcrichton]

Sounds like a bug! (added a checkbox)

[nodakai]

By going from Makefile to build.rs, are we not throwing away the notion of dependency graph?

Please enable resuming on curl if possible, for people with narrowband.

[alexcrichton]

No, I would expect any build system to take dependencies into account. It's an implicit assumption of mine that any sort of dependency tracking done by the current build system must be done by rustbuild as well.

[steveklabnik]

Regarding the documentation features... maybe we should try to land rust-lang/cargo#2256 and then they could just be their own crates?

[alexcrichton]

I suspect that may not end up helping too much unfortunately, most of our docs are rustbook-based rather than rustdoc-based.

[steveklabnik]

Oh duh. I should, uh, know that 😅

The new book is going to use mdBook, and I think it should just be a drop-in replacement. Maybe it's time to kill rustbook for real?

[alexcrichton]

It probably doesn't matter either way in terms of dependency management and instrumentation of the compiler building mdbook/rustbook, so I suspect switching over wouldn't change things too much really

[steveklabnik]

@azerupi is okay with mdbook becoming a dependency https://github.com/azerupi/mdBook/issues/105#issuecomment-183518235

I will poke at that docs branch sometime next week.

[japaric]

I'm going to look into building compiler-rt as a crate using cargo instead of having the bootstrap binary build it. This should get us closer to being able to (cross) compile the std crate with cargo, i.e. just cd src/libstd && cargo build.

[alexcrichton]

(added some bullet points to that effect)