Don't bootstrap with rustdoc

[alexcrichton]

Currently the rustdoc executable is bootstrapped in the same manner as rustc itself, meaning that we compile rustdoc itself once per stage. This isn't really necessary though as we only really need to bootstrap the compiler! As a result dev times are slower (stage0/stage1 need to compile rustdoc) and overall CI times are slower (we compile it twice instead of once).

I think we could instead move rustdoc to src/tools and instead compile it in only one stage, the final stage. This'll involve a few changes such as:

• Delete src/driver (I think this is an old vestigate at this point anyway?
• Add src/tools/rustdoc
• Add dependency on src/librustdoc from src/tools/rustdoc
• Make src/tools/rustdoc/src/main.rs a one-line shim to rustdoc::main
• Remove all librustc* dependencies in src/librustdoc/Cargo.toml
• Rework rustbuild rules and dependencies to account for this movement
• Rework assembling the distribution in dist.rs to account moving the rustdoc executable into place.

[alexcrichton]

cc @Mark-Simulacrum would you be interested in tackling this?

cc @aidanhs as well

[nikomatsakis]

cc @GuillaumeGomez -- this would help you with your html diff problems

[Mark-Simulacrum]

Sure, we can do this. However, is it worth the investment today? AIUI, @steveklabnik is working on a rewrite that won't even be in-tree perhaps (or just a submodule). Should we wait for that?

[alexcrichton]

@Mark-Simulacrum I don't think there's any need to block, all of this work will need to happen anyway for a rewrite regardless.

[Mark-Simulacrum]

True. I'd prefer not to implement this myself on the current rustbuild, but I'd be happy to do it once we make a decision on the rewrite.

Remove all librustc* dependencies in src/librustdoc/Cargo.toml

I'm not quite sure if this is what you meant to say -- this seems like a fairly large task; is the intent that we'd instead link against them via extern crate and the unstable feature gate rustc_private?

[alexcrichton]

Oh sorry I mean just literally delete these lines and nothing more. All those crates will be in the sysroot, so Cargo doesn't need to build them.

[steveklabnik]

I'm generally in support of this idea.

@Mark-Simulacrum it is still very early days with that project; I wouldn't be taking anything about it into consideration when making decisions generally.

[ollie27]

As long as this also fixes #38318 then 👍.

[GuillaumeGomez]

I approve!

[aidanhs]

May (or may not) also have an effect on the proc macro issues in #41991.

[QuietMisdreavus]

What would this change affect in terms of edit-compile-debug cycles on rustdoc or std documentation itself? I admit i don't know the placement of "tools" in terms of the build order, but it sounds like just working on rustdoc would have a longer single-setup time (since it would be part of "the final stage") but much shorter turnaround times (since you don't have to rebuild the world to get a working rustdoc)? How would this interact with the trick of using rustup to link a local toolchain that points to your local rustdoc? Would rustdoc now only appear in the stage2/bin folder? Would it no longer be part of that output?

As for regenerating std docs, how would this interact with that? When I'm working on rustdoc i like to test my changes on the std documentation so i don't have to write a purpose-built test crate for it. Putting rustdoc outside the regular staging could require some new considerations in regards to how it wants to rebuild dependencies of the crate you're documenting.

I don't see these as pure blockers to getting this done, just things that may need to be taken into account as part of shaking out the rustbuild dependency graph and the ./x.py doc command. Since this could also solve my own pet rustbuild issue (#42686) i'm mainly on board.