Tracking issue for stabilizing "pipelined compilation"

[alexcrichton]

This issue is intended to be an overall tracking issue for stabilizing the recently implemented feature of "pipelined compilation". This tracking issue spans both Cargo and rustc, as support currently lives in both. It also builds on top of a number of other tracking issues, since they're unstable features used to implement pipelined compilation!

For a bit of background, pipelined compilation has an initial set of notes about its implementation and initially had these tracking issues:

• Implement "pipelined" rustc compilation cargo#6660
• Implement "pipelined" rustc compilation (compiler side) #58465

Currently today everything is implemented and available in nightly rustc. There is a thread on internals which is tasked with gaining user experience about pipelining and see the performance impact. A collation of these measurements is quite promising and I feel personally is convincing enough to push on stabilizing all the sub-features!

Known blockers for stabilization:

• -Z emit-artifact-notifications - this is how Cargo learns the moment metadata is ready.
• --json-rendered - this is how Cargo continues to provide pretty error messages.
• There is no way to combine --message-format=short in Cargo with pipelining. The compiler currently has --error-format=json|short|human, but for this feature in Cargo we'll need something like --error-format=short-json or something like that. Basically some way to request that the rendered field in the JSON diagnostic is a "short message", not the long one.
• Cargo pipelining is known to break with sccache, this should be investigated and fixed one way or another. (fixed in Cache rmeta files whenever metadata is present in emit. mozilla/sccache#441)
• Pipelining fails with the RLS due to multiple --error-format flags (fixed in Update RLS #62176)

Possible future work:

• Should we game out what a "fully pipelined" story might look like? What if rustc blocked waiting for Cargo to tell it to proceed into the linking phase?

[Centril]

Reading the various issues and PRs, I got the impression that metadata is generated at earliest post static analysis, is that correct?

[nnethercote]

Yes: metadata is now generated right at the start of code generation, immediately after all the checking (typeck, borrowck, etc.) finish. This is the best place for the initial pipelining implementation.

One might argue that metadata generation should be moved earlier. That would help with successful builds, but slow down builds that trigger compile errors. Given that a lot of Rust compiler invocations end in compile errors during the normal development cycle, slowing down such invocations would be bad.

Another possibility is to generate metadata in parallel with checking. That might give the best of both worlds. But that's a bigger change.

[Centril]

One might argue that metadata generation should be moved earlier.

No no... ;) At the start of code-gen sounds good to me and we should be cautious with moving it too much earlier as that can mess with language design.

[oli-obk]

There is no way to combine --message-format=short in Cargo with pipelining

This is trivial to implement with --json-rendered. I can mentor this in #60987

[jsgf]

It might be useful to have multiple rendered forms in the json output.

[alexcrichton]

I've proposed stabilization of --json-rendered as well as -Z emit-artifact-notifications which, if stabilized, can allow Cargo to switch over to using these by default and stabilizing pipelined compilation.