Move LTO from the codegen coordinator thread to link_binary

[bjorn3]

Proposal

I've been working on fixing support for -Zno-link/-Zlink-only, which is supposed to also allow for build pipelining for linked crates (rust-lang/rust#141706). It is currently broken because the dependency_formats are calculated too early during the -Zno-link invocation, which forces all dependencies to be already available as fully compiled artifacts during the -Zno-link invocation rather than only during the -Zlink-only invocation. One of the reasons this is done is because LTO, which needs fully compiled artifacts, currently runs during the -Zno-link invocation. As such fixing -Zno-link/-Zlink-only requires moving LTO to the -Zlink-only invocation. In the process I also discovered that -Zdylib-lto is currently broken when combined with -Cprefer-dynamic and --crate-type staticlib,dylib as in that case both crate types disagree about whether or not LTO has to statically link (and include in LTO) or dynamically link (and exclude from LTO) the standard library. Fixing that requires doing LTO once for each crate type (if the dependency_formats are different between crate types at least), which is easier to do when running LTO inside link_binary.

Implementation steps

• Various refactorings and cleanups:
  • Move metadata object generation for dylibs to the linker code  rust#141769
  • Remove two trait methods from cg_ssa rust#141956
  • Various refactors to the LTO handling code rust#143388
  • Various refactors to the LTO handling code (part 2) rust#144062
  • Various refactors to the codegen coordinator code (part 3) rust#144503
  • Couple of minor cleanups rust#145004
  • Move metadata symbol export from exported_non_generic_symbols to exported_symbols rust#145055
  • Move exporting of profiler and sanitizer symbols to the LLVM backend rust#145965
  • Ensure we emit an allocator shim when only some crate types need one rust#145962 (this fixes --crate-type staticlib,dylib -Cprefer-dynamic for the non-LTO case)
  • Misc LTO cleanups rust#146209
  • Remove -Zemit-thin-lto flag rust#152527
  • Remove code for ThinLTO from cg_gcc rust#152566
  • Refactor WriteBackendMethods a bit rust#152801
  • Replace CodegenResults with CompiledModules rust#153184
  • Various LTO cleanups rust#153440
  • Store a PathBuf rather than SerializedModule for cached modules rust#155036
• Separate out the allocator shim module from the rest of the modules. This turns out to be a performance regression as it no longer participates in LTO. It likely makes later steps easier to do though. (Rework how the codegen coordinator code handles the allocator shim rust#145955)
• Split LTO out of the main codegen coordinator event loop into a separate event loop on the same thread rust#147810
• Move LTO to OngoingCodegen::join rust#149209
• Remove tm_factory field from CodegenContext rust#152478
• Support serializing CodegenContext rust#152528
• Simplify ThinLTO handling rust#152670
• Move the LTO handling to link_binary and handle serialization of modules to bitcode when using -Zno-link/-Zlink-only.
• Make the allocator shim participate in LTO again to fix the aforementioned performance regression.
• Run LTO once per crate type when dependency_formats are mismatched to fix -Zdylib-lto with multiple crate types.

Questions

• Is keeping the perf regression of Rework how the codegen coordinator code handles the allocator shim rust#145955 temporarily until the full implementation is finished acceptable? Edit: Turns out it causes some functional regressions. Make the allocator shim participate in LTO again rust#146232 will make the allocator shim part of LTO again.
• Is running LTO once per crate type acceptable? At least when the dependency_formats are mismatched between crate types, I think it is unavoidable.

Mentors or Reviewers

If you have a reviewer or mentor in mind for this work, mention them here. You can put your own name here if you are planning to mentor the work.

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member who is knowledgeable in the area can second by writing @rustbot second or kickoff a team FCP with @rfcbot fcp $RESOLUTION.
  • Refer to Proposals, Approvals and Stabilization docs for when a second is sufficient, or when a full team FCP is required.
• Once an MCP is seconded, the Final Comment Period begins.
  • Final Comment Period lasts for 10 days after all outstanding concerns are solved.
  • Outstanding concerns will block the Final Comment Period from finishing. Once all concerns are resolved, the 10 day countdown is restarted.
  • If no concerns are raised after 10 days since the resolution of the last outstanding concern, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

[rustbot]

Important

This issue is not meant to be used for technical discussion. There is a Zulip stream for that.
Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections can formally be registered here by adding a comment.

@rustbot concern reason-for-concern
<description of the concern>

Concerns can be lifted with:

@rustbot resolve reason-for-concern

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler

[nnethercote]

@rustbot second

[rustbot]

The final comment period is now complete, this major change is now accepted.

As the automated representative, I would like to thank the author for their work and everyone else who contributed to this major change proposal.

If you think this major change shouldn't have been accepted, feel free to remove the major-change-accepted label and reopen this issue.