rustc_metadata encodes all the crate information into an in-memory buffer in EncodedMetadata. This in-memory buffer is then passed around to be written to disk, as an rmeta file in rustc_interface::encode_and_write_metadata and/or in an rlib/dylib by codegen. Codegen wraps metadata into an object file using the object crate to be nice to linkers that will manipulate the rlib/dylib.
Keeping metadata in-memory increases the amount of memory used by rustc significantly. We want to investigate saving this memory by writing metadata to disk early, reading it back if required.
Instructions:
- Move
rustc_codegen_ssa::back::link::emit_metadata and rustc_codegen_ssa::METADATA_FILENAME to new module rustc_metadata::fs.
- Move
rustc_interface::passes::encode_and_write_metadata and rustc_metadata::util::non_durable_rename to rustc_metadata::fs.
- Write metadata into the temporary file in
encode_and_write_metadata even if !need_metadata_file.
- Use a
rustc_serialize::opaque::FileEncoder in rustc_metadata::rmeta::encoder instead of an opaque::Encoder, it should encode directly to the temporary file, and re-read this temporary file to build the EncodedMetadata.
- Change
EncodedMetadata to hold a Mmap of the on-disk metadata instead of a Vec. This Mmap can be of the temporary file, or of the proper renamed output file. In the former case, EncodedMetadata should carry the MaybeTempDir to avoid deleting the temporary directory while accessing the Mmap.
Extra: the current implementations of create_rmeta_file and create_compressed_metadata_file in rustc_codegen_ssa::back::metadata buffer everything to a vector before writing it all at once. This could be refactored to write directly to the file using a BufWriter.
Please reach out on Zulip if you have any questions. Preferably on a public stream so experts on codegen can weigh in when necessary.
rustc_metadataencodes all the crate information into an in-memory buffer inEncodedMetadata. This in-memory buffer is then passed around to be written to disk, as anrmetafile inrustc_interface::encode_and_write_metadataand/or in anrlib/dylibby codegen. Codegen wraps metadata into an object file using theobjectcrate to be nice to linkers that will manipulate therlib/dylib.Keeping
metadatain-memory increases the amount of memory used by rustc significantly. We want to investigate saving this memory by writing metadata to disk early, reading it back if required.Instructions:
rustc_codegen_ssa::back::link::emit_metadataandrustc_codegen_ssa::METADATA_FILENAMEto new modulerustc_metadata::fs.rustc_interface::passes::encode_and_write_metadataandrustc_metadata::util::non_durable_renametorustc_metadata::fs.encode_and_write_metadataeven if!need_metadata_file.rustc_serialize::opaque::FileEncoderinrustc_metadata::rmeta::encoderinstead of anopaque::Encoder, it should encode directly to the temporary file, and re-read this temporary file to build theEncodedMetadata.EncodedMetadatato hold aMmapof the on-disk metadata instead of aVec. ThisMmapcan be of the temporary file, or of the proper renamed output file. In the former case,EncodedMetadatashould carry theMaybeTempDirto avoid deleting the temporary directory while accessing theMmap.Extra: the current implementations of
create_rmeta_fileandcreate_compressed_metadata_fileinrustc_codegen_ssa::back::metadatabuffer everything to a vector before writing it all at once. This could be refactored to write directly to the file using aBufWriter.Please reach out on Zulip if you have any questions. Preferably on a public stream so experts on codegen can weigh in when necessary.