Re-organize build-dir by package + hash, rather than artifact type

Call of Testing

Also include anyone is following this issue and missed it, we put out a call of testing of the new v2 build-dir layout in a blog post here
See the blog post for details and testing instructions

Implementation:

Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#build-dir-new-layout

Resolves

Shadowing binaries on Windows (Workspace binary artifacts are added to target/debug/deps on Windows #7919)
Collisions between intermediate build artifacts (Output filename collision with dylib on windows release mode #8794, output filename collision with panic = "abort" #16673)
Inconsistent use of -Cextra-filename (Consider changing hashing behavior for executables. #8332)
Slow performance when intermediate artifacts accumulate (Presence of old builds can dramatically raise incremental build time, via deps dir size growth #16665)

Past efforts

Unify build directories. #6668

Overview

.fingerprint, deps/, build-script, and build-script-execution/ move under build/<pkgname>/<unit-id>
- -L params to rustc were updated
- cargo clean -p was updated
- incremental/ is untouched
-Cextra-filename is removed from all bins (fix(layout): Remove hashes from bins in new layout #16351)

Known issues

As a side effect, we pass a lot more parameters to rustc, likely making cargo -vv more annoying, similar to Reduce --verbose output of --check-cfg args #13941
It also makes impossible to just copy paste commands into your shell if the limit is exceeded. See bjorn3's comment.
This will break people relying on build-dir internals
- See #t-cargo > build-dir tool compatibility @ 💬
Increases the need for solutions for when people rely on build-dir internals
- Access to compiler artifact notifications messages for custom --emit flags #13672
- cargo ./target fills with outdated artifacts as toolchains are updated/changed #5026
- Allow build scripts to stage final artifacts #13663
- RFC: cargo-sbom rfcs#3553 (more precise determination of what is built than cargo metadata)
cargo clean -p only cleans up files for the current layout rather than supporting both

Unresolved issues

Open questions

Transition plan: while build-dir isn't stable, enough tools rely on the layout that we'd want to setup a transition plan so they can have time to test against the new layout and work to support both
What do we call the directory? I said build/ as its all encompassing
- #t-cargo > directory naming with new build layout @ 💬
Can the old build/ and deps/ content live in the same place?
- Seems to work
How should we handle incremental/?
- rustc loads incremental artifacts only for local crates. cross-crates is from rmeta. Therefore per-crate incremental artifacts should be fine.
- Given incremental directory has its own flock mechanism, we don't need to add flock for that directory.
- See #t-compiler > ✔ Cargo switching to one `-C incremental` directory per c...
Under the new layout, should cargo clean -p also cleans old layout paths? See feat(build-dir): Reorganize build-dir layout #15947 (comment)
Is the current handling of build scripts sufficient or should we explicitly split them into separate entries. We separated build script build and execution
Move the #[derive(Default)] back for CliUnstable — Stabilize build-dir layout v2 #16807 (comment)
build-dir legacy layout post stablization cleanup #17182

Future extensions

Deferred layout changes

Can we remove -Cextra-filename completely as it is redundant with the build-unit's <name>/<hash>
- It may not be possible to remove -Cextra-filename for rlibs. rustc relies on it. See feat(build-dir): Reorganize build-dir layout #15947 (comment)
Can we remove <tuple> from the parent path?
- In addition to reducing path length pressure on Windows, could allow sharing of host builds between platforms
- Blocked on More granular locking in cargo_rustc #4282
Can we remove <profile> from the parent path?
- In addition to reducing path length pressure on Windows, could allow sharing when build units have the same settings across profiles
- <hash> is the -C extra-filename hash and doesn't encompass all of fingerprinting, so we'd need to audit if there are cases that don't change the hash that we'd still need per-profile
- Changing of local source is one example, so at least local packages still need to be scoped by profile
- Blocked on More granular locking in cargo_rustc #4282
Can we simplify how fingerprints are stored, reducing pressure on path lengths
If Windows filesystem perf issues crop up with <pkgname>/<hash>, we can go with prefixes to reduce the number of items within a directory. See bjorn3's comment.
- if deps/ hasn't been a problem, this is unlikely to be a problem

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Original

note: I specify build-dir to clarify which half of #14125 I'm referring to. The files and layout of build-dir does not have compatibility guarantees (source).

Currently, build-dir is laid out like

target/
- <target-platform>/?
  - <profile>/
    - incremental/
    - build/
      - <package>-<hash>/
        
        build*script-*
    - deps/
      - <package>-<hash>*

Currently,

cargo clean -p <package> will operate on everything for <package>

In the future, we could have

GC will track and operate on everything for <package>-<hash> (cargo ./target fills with outdated artifacts as toolchains are updated/changed #5026)
Change the locking so only overlapping <package>-<hash> that are being built block (More granular locking in cargo_rustc #4282)
Centrally cache <package>-<hash> artifacts across all projects (Per-user compiled artifact cache #5931)
Reduced overhead on systems that are slow when there are a lot of files within a directory (see also Adding tests has a high friction #15691)

These could be aided by re-arranging the build-dir to be organized around <package>-<hash>, like

target/
- <target-platform>/?
  - <profile>/
    - incremental/
    - build/
      - <package>-<hash>/
        
        build*script-*
        
        *.d

Side effects

We'll have to change how we invoke rustc which will increase the length of the command-line
- Currently, we blindly point rustc at deps/ and rustc finds the files it needs. We'll instead need to point to each individual artifact rustc may need.

These could be aided by re-arranging the build-dir to be organized around <package>-<hash>, like

target/

<target-platform>/?

<profile>/

incremental/

build/

<package>-<hash>/

build*script-*

*.d

I am assuming that final binary would still be located at target/<target-platform>/<profile>/<bin-name> (ie. target/debug/foo).
Is that correct?

Transition plan: while build-dir isn't stable, enough tools rely on the layout that we'd want to setup a transition plan so they can have time to test against the new layout and work to support both

I think we should check how many of the most popular tools rely on the layout. Depending the impact we can adjust how aggressive we want to be with the migration.

Regarding strategies, I have 2 ideas.

Simply write the files to the new and old layout directories. This of course with the additional overheads of doubling the disk writes, storage and the complexity of cleaning up multiple directories. But this provides the best backwards compatibility story.
Only write the files to the new layout and create symlinks in the previous layout. This would help mitigate the overhead of the disk writes/storage.

What do we call the directory? I said build/ as its all encompassing

I do not have a strong opinion on this. I was thinking perhaps packages or crates since its a list of the built packages. But I think build also makes sense here too 😄 Maybe build might be better to leave the door open for future possibilities of adding other artifacts to this directory?

I am assuming that final binary would still be located at target/// (ie. target/debug/foo).
Is that correct?

This does not touch final artifacts. I'd recommend reading up on the following note

note: I specify build-dir to clarify which half of #14125 I'm referring to.

I think we should check how many of the most popular tools rely on the layout. Depending the impact we can adjust how aggressive we want to be with the migration.

I know there is at least

There might be some other tools that do weirder stuff, like inspecting debug files or rlibs.

Regarding strategies, I have 2 ideas.

Writing to both or symlinks won't work for the above two tools.

A common approach we take is to have a feature be opt-in and then transition it to opt-out. A question in this is if we'd want to still support the old layout, for which we'd do this through a config, or if we'll only support the new layout, for which we use an env variable and after a sufficient time we remove the opt-out.

Okay, I finally had some time to read up on #14125 some other related threads.

Would it would be better to focus on making progress on separating target-build-dir/target-artifact-dir out as proposed here before attempting re-organize the layout? Doing it all at once would lead to less fragmentation of "build layouts". But adding to the scope would be more work and make it harder to land #14125.

I am leaning towards doing this re-organization after separating the build/artifact dirs.

I've wondered about doing the build-dir change first, like you said. It would make the scope of the change clear and it would help to communicate out what has compatibility guarantees.

Added tracing spans for rustc invocations (#15464)

I did some investigating on this. I created a small (and incomplete) prototype on my fork (6a644ff) where the dep-info files are stored in target/<target-platform>/<profile>/build/<package>-<hash>.

One notable side effect of doing this is that rustc command starts to get very large for projects with many dependencies.
This is because as mentioned in the issue description we currently we only add target/<profile>/deps the library search path (-L) in the current implementation.
If we reorganize the deps dir we need to add each library to the rustc lib search path which makes the process command very large if you have many dependencies.

This does not seem like an immediate issue, but I am not sure if there limits on some operating systems that might limit the size of a process command. (I created a synthetic test on my Linux machine and was able to create a rustc process with a 60MB with no issues. I didn't both trying anything larger)

This does not seem like an immediate issue, but I am not sure if there limits on some operating systems that might limit the size of a process command. (I created a synthetic test on my Linux machine and was able to create a rustc process with a 60MB with no issues. I didn't both trying anything larger)

Further down in the layers of calls to rustc, we automatically roll over from CLI args to an argfile.

Some potentially relevant questions

How much more frequently are we using argfiles?
What is the cost of switching to argfiles?
What is the cost of making the argfiles bigger?

Unsure how much we need to answer this in depth.

On my Linux machine the limit is around 2.6MiB

$ getconf ARG_MAX
2621440

Looking at how the thresholds are calculated: https://github.com/rust-lang/rust/blob/d2eadb7a94ef8c9deb5137695df33cd1fc5aee92/compiler/rustc_codegen_ssa/src/back/command.rs#L145-L205. I feel like on Windows it is way easier to hit the limit.

What is the cost of switching to argfiles?

One failed rustc invocation + writing big argfiles + only allow UTF-8 encoding (IIRC).
The good news is that only the last few rustc calls would have such long command line arguments.

The good news is that only the last few rustc calls would have such long command line arguments.

Yes, this is what I observed while testing with my changes.
As we get closer to the root of the dependency graph the larger the command grows.

How much more frequently are we using argfiles?

I think it will depend on the system settings. My machine has an ARG_MAX of 2097152 (2MB)
I did some testing in a dummy project with about ~360 total dependencies and the final rustc invocation clocked in at about 190KB

With some basic extrapolation I would start hitting the arg max on my machine for a project with ~4,000 dependencies.

I think a project of this size is pretty large the overhead of a few argfiles on the last few rustc calls will probably not be noticeable.

However, I am not so sure about Windows. I don't have a windows machine handy but this Stack overflow seems to suggest that its much lower at 2^16 chars (~32KB). But this post was 13 years ago so I am not sure if it has increased since then.

However, I am not so sure about Windows. I don't have a windows machine handy but this Stack overflow seems to suggest that its much lower at 2^16 chars (~32KB). But this post was 13 years ago so I am not sure if it has increased since then.

See the link in my previous comment #15010 (comment) for Windows arg limit, which is 6k in rustc.

Would it make sense to always add the target-platform for build dir layout?

As far as I can tell, if we build using cargo build --target <platform> we will use build-dir/<target-platform>/<profile> as the main build directory, then uplift files to build-dir/<profile>.

Given that the build-dir is not for human consumption I think we don't need do any uplifting inside of the build-dir.
I think this would simplify the uplift logic and make the layout more consistent.

Within build-dir, we can change however much we want and if we can make things more consistent,. then sure.

@ranger-ross

Reopening this issue, due to us reverting the new layout in #17187 after reports of regressions (#17183)

In #17183 we started the conversation around re-stabilizing the new build-dir layout, moving that discussion here

@ranger-ross

I wanted to get a vibe check on re-enabling the new build-dir layout on nightly. I think with the optimizations in #17168, #17191, and #17236 we should be in a much better place now.

Due to the nature of the new build dir layout, the command line args can still get pretty long for large workspaces, but i think with the optimizations we should be able to handle large workspaces like Zed. Though it kind of raises a question of "What is the largest workspace Cargo can support?". I am not sure what was the previous bottleneck.

Also would we want to wait until the beginning of the next beta cycle? I guess so considering we lost a few weeks to this.

@Mark-Simulacrum

Also would we want to wait until the beginning of the next beta cycle? I guess so considering we lost a few weeks to this.

Historically we've often kept features only enabled on nightly for a few cycles - I think this one is reasonable to do the same with, we'd enable ~anytime Cargo is ready to conditioning on the channel Cargo is built for. Then we can keep it on nightly essentially indefinitely (until we're happy to ship to stable).

@ranger-ross

Sure, I think that is reasonable to me. The previous plan was to merge the stabilization early in the nightly cycle so we had time to back it out if there were major issues. (which there was and we did 😄)

I think the tricky part is Cargo's testsuite, there are a lot of snapshot tests that need to update and its not feasible to keep both versions. One option would be to enable the new layout on nightly but still keep the old layout in tests on nightly. I don't love this since its not really testing nightly and other changes could introduce bugs in the new layout, but I can't think of any better ideas.

fix: use assert_cmd cargo_bin_cmd! macro in tests (#101) (#115)

Enable build-dir layout v2 on nightly by default (rust-lang#17258)

build: Adjust Dockerfile to new Cargo build-dir layout

build: adjust Dockerfile to new Cargo build-dir layout (#117)

build: adjust Dockerfile to new Cargo build-dir layout (#94)

Re-stabilize build-dir layout v2 (rust-lang#17354)

test(xtask): make fixture resolution independent of the test binary's…

common(nix,packages): sweep build-dir layout v2 in cargo-clean-stale

Navigation Menu

Repository navigation

Re-organize build-dir by package + hash, rather than artifact type #15010

Call of Testing

Overview

Known issues

Unresolved issues

Future extensions

About tracking issues

Original

71 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Re-organize build-dir by package + hash, rather than artifact type #15010

Description

Call of Testing

Overview

Known issues

Unresolved issues

Future extensions

About tracking issues

Original

Activity

ranger-ross commented on Jan 3, 2025

epage commented on Jan 3, 2025

ranger-ross commented on Jan 18, 2025

epage commented on Jan 18, 2025

ranger-ross commented on Apr 30, 2025

epage commented on Apr 30, 2025

weihanglo commented on Apr 30, 2025

ranger-ross commented on May 4, 2025

weihanglo commented on May 10, 2025

ranger-ross commented on Jul 6, 2025

epage commented on Jul 7, 2025

71 remaining items

ranger-ross commented on Jul 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions