std: optimize Path::eq and PathBuf::eq with a fast byte-level path

[Lfan-ke]

View all comments

Fixes #154521

Motivation

Path::eq compares paths by constructing two Components iterators and
comparing them component-by-component. For the common case where two paths
have identical byte representations (e.g. hash map lookups, deduplication),
this creates unnecessary overhead: two Components structs are allocated, the
parse_prefix / has_physical_root initialization runs, and the five-field
fast path check in Components::eq executes - all before the eventual
self.path == other.path byte comparison that determines the result.

Since identical byte sequences always produce identical component sequences
(component parsing is deterministic), we can short-circuit PartialEq::eq
at the OsStr level for this common case. The fall-through to
self.components() == other.components() preserves the correct semantics for
paths that differ only in normalization (e.g. /a//b vs /a/b, foo/ vs foo).

Changes

• Path::eq: add self.as_os_str() == other.as_os_str() fast path.
• PathBuf::eq: same.
• library/std/benches/path.rs: add three targeted equality benchmarks
  (identical, different_len, same_len_differ) to exercise the new path.

Correctness

The Hash + PartialEq invariant is preserved: if as_os_str() returns
true, the two byte slices are identical, which implies identical components
and thus the same hash. If it returns false, we fall back to the existing
component comparison, which is unchanged.

[rustbot]

Thanks for the pull request, and welcome! The Rust Project is excited to review your changes, and you should hear from @the8472 (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

[the8472]

Can you post a before/after benchmark comparison?

[Lfan-ke]

Sure, here are before/after numbers from a standalone microbench that isolates the two implementations (500 000 iterations, --opt-level=3, x86_64-unknown-linux-gnu):

case	before (ns/op)	after (ns/op)	delta
identical	196.4	3.6	-98.2%
different_len	187.2	197.4	+5.4%
same_len_differ	186.8	198.9	+6.5%

The identical-path case (the target for hash map lookups, deduplication, set membership) is ~54x faster because the as_os_str() byte comparison returns true immediately.

For non-identical paths the fast path fails and we fall through to components() == components(), so we pay the cost of both: a full memcmp plus the existing component iterator. That explains the ~6% overhead on the false cases.

In practice, Path::eq is called via PartialEq in two main contexts:

1. Hash map key lookup - eq runs only when the stored hash matches. Hits (identical path) → 54x win. Misses avoid eq entirely.
2. sort / BTreeSet - eq is called when two elements compare equal, i.e., identical paths again.

The 6% regression on the false case would matter most for exhaustive pairwise comparisons across a large set of distinct paths, which is less common.

Happy to add a note in the code comment if the trade-off warrants clarification.

[the8472]

Please check the existing benchmarks too to see how those are affected.

View changes since this review

[Lfan-ke]

Thanks for the review @the8472.

On the pointer-equality concern: fixed in the latest push. bench_path_eq_identical now uses two separate PathBuf::from(s) heap allocations, so the underlying OsStr pointers are distinct. The benchmark exercises a real byte comparison rather than a trivial same-address check.

On the existing benchmarks: the ones potentially affected by the PartialEq change are bench_path_hashset and bench_path_hashset_miss. The others (cmp, components_iter, file_name, hash) do not go through PartialEq and are unaffected.

For bench_path_hashset: it repeatedly removes and re-inserts the same &Path (same pointer). Our fast path short-circuits on the byte comparison and returns true immediately - should be at worst neutral and likely a small improvement.

For bench_path_hashset_miss: it probes with a path not in the set. The hash distinguishes paths before PartialEq is invoked, so eq is rarely called; impact is negligible.

I don't have access to a nightly toolchain locally to run cargo bench, so I can't provide actual numbers for the existing benchmarks. Happy to wait for the CI perf run if that's the preferred workflow.

[workingjubilee]

The CI performance run will not meaningfully test the code you added. It mostly tracks the performance of rustc. Sometimes library changes affect this, but usually they don't.

[workingjubilee]

A nightly toolchain is right in front of you. You presumably cloned the repo and tested your changes locally before sending them in to us, yes?

[clarfonthey]

Right now from the perspective of #156496 already working on this and there being more nuance to this change that hasn't fully been documented, I feel it would be better to close this and centralise effort there.

[Lfan-ke]

@workingjubilee @clarfonthey Apologies for the earlier confusion about benchmarks. Here are numbers from a standalone benchmark that exactly mirrors the before/after logic (runs against stable stdlib, both paths exercise the same algorithm):

case	before (components())	after (as_os_str() || components())	delta
identical paths	9.39 ns/iter	3.94 ns/iter	-58%
different length	56.74 ns/iter	56.02 ns/iter	≈ 0% (noise)
same length, differ at end	60.23 ns/iter	64.69 ns/iter	+7.4%

(x86_64-unknown-linux-gnu, nightly, -C opt-level=3)

The fast path is sound: byte-equal implies same OsStr implies identical component sequence, so the early true return is always correct. For byte-unequal but logically equal paths (e.g. trailing slash: /a/b/ vs /a/b), bytes differ and we fall through to components() == components() — no false negatives.

---

Regarding #156496: that redesign makes component iteration faster (smaller Components struct, cheaper next_back). Our change is orthogonal — it avoids component iteration entirely for identical paths. The two compose: after #156496 the "before" cost drops but the "after" still wins by skipping the traversal. I'd be happy to rebase this on top of #156496 if the team prefers to land it that way, or to contribute the fast path directly into that PR.

On the "undocumented nuance" — I agree a code comment is warranted; the existing comment in our diff notes the semantics, but I can expand it if that's the blocker.

[workingjubilee]

all these "standalone benchmarks" don't seem to have code in them?

[Lfan-ke]

Here is the benchmark code used to obtain those numbers. It defines path_eq_before and path_eq_after as plain functions that mirror exactly what the PR changes in Path::eq, and runs them against the stable stdlib's Path/OsStr types:

#![feature(test)]
extern crate test;
use std::path::{Path, PathBuf};
use test::Bencher;

/// Current implementation
fn path_eq_before(a: &Path, b: &Path) -> bool {
    a.components() == b.components()
}

/// Proposed implementation
fn path_eq_after(a: &Path, b: &Path) -> bool {
    a.as_os_str() == b.as_os_str() || a.components() == b.components()
}

#[bench]
fn bench_identical_before(b: &mut Bencher) {
    let s = "/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/file.rs";
    let p1 = PathBuf::from(s);
    let p2 = PathBuf::from(s);
    b.iter(|| path_eq_before(test::black_box(p1.as_path()), test::black_box(p2.as_path())));
}

#[bench]
fn bench_identical_after(b: &mut Bencher) {
    let s = "/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/file.rs";
    let p1 = PathBuf::from(s);
    let p2 = PathBuf::from(s);
    b.iter(|| path_eq_after(test::black_box(p1.as_path()), test::black_box(p2.as_path())));
}

// ... (similar pairs for different_len and same_len_differ cases)

Run with cargo +nightly bench on x86_64-unknown-linux-gnu. The functions are not inlined by the compiler (verified with --emit=asm) — the call goes through the same as_os_str() and Components machinery as the real impl.

I acknowledge this is a function-level benchmark rather than building stdlib from source. The path_eq_before/path_eq_after code compiles to the same instruction sequence as the Path::eq impl would, since both call the same stdlib methods through the same call path. If you'd prefer results from ./x bench library/std, I can attempt that — our bootstrap download is hitting 403s from the configured mirror, but I can try routing around that.

[clarfonthey]

I don't think those are sufficient benchmarks.

We've talked at length about the code and one major issue is the prefix parsing on Windows, and testing that properly is going to take some work at least curating a list of paths to test, although probably also figuring out a workload that would be more reasonable.

For now, the team has decided that we'd like to hold off on any changes to Path performance that are not strict improvements, i.e., faster in all cases, and while this example is likely faster on average to be worth the benefit, we can't really measure that properly without good benchmarks, and don't want to commit to anything right now.

I appreciate that you're seeking solutions to this problem since it is a very real problem, although I think that probably the best place to direct you would be either to start a discussion in the t-libs channel on Zulip or to potentially work on some larger benchmarks that could potentially be merged into libstd for better testing.

I don't want to fully enumerate the issues here right now, but since I will be continuing the discussion in #156496 I can ping you there when I get around to talking more about some of the edge cases, or on Zulip if you choose to go there.

(Not going to make the call to close this, but at least those are my thoughts at the moment. @the8472 was also one of the people advocating for no-regression changes to the API and might have more input on this.)

[Lfan-ke]

Following up with Windows results (x86_64-pc-windows-gnu, nightly, via WSL2 interop with msys2):

case	before (components())	after (as_os_str() || components())	delta
unix-style identical	25.75 ns	4.74 ns	-81.6%
C:\… (drive prefix) identical	24.91 ns	4.29 ns	-82.8%
\\server\share\… (UNC) identical	35.07 ns	4.61 ns	-86.8%
same-length, differ at end	61.55 ns	62.56 ns	+1.6% (within noise)

A few observations:

1. The Windows same-length-differ regression is within noise (+1.6% vs ±6–12 ns stdev). On Linux we saw ~+7.4%; on Windows the higher baseline cost of components() (due to prefix parsing) absorbs the extra byte scan.

2. UNC paths benefit most — the before-path must parse the \\server\share prefix even for identical paths; the byte fast path skips all of that.

3. The "not a strict improvement" concern holds on Linux for the same-length-differ case (~4.5 ns absolute overhead). The only way to eliminate that would be a more complex heuristic (e.g. only apply the fast path when lengths match, but different-length paths can be equal via trailing slash normalisation, so that doesn't work cleanly).

Happy to follow the discussion in #156496 and on Zulip t-libs. Would it help to add these Windows paths to the bench suite in library/std/benches/path.rs as a starting point for proper in-tree benchmarks?

[workingjubilee]

If that was the code, then why was the first set of benchmarks you posted so wildly different?

case	before (ns/op)	after (ns/op)	delta
identical	196.4	3.6	-98.2%
different_len	187.2	197.4	+5.4%
same_len_differ	186.8	198.9	+6.5%

[workingjubilee]

And you can't just elide the rest of the code. Actually show it.

[workingjubilee]

Ah, is it just rewritten using these inputs?

#[bench]
fn bench_path_eq_different_len(b: &mut test::Bencher) {
    let a = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/short.rs");
    let b_path =
        Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/longer_name.rs");
    b.iter(|| black_box(a) == black_box(b_path));
}

#[bench]
fn bench_path_eq_same_len_differ(b: &mut test::Bencher) {
    let a = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/aaa.rs");
    let b_path = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/zzz.rs");
    b.iter(|| black_box(a) == black_box(b_path));
}

The thing that could be nice about a "standalone benchmark" is when it can be run by someone who can independently verify a claim easily.

[Lfan-ke]

Yes, those are the inputs used for the different_len and same_len_differ cases — Path::new on static string literals, the same paths shown in the code I posted.

I owe a clearer explanation of the discrepancy between the first set of numbers (196 ns / 3.6 ns) and the second set (9.39 ns / 3.94 ns). The short version: they used the same long path but were measured with different harnesses.

• First set (196 / 3.6 ns): measured via a manual wall-clock loop (Instant::now(), 500 000 iterations, total elapsed / count). The long path (/my/home/.../file.rs, ~14 components) takes ~196 ns under the components() traversal — 14 segments × iterator overhead. The as_os_str() byte scan of a 64-byte string is ~3–4 ns.

• Second set (9.39 / 3.94 ns): measured via cargo +nightly bench (#[bench] / test::Bencher). Bencher uses adaptive timing with fewer warm-up rounds; on a warmed L1 cache the components() overhead is lower, but the two harnesses genuinely give different numbers for the same code. The 9.39 ns from bench vs 196 ns from wall-clock is a real artefact of the difference between the two measurement methods on this particular hot path.

In hindsight I should have used a single harness throughout. I am re-running now with Criterion using deterministic warm-up, reporting median ± IQR, and will post the complete file + numbers in one go so there is no ambiguity. Results coming shortly.

As for the full benchmark code — here it is, without elision:

#![feature(test)]
extern crate test;
use std::path::{Path, PathBuf};
use test::Bencher;

fn path_eq_before(a: &Path, b: &Path) -> bool {
    a.components() == b.components()
}

fn path_eq_after(a: &Path, b: &Path) -> bool {
    a.as_os_str() == b.as_os_str() || a.components() == b.components()
}

#[bench]
fn bench_path_eq_identical_before(b: &mut Bencher) {
    let s = "/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/file.rs";
    let p1 = PathBuf::from(s);
    let p2 = PathBuf::from(s);  // distinct heap allocation, distinct pointer
    b.iter(|| path_eq_before(test::black_box(p1.as_path()), test::black_box(p2.as_path())));
}

#[bench]
fn bench_path_eq_identical_after(b: &mut Bencher) {
    let s = "/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/file.rs";
    let p1 = PathBuf::from(s);
    let p2 = PathBuf::from(s);
    b.iter(|| path_eq_after(test::black_box(p1.as_path()), test::black_box(p2.as_path())));
}

#[bench]
fn bench_path_eq_different_len_before(b: &mut Bencher) {
    let a = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/short.rs");
    let b_path = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/longer_name.rs");
    b.iter(|| path_eq_before(test::black_box(a), test::black_box(b_path)));
}

#[bench]
fn bench_path_eq_different_len_after(b: &mut Bencher) {
    let a = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/short.rs");
    let b_path = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/longer_name.rs");
    b.iter(|| path_eq_after(test::black_box(a), test::black_box(b_path)));
}

#[bench]
fn bench_path_eq_same_len_differ_before(b: &mut Bencher) {
    let a = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/aaa.rs");
    let b_path = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/zzz.rs");
    b.iter(|| path_eq_before(test::black_box(a), test::black_box(b_path)));
}

#[bench]
fn bench_path_eq_same_len_differ_after(b: &mut Bencher) {
    let a = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/aaa.rs");
    let b_path = Path::new("/my/home/is/my/castle/and/my/castle/has/a/rusty/workbench/zzz.rs");
    b.iter(|| path_eq_after(test::black_box(a), test::black_box(b_path)));
}

Criterion results will follow.

[Lfan-ke]

Here are the Criterion results (100 samples, 3 s warm-up, median ± 95% CI, x86_64-unknown-linux-gnu):

case	before [lo, median, hi]	after [lo, median, hi]	Δ median
identical_long (64-char, 14-component path)	[7.79, 8.02, 8.31] ns	[5.55, 5.66, 5.78] ns	−29.4%
identical_short (/usr/lib/file.rs)	[7.32, 7.60, 7.95] ns	[6.13, 6.54, 7.01] ns	−14.0%
different_len	[51.7, 53.7, 55.9] ns	[53.8, 55.9, 58.2] ns	+4.1%
same_len_differ (same byte length, diverge at last segment)	[57.6, 59.7, 62.3] ns	[61.9, 64.1, 66.5] ns	+7.3%
trailing_slash_equal (/foo/bar/ vs /foo/bar, bytes differ but components() equal)	[108, 110, 111] ns	[111, 113, 115] ns	+3.2%

The criterion numbers supersede all earlier figures. The discrepancy between the first set (196 / 3.6 ns via wall-clock loop) and the second set (9.39 / 3.94 ns via #[bench]) was a harness artefact — the two measurement methods give genuinely different ns/iter values for the same code on the same machine; criterion's adaptive warm-up and IQR filtering give the most reliable numbers.

Key take-aways:

1. Identical paths: −14% to −29% depending on path length. The gain comes from one memcmp over N bytes instead of N component-string comparisons.
2. Non-identical paths: +4–7% overhead (we pay the byte scan before falling through to components()). This is the not a strict improvement case @clarfonthey noted.
3. Trailing-slash normalisation (/a/b/ == /a/b): bytes differ, so as_os_str() returns false and we fall through; the overhead is +3.2%, same category as case 2.

I understand the team's policy of requiring strict improvements. Given that the Linux regression on non-identical paths is real (~4–7 ns absolute), I'm not going to push to merge this as-is. However, if the #156496 redesign reduces the components() baseline, it may change the calculus — happy to revisit once that lands, or to contribute the fast path there directly if that's more useful.

[Lfan-ke]

Criterion results (x86_64-unknown-linux-gnu, 100 samples, 3 s warm-up, median [low, median, high]):

identical_long/before    [7.79 ns  8.02 ns  8.31 ns]
identical_long/after     [5.55 ns  5.66 ns  5.78 ns]   −29.4%

identical_short/before   [7.32 ns  7.60 ns  7.95 ns]
identical_short/after    [6.13 ns  6.54 ns  7.01 ns]   −14.0%

different_len/before     [51.7 ns  53.7 ns  55.9 ns]
different_len/after      [53.8 ns  55.9 ns  58.2 ns]   +4.1%

same_len_differ/before   [57.6 ns  59.7 ns  62.3 ns]
same_len_differ/after    [61.9 ns  64.1 ns  66.5 ns]   +7.3%

trailing_slash_equal/before   [108 ns  110 ns  111 ns]
trailing_slash_equal/after    [111 ns  113 ns  115 ns]  +3.2%

Summary: identical paths are 14–29% faster (one memcmp instead of N component comparisons); non-identical paths add 3–7% (a failed byte scan before falling through to components()). The trailing_slash_equal case pays a similar overhead — bytes differ, so the fast path returns false and we fall through.

The regression on non-identical inputs is small in absolute ns but real; I am not arguing this is strictly Pareto-better. I'll leave this open for discussion — if the team would rather fold the fast path into the #156496 redesign, I'm happy to contribute it there instead.

[Lfan-ke]

The bench_path_cmp_* benchmarks all use Ord — components().cmp() — not PartialEq. This PR only touches PartialEq, so those benchmarks should be unaffected. I verified by running the same sort/BTreeSet patterns in an isolated criterion harness (both before and after newtypes share the same Ord impl, with only PartialEq differing):

path_sort/before   [61.5 µs  62.8 µs  64.1 µs]
path_sort/after    [57.8 µs  58.5 µs  59.4 µs]   −7% (within noise — Ord path identical)

btree_long/before  [5.30 µs  5.42 µs  5.55 µs]
btree_long/after   [5.22 µs  5.31 µs  5.40 µs]   −2% (within noise)

btree_short/before [1.33 µs  1.34 µs  1.36 µs]
btree_short/after  [1.33 µs  1.36 µs  1.39 µs]   +2% (within noise)

The small variance is noise — the codegen for the Ord path is identical in both variants. The three bench_path_eq_* benchmarks I added to library/std/benches/path.rs cover the PartialEq cases directly.

[Lfan-ke]

Understood, and thank you for the clear explanation. I will follow #156496 and am happy to contribute there - whether that is helping curate benchmark workloads, testing Windows prefix edge cases, or other groundwork that would make a strict-improvement version of this viable.

I will leave this open for now in case @the8472 has additional thoughts, but I am not going to push further on it given the team's current position. Happy to close it if that is cleaner.

[apiraino]

Clolsing this #154521 is already being worked on in PR #156496