test: Auto-redact ... after last build at ...; Migrate freshness to Snapbox#14161
Conversation
4e3688f to
3f80722
Compare
freshness to Snapbox
| // Following 3 subs redact: | ||
| // "1719325877.527949100s, 61549498ns after last build at 1719325877.466399602s" | ||
| // "1719503592.218193216s, 1h 1s after last build at 1719499991.982681034s" | ||
| // into "[DIRTY_REASON_NEW_TIME], [DIRTY_REASON_DIFF] after last build at [DIRTY_REASON_OLD_TIME]" |
There was a problem hiding this comment.
@weihanglo what are your thoughts on these redactions?
There was a problem hiding this comment.
Should we instead reduct everything in () that contains "after last build"?
There was a problem hiding this comment.
I am fine with the current redaction. Would could remove/change it when needed.
freshness to Snapbox... after last build a ...; Migrate freshness to Snapbox
... after last build a ...; Migrate freshness to Snapbox... after last build at ...; Migrate freshness to Snapbox
|
Could you squash your "fixup" commits into their appropriate commit? |
cbf4b1b to
56ca35f
Compare
tests/testsuite/freshness.rs
Outdated
| .with_stderr_data( | ||
| str![[r#" | ||
| [LOCKING] 3 packages to latest compatible versions | ||
| [COMPILING] bar [..] | ||
| [RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..] | ||
| [RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..] | ||
| [COMPILING] somepm [..] | ||
| [RUNNING] `rustc --crate-name somepm [..] | ||
| [COMPILING] foo [..] | ||
| [RUNNING] `rustc --crate-name foo --edition=2015 src/lib.rs [..]-C panic=abort[..] | ||
| [FINISHED] [..] | ||
| ", | ||
| [COMPILING] bar v0.0.1 ([ROOT]/foo/bar) | ||
| [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..] | ||
| [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..] | ||
| [COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm) | ||
| [RUNNING] `rustc --crate-name somepm [..]` | ||
| [COMPILING] foo v0.0.1 ([ROOT]/foo) | ||
| [RUNNING] `rustc --crate-name foo [..]-C panic=abort[..]` | ||
| [FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s | ||
|
|
||
| "#]] |
There was a problem hiding this comment.
I'm trying to fix this reuse_panic_pm CI failure, which passed on Tests on macOS aarch64 stable but failed on Test on macOS x86_64 stable with below:
---- expected: tests/testsuite/freshness.rs:1616:13
++++ actual: stderr
1 1 | [LOCKING] 3 packages to latest compatible versions
2 2 | [COMPILING] bar v0.0.1 ([ROOT]/foo/bar)
3 3 | [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..]
- 4 - [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..]
5 4 | [COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm)
6 5 | [RUNNING] `rustc --crate-name somepm [..]`
7 6 | [COMPILING] foo v0.0.1 ([ROOT]/foo)
8 7 | [RUNNING] `rustc --crate-name foo [..]-C panic=abort[..]`
9 8 | [FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
+ 9 + [RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=9229e8e86d572a0d -C extra-filename=-9229e8e86d572a0d --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps`Some thought & findings so far:
Since the test logic was unchanged, and it passed on Test on macOS x86_64 stable in next rerun (CI), I think the arch difference can be ruled out first.
Me next suspect is flakiness, which leads me to this with_stderr_unordered caveat from its nice comment:
/// Be careful when using patterns such as `[..]`, because you may end up /// with multiple lines that might match, and this is not smart enough to /// do anything like longest-match. For example, avoid something like: /// /// ```text /// [RUNNING] `rustc [..] /// [RUNNING] `rustc --crate-name foo [..] /// ``` /// /// This will randomly fail if the other crate name is `bar`, and the /// order changes.
The original L1531-L1532 seems to already match this scenario (A line matching L1532 could also match L1531). However, I assume reuse_panic_pm has been stable until this something introduced in my PR.
According to the comment form test:
bar is built once without panic (for proc-macro) and once with (for the normal dependency).
If the order of two bar build is non-deterministic this could be a problem; In contrast, if the order is deterministic, that makes me wonder why using unordered in the first place.
In fact, I removed .unordered, reran and passed reuse_panic_pm a few times locally, so let my try this first 👉 a453efd
I also tried to reproduce any randomness introduce by Snapbox unordered,
but no finding:
use snapbox::prelude::*;
use snapbox::str;
use snapbox::assert_data_eq;
#[cargo_test]
fn test_snapbox_unorder_randomness() {
let actual = str![[r#"
rustc --crate-name foo --a=b --ccc=ddd
rustc --crate-name foo --a=b --panic=abort --ccc=ddd
eee fff
"#]]
.unordered();
let expected = str![[r#"
rustc --crate-name foo [..]
rustc --crate-name foo [..] --panic=abort [..]
eee fff
"#]]
.unordered();
assert_data_eq!(actual, expected); // Always pass. Not flaky.
}P.s. For reference, the full output from my local runs are like:
[LOCKING] 3 packages to latest compatible versions
[COMPILING] bar v0.0.1 ([ROOT]/foo/bar)
[RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=3eb56625d2059feb -C extra-filename=-3eb56625d2059feb --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps`
[RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=e34f62847e3b5fb0 -C extra-filename=-e34f62847e3b5fb0 --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps`
[COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm)
[RUNNING] `rustc --crate-name somepm --edition=2015 somepm/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=a7ef9925b27e01b6 -C extra-filename=-a7ef9925b27e01b6 --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps --extern bar=[ROOT]/foo/target/debug/deps/libbar-[HASH].rlib --extern proc_macro`
[COMPILING] foo v0.0.1 ([ROOT]/foo)
[RUNNING] `rustc --crate-name foo --edition=2015 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=ebe9fb0a09bbfc5f -C extra-filename=-ebe9fb0a09bbfc5f --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps --extern bar=[ROOT]/foo/target/debug/deps/libbar-[HASH].rmeta --extern somepm=[ROOT]/foo/target/debug/deps/libsomepm-[HASH].dylib`
[FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
I guess some of my questions are:
- Just in case, was there any similar
reuse_panic_pmflakiness you recall seeing prior to this PR? - Does
reuse_panic_pmreally needwith_stderr_unorderedor.unordered()? - Anything else I should look into?
There was a problem hiding this comment.
Update: a453efd CI passes reuse_panic_pm on all stable pipeline: linux, mac x86, mac aarch64, Windows
There was a problem hiding this comment.
Update: The same error happened again after a453efd on Tests Windows x86_64 gnu nightly, so a453efd didn't fix the problem 😞
CI
https://github.com/rust-lang/cargo/actions/runs/9747539766/job/26900484318#step:11:4053
thread 'freshness::reuse_panic_pm' panicked at tests\testsuite\freshness.rs:1608:7:
---- expected: tests\testsuite\freshness.rs:1597:42
++++ actual: stderr
1 1 | [LOCKING] 3 packages to latest compatible versions
2 2 | [COMPILING] bar v0.0.1 ([ROOT]/foo/bar)
3 3 | [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..]
- 4 - [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..]
+ 4 + [RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no --check-cfg cfg(docsrs) --check-cfg "cfg(feature, values())" -C metadata=7b256d6ddbd506b7 -C extra-filename=-7b256d6ddbd506b7 --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps`
5 5 | [COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm)
6 6 | [RUNNING] `rustc --crate-name somepm [..]`
7 7 | [COMPILING] foo v0.0.1 ([ROOT]/foo)
8 8 | [RUNNING] `rustc --crate-name foo [..]-C panic=abort[..]`
9 9 | [FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]shttps://github.com/rust-lang/cargo/actions/runs/9781022885/job/27004092959?pr=14161
---- expected: tests/testsuite/freshness.rs:1620:13
++++ actual: stderr
1 1 | [LOCKING] 3 packages to latest compatible versions
2 2 | [COMPILING] bar v0.0.1 ([ROOT]/foo/bar)
3 3 | [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..]
- 4 - [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..]
5 4 | [COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm)
6 5 | [RUNNING] `rustc --crate-name somepm [..]`
7 6 | [COMPILING] foo v0.0.1 ([ROOT]/foo)
8 7 | [RUNNING] `rustc --crate-name foo [..]-C panic=abort[..]`
9 8 | [FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
+ 9 + [RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=9229e8e86d572a0d -C extra-filename=-9229e8e86d572a0d --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps`This seems like a flaky error either already exist or somehow related to my change on reuse_panic_pm.
Dropping a453efd and revert my change to reuse_panic_pm for now.
There was a problem hiding this comment.
Might need a deeper investigation of it. Fine with holding off for now.
afb4e9c to
14cbe9e
Compare
c826531 to
133ea7e
Compare
133ea7e to
b8a62da
Compare
weihanglo
left a comment
There was a problem hiding this comment.
Let's not block this. Thank you!
| // Following 3 subs redact: | ||
| // "1719325877.527949100s, 61549498ns after last build at 1719325877.466399602s" | ||
| // "1719503592.218193216s, 1h 1s after last build at 1719499991.982681034s" | ||
| // into "[DIRTY_REASON_NEW_TIME], [DIRTY_REASON_DIFF] after last build at [DIRTY_REASON_OLD_TIME]" |
There was a problem hiding this comment.
I am fine with the current redaction. Would could remove/change it when needed.
tests/testsuite/freshness.rs
Outdated
| .with_stderr_data( | ||
| str![[r#" | ||
| [LOCKING] 3 packages to latest compatible versions | ||
| [COMPILING] bar [..] | ||
| [RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..] | ||
| [RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..] | ||
| [COMPILING] somepm [..] | ||
| [RUNNING] `rustc --crate-name somepm [..] | ||
| [COMPILING] foo [..] | ||
| [RUNNING] `rustc --crate-name foo --edition=2015 src/lib.rs [..]-C panic=abort[..] | ||
| [FINISHED] [..] | ||
| ", | ||
| [COMPILING] bar v0.0.1 ([ROOT]/foo/bar) | ||
| [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..] | ||
| [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..] | ||
| [COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm) | ||
| [RUNNING] `rustc --crate-name somepm [..]` | ||
| [COMPILING] foo v0.0.1 ([ROOT]/foo) | ||
| [RUNNING] `rustc --crate-name foo [..]-C panic=abort[..]` | ||
| [FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s | ||
|
|
||
| "#]] |
There was a problem hiding this comment.
Might need a deeper investigation of it. Fine with holding off for now.
|
@bors r+ |
|
☀️ Test successful - checks-actions |
Update cargo 13 commits in a515d463427b3912ec0365d106791f88c1c14e1b..f86ce56396168edf5d0e1c412ddda0b2edba7d46 2024-07-02 20:53:36 +0000 to 2024-07-05 17:52:05 +0000 - test: Migrate jobserver to snapbox (rust-lang/cargo#14191) - chore(deps): update msrv (3 versions) to v1.77 (rust-lang/cargo#14186) - test: migrate build_plan and build_script to snapbox (rust-lang/cargo#14193) - test: migrate cfg and check to snapbox (rust-lang/cargo#14185) - test: migrate install* and inheritable_workspace_fields to snapbox (rust-lang/cargo#14170) - Pass rustflags to artifacts built with implicit targets when using target-applies-to-host (rust-lang/cargo#13900) - test: Migrate network tests to snapbox (rust-lang/cargo#14187) - test: migrate some files to snapbox (rust-lang/cargo#14113) - test: Auto-redact `... after last build at ...`; Migrate `freshness` to Snapbox (rust-lang/cargo#14161) - chore: fix some typos (rust-lang/cargo#14182) - fix: improve message for inactive weak optional feature with edition2024 through unused dep collection (rust-lang/cargo#14026) - test:migrate `doc/directory/docscrape` to snapbox (rust-lang/cargo#14171) - test: Migrate git_auth to snapbox (rust-lang/cargo#14172)
Update cargo 20 commits in a515d463427b3912ec0365d106791f88c1c14e1b..154fdac39ae9629954e19e9986fd2cf2cdd8d964 2024-07-02 20:53:36 +0000 to 2024-07-07 01:28:23 +0000 - test: relax redactions for rust-lang/rust (rust-lang/cargo#14203) - use "bootstrap" instead of "rustbuild" (rust-lang/cargo#14207) - test: migrate serveral files to snapbox (rust-lang/cargo#14180) - Add rustdocflags to Unit's Debug impl (rust-lang/cargo#14201) - Allow enabling `config-include` feature in config (rust-lang/cargo#14196) - fix(test): Restore `does_not_contain` for check (rust-lang/cargo#14198) - test: migrate patch, pkgid, proc_macro and progress to snapbox (rust-lang/cargo#14181) - test: Migrate jobserver to snapbox (rust-lang/cargo#14191) - chore(deps): update msrv (3 versions) to v1.77 (rust-lang/cargo#14186) - test: migrate build_plan and build_script to snapbox (rust-lang/cargo#14193) - test: migrate cfg and check to snapbox (rust-lang/cargo#14185) - test: migrate install* and inheritable_workspace_fields to snapbox (rust-lang/cargo#14170) - Pass rustflags to artifacts built with implicit targets when using target-applies-to-host (rust-lang/cargo#13900) - test: Migrate network tests to snapbox (rust-lang/cargo#14187) - test: migrate some files to snapbox (rust-lang/cargo#14113) - test: Auto-redact `... after last build at ...`; Migrate `freshness` to Snapbox (rust-lang/cargo#14161) - chore: fix some typos (rust-lang/cargo#14182) - fix: improve message for inactive weak optional feature with edition2024 through unused dep collection (rust-lang/cargo#14026) - test:migrate `doc/directory/docscrape` to snapbox (rust-lang/cargo#14171) - test: Migrate git_auth to snapbox (rust-lang/cargo#14172)
test: Update some emaining unordered tests to snapbox ### What does this PR try to resolve? This is part of #14039 This leaves `global_cache_tracker.rs` as it requires some more thinking. As for the flakiness in `freshness.rs` that was seen in #14161, `compare.rs` would prioritize expected lines according to their length (assuming its more specific). Currently, snapbox prioritizes according to the line order. So we just need to put the proc-macro line before the other one to ensure it gets precedence. ### How should we test and review this PR? ### Additional information
test: Update some emaining unordered tests to snapbox ### What does this PR try to resolve? This is part of #14039 This leaves `global_cache_tracker.rs` as it requires some more thinking. As for the flakiness in `freshness.rs` that was seen in #14161, `compare.rs` would prioritize expected lines according to their length (assuming its more specific). Currently, snapbox prioritizes according to the line order. So we just need to put the proc-macro line before the other one to ensure it gets precedence. ### How should we test and review this PR? ### Additional information
What does this PR try to resolve?
Part of #14039.
Migrate
tests/testsuite/freshness.rsto snapbox.How should we test and review this PR?
I followed #14039. The
#![allow(deprecated)]was removed andtests/testsuite/freshness.rsis still passing.Additional information
The redaction for "dirty reason" was a bit tricky
cargo/src/cargo/core/compiler/fingerprint/dirty_reason.rs
Line 131 in 7dcf764
Current implementation would redact
1719325877.527949100s, 61549498ns after last build at 1719325877.466399602s1719503592.218193216s, 1h 1s after last build at 1719499991.982681034sinto
[DIRTY_REASON_NEW_TIME], [DIRTY_REASON_DIFF] after last build at [DIRTY_REASON_OLD_TIME]Please let me know if that seems right.