Skip to content

Tracking Issue for high failure rates on Windows MSVC CI with filesystem errors #127883

Open
Open
Tracking Issue for high failure rates on Windows MSVC CI with filesystem errors#127883
Labels
A-CIArea: Our Github Actions CIC-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFCCI-spurious-fail-msvcCI spurious failure: target env msvcO-windowsOperating system: WindowsO-windows-msvcToolchain: MSVC, Operating system: WindowsT-infraRelevant to the infrastructure team, which will review and decide on the PR/issue.

Description

@ehuss
ehuss
opened on Jul 17, 2024

Starting around 2024-06-27, the rust-lang CI has started to encounter a very high failure rate on the MSVC Windows builders (~15% of all builds?). These builders are encountering various filesystems errors, such as "Access is denied", "used by another process", "cannot open file", etc.

If you run into them, please tag the PR with CI-spurious-fail-msvc CI spurious failure: target env msvc so we can try to find clues about what could be causing the high failures. You can also click on the label to see the list of PRs that were/are affected by this class of failures.

From what I can tell, this doesn't seem to be affecting the GNU builders.

Discussions

Zulip discussion: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/Spurious.20CI.20errors.20on.20x86_64-msvc-ext

Past Investigations

Starting 2024-07-02, #127152 added a mitigation measure in bootstrap to cover the most common culprit of bootstrap attempting to delete an executable. However, there are several other programs that are still having problems, such as rustc itself, the msvc linker, and cargo.

I have tried using handle.exe, and the RestartManager API to try to detect if there is another process with an open handle on a file, but no success.

I have tried rolling back the source tree to 2024-06-25, and it still reproduces the problem (before we started seeing it in CI).

The last Windows image release before this started was https://github.com/actions/runner-images/releases/tag/win22%2F20240618.1. From what I can tell, this rolled out about 5 days earlier, during which there weren't any failures, but it is difficult to tell if that could be related.

The last stage0 bump was 2024-06-11, several weeks before it started.

actions/runner-images#4086 is a similar issue we've had in the past, though we don't know what the fix was.

Example

Example errors seen:

  • couldn't create a temp dir: Access is denied.
  • failed to copy C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe to C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe
  • failed to copy C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage0-rustc\x86_64-pc-windows-msvc\release\rustc-main.exe to C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1\bin\rustc.exe
  • note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-2a5d00462f8c666b.exe'
  • failed to remove file C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe

Example log entries

Example CI log 
2024-05-15T17:13:48 45_auto - x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9099930545.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9099930545
2024-05-15T18:00:27.5922941Z --- stdout -------------------------------
2024-05-15T18:00:27.5924398Z MONO_ITEM fn inline::inlined_function @@ local_inlining.8ec87a7fbea559a-user2[Internal] local_inlining.8ec87a7fbea559a-user1[Internal]
2024-05-15T18:00:27.5925974Z MONO_ITEM fn non_user::baz @@ local_inlining.8ec87a7fbea559a-non_user[External]
2024-05-15T18:00:27.5926735Z MONO_ITEM fn user1::foo @@ local_inlining.8ec87a7fbea559a-user1[External]
2024-05-15T18:00:27.5927446Z MONO_ITEM fn user2::bar @@ local_inlining.8ec87a7fbea559a-user2[External]
2024-05-15T18:00:27.5927993Z ------------------------------------------
2024-05-15T18:00:27.5928393Z --- stderr -------------------------------
2024-05-15T18:00:27.5929309Z error: couldn't create a temp dir: Access is denied. (os error 5) at path "C:\\a\\_temp\\msys64\\tmp\\rustcFKhJYE"
2024-05-15T18:00:27.5929887Z 
2024-05-15T18:00:27.5930041Z error: aborting due to 1 previous error
2024-05-15T18:00:27.5930467Z ------------------------------------------
2024-05-15T18:00:27.5930772Z 
2024-05-15T18:00:27.5930778Z 
2024-05-15T18:00:27.5930784Z 
2024-05-15T18:00:27.5930879Z failures:
2024-05-15T18:00:27.5931329Z     [codegen-units] tests\codegen-units\partitioning\local-inlining.rs



2024-06-21T23:51:42 45_auto - x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9621092085.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9621092085
2024-06-22T00:38:19.3827973Z --- stderr -------------------------------
2024-06-22T00:38:19.3828801Z error: couldn't create a temp dir: Access is denied. (os error 5) at path "C:\\a\\_temp\\msys64\\tmp\\rustcMwb8Le"
2024-06-22T00:38:19.3829381Z 
2024-06-22T00:38:19.3829535Z error: aborting due to 1 previous error
2024-06-22T00:38:19.3829971Z ------------------------------------------
2024-06-22T00:38:19.3830211Z 
2024-06-22T00:38:19.3830217Z 
2024-06-22T00:38:19.3830223Z 
2024-06-22T00:38:19.3830318Z failures:
2024-06-22T00:38:19.3830672Z     [ui] tests\ui\lto\lto-duplicate-symbols.rs




2024-06-27T21:05:13 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9703582059.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9703582059
2024-06-27T22:37:13.7595098Z [TIMING] core::build_steps::tool::LlvmBitcodeLinker { compiler: Compiler { stage: 0, host: x86_64-pc-windows-msvc }, target: x86_64-pc-windows-msvc, extra_features: [] } -- 0.396
2024-06-27T22:37:13.7611176Z thread 'main' panicked at src/lib.rs:1713:17:
2024-06-27T22:37:13.7613255Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage0-rustc\x86_64-pc-windows-msvc\release\rustc-main.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1\bin\rustc.exe`: The process cannot access the file because it is being used by another process. (os error 32)

2024-06-27T23:42:02 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9705061663.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9705061663
2024-06-28T01:08:22.3300116Z     Finished `release` profile [optimized] target(s) in 0.41s
2024-06-28T01:08:22.3517842Z thread 'main' panicked at src/lib.rs:1713:17:
2024-06-28T01:08:22.3520184Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-06-28T01:10:46 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9705841544.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9705841544
2024-06-28T02:37:41.9366820Z     Finished `release` profile [optimized] target(s) in 0.39s
2024-06-28T02:37:41.9583934Z thread 'main' panicked at src/lib.rs:1713:17:
2024-06-28T02:37:41.9586316Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)



2024-06-28T06:30:49 51_auto - dist-x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9708494009.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9708494009
2024-06-28T07:23:27.6412089Z   = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-2a5d00462f8c666b.exe'
2024-06-28T07:23:27.6413223Z 
2024-06-28T07:23:27.6413471Z           
2024-06-28T07:23:27.6413651Z 
2024-06-28T07:23:27.6558489Z error: could not compile `rustc-main` (bin "rustc-main") due to 1 previous error



2024-06-28T09:36:50 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9710725566.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9710725566
2024-06-28T11:03:13.0991016Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-28T11:03:13.0993328Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)

2024-06-28T11:05:47 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9711788661.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9711788661
2024-06-28T12:32:22.9420083Z     Finished `release` profile [optimized] target(s) in 0.39s
2024-06-28T12:32:22.9679265Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-28T12:32:22.9681613Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-06-28T14:28:38 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9714425374.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9714425374
2024-06-28T16:03:54.7079854Z     Finished `release` profile [optimized] target(s) in 0.38s
2024-06-28T16:03:54.7298453Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-28T16:03:54.7300791Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)

2024-06-28T16:06:30 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9715662476.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9715662476
2024-06-28T17:40:10.3608665Z     Finished `release` profile [optimized] target(s) in 0.39s
2024-06-28T17:40:10.3838225Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-28T17:40:10.3840547Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)

2024-06-28T22:09:33 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9719051471.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9719051471
2024-06-28T23:42:05.4732924Z     Finished `release` profile [optimized] target(s) in 0.40s
2024-06-28T23:42:05.4950520Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-28T23:42:05.4952824Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-06-29T02:00:21 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9720709773.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9720709773
2024-06-29T03:30:57.6850989Z     Finished `release` profile [optimized] target(s) in 0.39s
2024-06-29T03:30:57.7066422Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-29T03:30:57.7068713Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)

2024-06-29T03:33:22 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9721279263.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9721279263
2024-06-29T04:58:15.8326565Z     Finished `release` profile [optimized] target(s) in 0.38s
2024-06-29T04:58:15.8683528Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-29T04:58:15.8685846Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-06-29T16:14:43 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9725396338.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9725396338
2024-06-29T17:48:11.1950874Z     Finished `release` profile [optimized] target(s) in 0.39s
2024-06-29T17:48:11.2163942Z thread 'main' panicked at src/lib.rs:1697:17:
2024-06-29T17:48:11.2166059Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-06-30T05:53:23 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9729777789.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9729777789
2024-06-30T07:18:43.4197375Z     Finished `release` profile [optimized] target(s) in 0.39s
2024-06-30T07:18:43.4410518Z thread 'main' panicked at src/lib.rs:1697:17:
2024-06-30T07:18:43.4412624Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)

2024-06-30T08:15:48 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9730548394.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9730548394
2024-06-30T09:40:47.0630932Z thread 'main' panicked at src/lib.rs:1697:17:
2024-06-30T09:40:47.0633012Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-06-30T09:43:08 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9731047889.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9731047889
2024-06-30T11:07:56.3788705Z     Finished `release` profile [optimized] target(s) in 0.39s
2024-06-30T11:07:56.4006304Z thread 'main' panicked at src/lib.rs:1697:17:
2024-06-30T11:07:56.4008397Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-06-30T15:35:38 57_auto - dist-x86_64-msvc-alt.txt "../logs-rust-lang-ci-rust\\logs_9733131162.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9733131162
2024-06-30T16:21:12.2870212Z thread 'main' panicked at src/lib.rs:1697:17:
2024-06-30T16:21:12.2871936Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2\bin\rustdoc.exe` to `C:\a\rust\rust\build\tmp\tarball\rustc\x86_64-pc-windows-msvc\image\bin\rustdoc.exe`: The process cannot access the file because it is being used by another process. (os error 32)

2024-06-30T18:43:09 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9734174198.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9734174198
2024-06-30T20:12:39.4763461Z thread 'main' panicked at src/lib.rs:1668:17:
2024-06-30T20:12:39.4765689Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-07-01T11:52:50 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9743687544.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9743687544
2024-07-01T13:17:07.6056787Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-01T13:17:07.6059053Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)

2024-07-01T13:19:31 52_auto - dist-x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9745004151.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9745004151
2024-07-01T14:07:36.3302291Z   = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-8c9e059926f63509.exe'
2024-07-01T14:07:36.3303400Z 
2024-07-01T14:07:36.3303642Z           
2024-07-01T14:07:36.3303843Z 
2024-07-01T14:07:36.3425523Z error: could not compile `rustc-main` (bin "rustc-main") due to 1 previous error
2024-07-01T14:07:36.3853896Z [2024-07-01T14:07:36.385Z INFO  opt_dist::timer] Section `Stage 1 (Rustc PGO) > Build PGO optimized rustc` ended: FAIL (562.38s)`
2024-07-01T14:07:36.3855267Z [2024-07-01T14:07:36.385Z INFO  opt_dist::timer] Section `Stage 1 (Rustc PGO)` ended: FAIL (2419.29s)`
2024-07-01T14:07:36.3856022Z [2024-07-01T14:07:36.385Z INFO  opt_dist] Timer results
2024-07-01T14:07:36.3856571Z     -----------------------------------------------------------------

2024-07-01T16:22:23 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9747722395.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9747722395
2024-07-01T17:54:39.4755072Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-01T17:54:39.4757239Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)

2024-07-01T17:56:59 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9748873919.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9748873919
2024-07-01T19:31:13.5443614Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-01T19:31:13.5445798Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-07-02T00:13:28 57_auto - dist-x86_64-msvc-alt.txt "../logs-rust-lang-ci-rust\\logs_9752796989.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9752796989
2024-07-02T01:01:09.3044765Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-02T01:01:09.3046561Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2\bin\rustdoc.exe` to `C:\a\rust\rust\build\tmp\tarball\rustc\x86_64-pc-windows-msvc\image\bin\rustdoc.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-07-02T01:03:31 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9753269047.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9753269047
2024-07-02T02:30:23.7862414Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-02T02:30:23.7864496Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)


2024-07-02T12:53:47 52_auto - dist-x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9761329541.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9761329541
2024-07-02T13:48:36.1493324Z   = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-8c9e059926f63509.exe'
2024-07-02T13:48:36.1494467Z 
2024-07-02T13:48:36.1494734Z           
2024-07-02T13:48:36.1494906Z 
2024-07-02T13:48:36.1608837Z error: could not compile `rustc-main` (bin "rustc-main") due to 1 previous error


2024-07-02T13:51:08 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9762243104.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9762243104
2024-07-02T15:17:39.8880874Z     Finished `release` profile [optimized] target(s) in 0.41s
2024-07-02T15:17:39.9100976Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-02T15:17:39.9103158Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)



2024-07-04T13:14:14 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9795018238.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9795018238
2024-07-04T14:21:22.8765500Z 
2024-07-04T14:21:35.0891504Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-04T14:21:35.0892319Z 
2024-07-04T14:21:35.0892437Z Caused by:
2024-07-04T14:21:35.0892781Z   Access is denied. (os error 5)


2024-07-06T09:36:09 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9818299526.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9818299526
2024-07-06T10:39:24.5995051Z 
2024-07-06T10:39:38.3349340Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-06T10:39:38.3350150Z 
2024-07-06T10:39:38.3350315Z Caused by:
2024-07-06T10:39:38.3350655Z   Access is denied. (os error 5)

2024-07-06T10:55:59 45_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9818726134.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9818726134
2024-07-06T12:05:59.8544699Z   = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps\rustdoc_tool_binary-a7c1152b8abe3e78.exe'
2024-07-06T12:05:59.8545901Z 
2024-07-06T12:05:59.8546164Z           
2024-07-06T12:05:59.8546344Z 
2024-07-06T12:05:59.8624055Z error: could not compile `rustdoc-tool` (bin "rustdoc_tool_binary") due to 1 previous error


2024-07-06T14:21:23 45_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9819935411.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9819935411
2024-07-06T15:54:19.8152269Z   = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps\miri-ff57ced5dff426fc.exe'
2024-07-06T15:54:19.8153375Z 
2024-07-06T15:54:19.8153630Z           
2024-07-06T15:54:19.8153804Z 
2024-07-06T15:54:19.8338918Z error: could not compile `miri` (bin "miri") due to 1 previous error

2024-07-07T10:30:30 47_auto - x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9826600375.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9826600375
2024-07-07T11:29:01.8968721Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-std\x86_64-pc-windows-msvc\release\deps\libcompiler_builtins-86bc7a5c308aedf8.rlib`
2024-07-07T11:29:01.8970410Z 
2024-07-07T11:29:01.8970609Z Caused by:
2024-07-07T11:29:01.8971101Z   Access is denied. (os error 5)


2024-07-07T18:50:57 48_auto - i686-msvc.txt "../logs-rust-lang-ci-rust\\logs_9829572529.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9829572529
2024-07-07T19:50:51.1512300Z error: failed to remove file `C:\a\rust\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\libcompiler_builtins-7b6e8fab8041690f.rlib`
2024-07-07T19:50:51.1513505Z 
2024-07-07T19:50:51.1513617Z Caused by:
2024-07-07T19:50:51.1513898Z   Access is denied. (os error 5)
2024-07-07T19:52:17.1913925Z make: *** [Makefile:102: ci-msvc-ps1] Error 1


2024-07-07T22:06:12 47_auto - x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9830624681.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9830624681
2024-07-07T23:00:35.2960420Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-std\x86_64-pc-windows-msvc\release\deps\libcompiler_builtins-86bc7a5c308aedf8.rlib`
2024-07-07T23:00:35.2962043Z 
2024-07-07T23:00:35.2962208Z Caused by:
2024-07-07T23:00:35.2962633Z   Access is denied. (os error 5)


2024-07-08T09:07:11 47_auto - x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9836726350.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9836726350
2024-07-08T10:02:01.6196210Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-std\x86_64-pc-windows-msvc\release\deps\libcompiler_builtins-86bc7a5c308aedf8.rlib`
2024-07-08T10:02:01.6198226Z 
2024-07-08T10:02:01.6198438Z Caused by:
2024-07-08T10:02:01.6198942Z   Access is denied. (os error 5)



2024-07-11T05:27:59 52_auto - dist-x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9886006325.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9886006325
2024-07-11T06:21:56.9020006Z   = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-aa69889b5edbcee3.exe'
2024-07-11T06:21:56.9021130Z 
2024-07-11T06:21:56.9021508Z           
2024-07-11T06:21:56.9021752Z 
2024-07-11T06:21:56.9148094Z error: could not compile `rustc-main` (bin "rustc-main") due to 1 previous error



2024-07-12T10:55:07 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9906744272.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9906744272
2024-07-12T11:55:13.9048987Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-12T11:55:13.9049831Z 
2024-07-12T11:55:13.9049952Z Caused by:
2024-07-12T11:55:13.9050247Z   Access is denied. (os error 5)


2024-07-13T12:06:40 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9919764593.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9919764593
2024-07-13T13:09:34.8085554Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-13T13:09:34.8086897Z 
2024-07-13T13:09:34.8087087Z Caused by:
2024-07-13T13:09:34.8087535Z   Access is denied. (os error 5)


2024-07-15T02:21:06 52_auto - dist-x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9932760975.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9932760975
2024-07-15T03:17:43.2981649Z   = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-4e69e1e45706632e.exe'
2024-07-15T03:17:43.2982781Z 
2024-07-15T03:17:43.2983034Z           
2024-07-15T03:17:43.2983208Z 
2024-07-15T03:17:43.3107182Z error: could not compile `rustc-main` (bin "rustc-main") due to 1 previous error


2024-07-16T05:02:37 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9951025123.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9951025123
2024-07-16T06:08:12.4360760Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-16T06:08:12.4362216Z 
2024-07-16T06:08:12.4362481Z Caused by:
2024-07-16T06:08:12.4363081Z   Access is denied. (os error 5)


2024-07-17T18:19:38 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9979412052.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9979412052
2024-07-17T19:24:48.6223853Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-17T19:24:48.6225041Z 
2024-07-17T19:24:48.6225224Z Caused by:
2024-07-17T19:24:48.6225659Z   Access is denied. (os error 5)

Activity

  1. Imageehuss
    added
    O-windowsOperating system: Windows
    O-windows-msvcToolchain: MSVC, Operating system: Windows
    A-CIArea: Our Github Actions CI
    on Jul 17, 2024
  2. Imagerustbot
    added
    needs-triageThis issue may need triage. Remove when done. See docs forge.rust-lang.org/release/issue-triaging
    on Jul 17, 2024
  3. Imagejieyouxu
    added
    T-infraRelevant to the infrastructure team, which will review and decide on the PR/issue.
    and removed
    needs-triageThis issue may need triage. Remove when done. See docs forge.rust-lang.org/release/issue-triaging
    on Jul 18, 2024

  4. michaelwoerister commented on Jul 19, 2024

    @michaelwoerister
    michaelwoerister
    on Jul 19, 2024
    Member

    Do you have a link to a failed pipeline run?

  5. ChrisDenton commented on Jul 19, 2024

    @ChrisDenton
    ChrisDenton
    on Jul 19, 2024
    Member

    Most recent https://github.com/rust-lang-ci/rust/actions/runs/9979412052/job/27578504068

  6. ehuss commented on Jul 19, 2024

    @ehuss
    ehuss
    on Jul 19, 2024
    ContributorAuthor

    You can also copy any of the https://github.com/rust-lang-ci/rust/actions/runs/... links in the logs above (sorry I didn't make them clickable). Just beware the logs get deleted (90 days?).

  7. jieyouxu commented on Aug 15, 2024

    @jieyouxu
    jieyouxu
    on Aug 15, 2024
    Member
    PR Failure cause
    #128032 LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps\miri-bdcbed7500be44cb.exe'
    #128031 (1) error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe
    #128031 (2) LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-4206913879fae405.exe'
    #124895 LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-cac7a0b14c781d14.exe'
    #128270 error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`, Access is denied (os error 5)
    #128577 LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-4f38a2f26fa25d5a.exe'
    #128037 (1) LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-abe621a284360afc.exe'
    #128037 (2) LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-abe621a284360afc.exe'
    #128936 error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`, Access is denied (os error 5)
    #129054 error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`, Access is denied (os error 5)
    #129066 std\src\os\windows\process.rs - os::windows::process::CommandExt::raw_attribute (line 330) stdlib test failed which is very weird, called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: PermissionDenied, message: "Access is denied."
    #128742 LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-abe621a284360afc.exe'
    #128371 LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-abe621a284360afc.exe'
    #128677 LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-abe621a284360afc.exe'
    #128982 error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`, Access is denied (os error 5)
    #128722 error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`, Access is denied. (os error 5)
    #129226 error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`, Access is denied. (os error 5)
    #127672 error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`, Access is denied. (os error 5)
    #129304 LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-ff540feda8314309.exe'

  8. saethlin commented on Aug 18, 2024

    @saethlin
    saethlin
    on Aug 18, 2024
    Member

    Another: #128982 (comment)

  9. tgross35 commented on Aug 23, 2024

    @tgross35
    tgross35
    on Aug 23, 2024
    Member

    It seems like a lot of these failures are specifically miri.exe for whatever reason, especially recently. Could we just adjust bootstrap to skip the rest of the step on MSVC once it gets to the point that it would remove this file? I'm not sure what exactly relies on this.

  10. jieyouxu commented on Aug 23, 2024

    @jieyouxu
    jieyouxu
    on Aug 23, 2024
    Member

    I'm planning to investigate some of the file removal logic in bootstrap. Working theory is some symlink removal related logic on Windows.

    EDIT: some of the failures seem to be emitted from cargo, but bootstrap seems to also have its share

  11. tgross35 commented on Aug 23, 2024

    @tgross35
    tgross35
    on Aug 23, 2024
    Member

    Crosslinking some ongoing Zulip discussion: https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Cargo.20investigation.3A.20High.20failure.20rate.20in.20rust-lang.20CI

  12. tgross35 commented on Aug 25, 2024

    @tgross35
    tgross35
    on Aug 25, 2024
    Member

    I opened actions/runner-images#10483 to see if GitHub is able to help us out.

  13. 20 remaining items

  14. jieyouxu commented on Sep 21, 2024

    @jieyouxu
    jieyouxu
    on Sep 21, 2024
    Member

    #130569 (comment) failure has a case where:

    failed to copy
`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage0-tools\x86_64-pc-windows-msvc\release\miri.exe`
to
`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage0-tools-bin\miri.exe`:
The process cannot access the file because it is being used by another process. (os error 32)

    in addition to a different failure

    error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`

    so there's definitely something stepping over something

  15. Imagetgross35
    added 2 commits that reference this issue on Sep 30, 2024

    Unpin `cc` and upgrade to the latest version

    ee83f74

    Unpin `cc` and upgrade to the latest version

    eaaa943
  16. Imagebors
    added 2 commits that reference this issue on Sep 30, 2024

    Auto merge of rust-lang#131070 - tgross35:update-root-cc, r=<try>

    4fc810a

    Auto merge of rust-lang#131070 - tgross35:update-root-cc, r=wesleywiser

    1d71891

  17. tgross35 commented on Oct 2, 2024

    @tgross35
    tgross35
    on Oct 2, 2024
    Member

    Small update; some Cargo.lock updates were hitting a 100% success rate and failing to merge due to these sort of failures on MSVC jobs. The problem was identified to be a change to cc that was released in 1.0.101 only a couple days before we started to notice this issue. None of our lockfiles contained the relevant version until later, but the timing seems too convenient to be coincidence (maybe we were using an unlocked version somewhere).

    Anyway, Chris fixed that issue at rust-lang/cc-rs#1215 and that was released in 1.1.22. We just finished bumping cc to that version in all the lockfiles (except for RA and a Clippy test). It still isn't certain whether the cc bug caused the problem in the first place or if it just made some other problem worse, time to experiment.

    Edit: nope, not solved :( #131133 (comment)

  18. Imagejieyouxu
    changed the title [-]High failure rate on Windows MSVC CI with filesystem errors[/-] [+]Tracking Issue for high failure rates on Windows MSVC CI with filesystem errors[/+] on Oct 15, 2024
  19. Imagejieyouxu
    added
    C-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFC
    CI-spurious-fail-msvcCI spurious failure: target env msvc
    on Oct 15, 2024
  20. Imageintel-media-ci
    added a commit that references this issue on Oct 23, 2024

    ci: Add scripts to retry windows builds on spurious errors

    15b0b49

  21. jieyouxu commented on Nov 16, 2024

    @jieyouxu
    jieyouxu
    on Nov 16, 2024 · Hidden as off-topic

  22. tgross35 commented on Feb 3, 2025

    @tgross35
    tgross35
    on Feb 3, 2025
    Member

    I haven't seen this apply to rustc-main.exe but #135265 just failed with

    failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\rustc-main.exe`

    Edit: oh no, #136399 (comment) is the same.

  23. RalfJung commented on Feb 3, 2025

    @RalfJung
    RalfJung
    on Feb 3, 2025
    Member

    #136146 also got that same error

    error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\rustc-main.exe`
  24. Imageaslobodeniuk
    added a commit that references this issue on Apr 23, 2025

    ci: Add scripts to retry windows builds on spurious errors

    04777d3
  25. Imagesuavecito585
    added a commit that references this issue on Aug 9, 2026

    fix: report an unreadable output stream instead of returning it as em…

    68845e7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-CIArea: Our Github Actions CIC-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFCCI-spurious-fail-msvcCI spurious failure: target env msvcO-windowsOperating system: WindowsO-windows-msvcToolchain: MSVC, Operating system: WindowsT-infraRelevant to the infrastructure team, which will review and decide on the PR/issue.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions