Skip to content

[CI] Add workaround for dotnet/runtime#91987 (NuGet-Migrations race condition) - #25797

Merged
rolfbjarne merged 3 commits into
mainfrom
release-test/workaround-nuget-migrations
Jun 25, 2026
Merged

[CI] Add workaround for dotnet/runtime#91987 (NuGet-Migrations race condition)#25797
rolfbjarne merged 3 commits into
mainfrom
release-test/workaround-nuget-migrations

Conversation

@rolfbjarne

@rolfbjarne rolfbjarne commented Jun 24, 2026

Copy link
Copy Markdown
Member

Run 'dotnet --info' once right after downloading the .NET SDK to complete the first-time NuGet migrations logic. This avoids the race condition where multiple parallel dotnet processes attempt the first-time setup simultaneously, causing 'The system cannot open the device or file specified' errors.

The workaround only runs in CI (when the ACES environment variable is set).

Ref: dotnet/runtime#91987

…ondition)

Run 'dotnet --info' once right after downloading the .NET SDK to
complete the first-time NuGet migrations logic. This avoids the race
condition where multiple parallel dotnet processes attempt the
first-time setup simultaneously, causing 'The system cannot open the
device or file specified' errors.

The workaround only runs in CI (when the ACES environment variable is
set).

Ref: dotnet/runtime#91987

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 24, 2026 12:36
@rolfbjarne

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 3 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a CI-only workaround in the build system to avoid the NuGet first-time migrations race condition described in dotnet/runtime#91987 by forcing a single dotnet --info invocation immediately after the SDK download/installation step.

Changes:

  • Run dotnet --info once after installing the local SDK to complete first-time NuGet migrations.
  • Gate the workaround behind the ACES environment variable to only run on CI/ACES agents.

Comment thread builds/Makefile Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@rolfbjarne

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 3 pipeline(s).

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@rolfbjarne

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 3 pipeline(s).

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

✅ [PR Build #3dbdd61] Build passed (Detect API changes) ✅

Pipeline on Agent
Hash: 3dbdd61d232fe644cad9827144f44382db28bf9f [PR build]

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

🔥 [CI Build #3dbdd61] Build failed (Detect API changes) 🔥

Build failed for the job 'Detect API changes' (with job status 'Failed')

Pipeline on Agent
Hash: 3dbdd61d232fe644cad9827144f44382db28bf9f [PR build]

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

✅ [CI Build #3dbdd61] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 3dbdd61d232fe644cad9827144f44382db28bf9f [PR build]

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

✅ API diff for current PR / commit

NET (empty diffs)

✅ API diff vs stable

NET (empty diffs)

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: 3dbdd61d232fe644cad9827144f44382db28bf9f [PR build]

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

✅ [CI Build #3dbdd61] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: 3dbdd61d232fe644cad9827144f44382db28bf9f [PR build]

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

❌ [CI Build #3dbdd61] Prepare .NET Release failed ❌

The following jobs reported a non-successful result:

Job Result
Sign NuGets ✅ Succeeded
Convert NuGet to MSI ❌ Failed
Push NuGets ⏭️ Skipped
📦 Signed NuGet packages (32 packages)

iOS

  • Microsoft.iOS.Ref.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.iOS.Runtime.ios-arm64.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.iOS.Runtime.ios.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.iOS.Runtime.iossimulator-arm64.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.iOS.Runtime.iossimulator-x64.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.iOS.Sdk.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.iOS.Templates.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.iOS.Windows.Sdk.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.NET.Sdk.iOS.Manifest-10.0.400-preview.0.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg

MacCatalyst

  • Microsoft.MacCatalyst.Ref.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.MacCatalyst.Runtime.maccatalyst-arm64.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.MacCatalyst.Runtime.maccatalyst-x64.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.MacCatalyst.Runtime.maccatalyst.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.MacCatalyst.Sdk.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.MacCatalyst.Templates.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.NET.Sdk.MacCatalyst.Manifest-10.0.400-preview.0.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg

macOS

  • Microsoft.macOS.Ref.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.macOS.Runtime.osx-arm64.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.macOS.Runtime.osx-x64.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.macOS.Runtime.osx.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.macOS.Sdk.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.macOS.Templates.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.NET.Sdk.macOS.Manifest-10.0.400-preview.0.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg

tvOS

  • Microsoft.NET.Sdk.tvOS.Manifest-10.0.400-preview.0.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.tvOS.Ref.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.tvOS.Runtime.tvos-arm64.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.tvOS.Runtime.tvos.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.tvOS.Runtime.tvossimulator-arm64.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.tvOS.Runtime.tvossimulator-x64.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.tvOS.Sdk.net10.0_26.5.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg
  • Microsoft.tvOS.Templates.26.5.10379-ci.release-test-workaround-nuget-migrations.nupkg

Other

  • Sharpie.Bind.Tool.26.5.0.379-ci.release-test-workaround-nuget-migrations.nupkg

Pipeline on Agent
Hash: 3dbdd61d232fe644cad9827144f44382db28bf9f [PR build]

@vs-mobiletools-engineering-service2

This comment has been minimized.

@rolfbjarne
rolfbjarne enabled auto-merge (squash) June 25, 2026 09:35
@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

🚀 [CI Build #3dbdd61] Test results 🚀

Test results

✅ All tests passed on VSTS: test results.

🎉 All 207 tests passed 🎉

Tests counts

✅ assembly-processing: All 1 tests passed. Html Report (VSDrops) Download
✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (iOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 4 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ linker (iOS): All 15 tests passed. Html Report (VSDrops) Download
✅ linker (MacCatalyst): All 15 tests passed. Html Report (VSDrops) Download
✅ linker (macOS): All 21 tests passed. Html Report (VSDrops) Download
✅ linker (tvOS): All 15 tests passed. Html Report (VSDrops) Download
✅ monotouch (iOS): All 20 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 19 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 20 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 20 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ windows: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Ventura (13): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

Linux build succeeded

Pipeline on Agent
Hash: 3dbdd61d232fe644cad9827144f44382db28bf9f [PR build]

@rolfbjarne
rolfbjarne merged commit 9651ecd into main Jun 25, 2026
62 of 64 checks passed
@rolfbjarne
rolfbjarne deleted the release-test/workaround-nuget-migrations branch June 25, 2026 09:53
rolfbjarne added a commit that referenced this pull request Jun 25, 2026
…e condition) (#25797)

Run 'dotnet --info' once right after downloading the .NET SDK to complete the first-time NuGet migrations logic. This avoids the race condition where multiple parallel dotnet processes attempt the first-time setup simultaneously, causing 'The system cannot open the device or file specified' errors.

The workaround only runs in CI (when the ACES environment variable is set).

Ref: dotnet/runtime#91987

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Rolf Bjarne Kvinge <rokvin@microsoft.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
rolfbjarne added a commit that referenced this pull request Jun 29, 2026
The previous workaround (from #25797) ran `dotnet --info` after downloading the SDK to trigger NuGet's first-time migration and avoid the race condition described in dotnet/runtime#91987. However, `dotnet --info` doesn't reliably trigger the migration code path -- the migration (`MigrationRunner.Run()`) is invoked from:

1. `DotnetFirstTimeUseConfigurer` in the dotnet CLI (first-use flow, which `--info` may bypass)
2. `NuGetSdkResolver` during MSBuild SDK resolution (only during build/restore)

This PR replaces the `dotnet --info` call with directly creating the NuGet migration marker file (`~/.local/share/NuGet/Migrations/1`). When this file exists, NuGet's `MigrationRunner` returns immediately without acquiring the "NuGet-Migrations" mutex -- completely eliminating the race condition.

The migration itself (`Migration1`) only cleans up old NuGet directories, which is a no-op on a fresh SDK installation, so skipping it is safe.

Ref: dotnet/runtime#91987
Ref: https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.Common/Migrations/MigrationRunner.cs

---
🤖 Pull request created by Copilot

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dalexsoto pushed a commit that referenced this pull request Jul 7, 2026
…n workaround (#25849) (#25960)

The previous workaround (from #25797) ran `dotnet --info` after
downloading the SDK to trigger NuGet's first-time migration and avoid
the race condition described in dotnet/runtime#91987. However, `dotnet
--info` doesn't reliably trigger the migration code path -- the
migration (`MigrationRunner.Run()`) is invoked from:

1. `DotnetFirstTimeUseConfigurer` in the dotnet CLI (first-use flow,
which `--info` may bypass)
2. `NuGetSdkResolver` during MSBuild SDK resolution (only during
build/restore)

This PR replaces the `dotnet --info` call with directly creating the
NuGet migration marker file (`~/.local/share/NuGet/Migrations/1`). When
this file exists, NuGet's `MigrationRunner` returns immediately without
acquiring the "NuGet-Migrations" mutex -- completely eliminating the
race condition.

The migration itself (`Migration1`) only cleans up old NuGet
directories, which is a no-op on a fresh SDK installation, so skipping
it is safe.

Ref: dotnet/runtime#91987
Ref:
https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.Common/Migrations/MigrationRunner.cs

---
🤖 Pull request created by Copilot

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rolfbjarne added a commit that referenced this pull request Jul 10, 2026
…und (#25996)

The previous workaround (from #25797) ran `dotnet --info` after downloading the SDK to trigger NuGet's first-time migration and avoid the race condition described in dotnet/runtime#91987. However, `dotnet --info` doesn't reliably trigger the migration code path -- the migration (`MigrationRunner.Run()`) is invoked from:

1. `DotnetFirstTimeUseConfigurer` in the dotnet CLI (first-use flow, which `--info` may bypass)
2. `NuGetSdkResolver` during MSBuild SDK resolution (only during build/restore)

This PR replaces the `dotnet --info` call with directly creating the NuGet migration marker file (`~/.local/share/NuGet/Migrations/1`). When this file exists, NuGet's `MigrationRunner` returns immediately without acquiring the "NuGet-Migrations" mutex -- completely eliminating the race condition.

The migration itself (`Migration1`) only cleans up old NuGet directories, which is a no-op on a fresh SDK installation, so skipping it is safe.

Ref: dotnet/runtime#91987
Ref:
https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.Common/Migrations/MigrationRunner.cs

---
🤖 Pull request created by Copilot

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Backport of #25960.
Backport of #25849.

---------

Co-authored-by: Rolf Bjarne Kvinge <rokvin@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants