The system cannot open the device or file specified 'NuGet-Migrations'

[lonix1]

Description

In a CI build, in the official sdk docker image, I run dotnet nuget locals all --clear, and get this:

RUN dotnet nuget locals all --clear
System.IO.IOException: The system cannot open the device or file specified. : 'NuGet-Migrations'
   at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
   at System.Threading.Mutex..ctor(Boolean initiallyOwned, String name)
   at NuGet.Common.Migrations.MigrationRunner.Run()
   at Microsoft.DotNet.Configurer.DotnetFirstTimeUseConfigurer.Configure()
   at Microsoft.DotNet.Cli.Program.ConfigureDotNetForFirstTimeUse(IFirstTimeUseNoticeSentinel firstTimeUseNoticeSentinel, IAspNetCertificateSentinel aspNetCertificateSentinel, IFileSentinel toolPathSentinel, Boolean isDotnetBeingInvokedFromNativeInstaller, DotnetFirstRunConfiguration dotnetFirstRunConfiguration, IEnvironmentProvider environmentProvider, Dictionary`2 performanceMeasurements)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)
ERROR: process "/bin/sh -c dotnet nuget locals all --clear" did not complete successfully: exit code: 1

Reproduction Steps

inside the sdk container:

RUN \
  export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 && \
  dotnet nuget locals all --clear

Expected behavior

No error

Actual behavior

Error

Regression?

uknown

Known Workarounds

none

Configuration

mcr.microsoft.com/dotnet/sdk:7.0.400-bookworm-slim-amd64

Other information

Might be related to NuGet/Home#12159 and #80619, but those have been locked.

If I remove that offending line, I can build the docker image. If I then run that image and do something as simple as dotnet nuget it will give the same error as above.

{
  "ErrorMessage": "The system cannot open the device or file specified. : 'NuGet-Migrations'",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=426482
Error message validated: The system cannot open the device or file specified. : 'NuGet-Migrations'
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 10/4/2023 1:44:58 AM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[lonix1]

Update: when I run the container as root, the problem disappears.

But even so, that error is misleading.

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

In a CI build, in the official sdk docker image, I run dotnet nuget locals all --clear, and get this:

RUN dotnet nuget locals all --clear
System.IO.IOException: The system cannot open the device or file specified. : 'NuGet-Migrations'
   at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
   at System.Threading.Mutex..ctor(Boolean initiallyOwned, String name)
   at NuGet.Common.Migrations.MigrationRunner.Run()
   at Microsoft.DotNet.Configurer.DotnetFirstTimeUseConfigurer.Configure()
   at Microsoft.DotNet.Cli.Program.ConfigureDotNetForFirstTimeUse(IFirstTimeUseNoticeSentinel firstTimeUseNoticeSentinel, IAspNetCertificateSentinel aspNetCertificateSentinel, IFileSentinel toolPathSentinel, Boolean isDotnetBeingInvokedFromNativeInstaller, DotnetFirstRunConfiguration dotnetFirstRunConfiguration, IEnvironmentProvider environmentProvider, Dictionary`2 performanceMeasurements)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)
ERROR: process "/bin/sh -c dotnet nuget locals all --clear" did not complete successfully: exit code: 1

Reproduction Steps

inside the sdk container:

RUN \
  export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 && \
  dotnet nuget locals all --clear

Expected behavior

No error

Actual behavior

Error

Regression?

uknown

Known Workarounds

none

Configuration

mcr.microsoft.com/dotnet/sdk:7.0.400-bookworm-slim-amd64

Other information

Might be related to NuGet/Home#12159 and #80619, but those have been locked.

If I remove that offending line, I can build the docker image. If I then run that image and do something as simple as dotnet nuget it will give the same error as above.

Author:	lonix1
Assignees:	-
Labels:	area-System.Threading, untriaged, needs-area-label
Milestone:	-

[jkotas]

cc @kouvel

[kouvel]

This could be similar to #80619. If the container get set up as root or some other user and then a different user uses the container, a workaround may be to delete the /tmp/.dotnet/shm and /tmp/.dotnet/lockfiles directories as root as the last step before switching to a different user. There is a fix that is targeting the next servicing versions for .NET 7 and .NET 6 that may help if it's the same issue.

But even so, that error is misleading.

There is an issue for that, which would hopefully be addressed soon: #89090

[lonix1]

Thanks. I'll try the new version when it's released to see if it fixes this.

Is it one of those fixes that's released every month or so?

[kouvel]

Is it one of those fixes that's released every month or so?

Yes, the change should be in the next version that is released, 6.0.23 and 7.0.12.

[carlossanlop]

@kouvel this might not be a System.Threading specific issue.

I'm also seeing this failure in 7.0 CI runs for JIT HardwareIntrinsics in WASM like this one (completely unrelated):

• 7.0 PR: Update branding to 7.0.13 #92970
• Queue: Build Browser wasm Linux Release AllSubsets_Mono_RuntimeTests
• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=426482&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552&l=37
• Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-92970-merge-dd168e95f4c34310ba/JIT.HardwareIntrinsics.X86.Aes/1/console.3ae4e55e.log?helixlogtype=result

Output (Expand):

Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 7.0.10)
  Discovering: JIT.HardwareIntrinsics.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.HardwareIntrinsics.XUnitWrapper (found 2 of 362 test cases)
  Starting:    JIT.HardwareIntrinsics.XUnitWrapper (parallel test collections = on, max threads = 2)
    JIT/HardwareIntrinsics/X86/Aes/Aes_r/Aes_r.sh [FAIL]
      System.IO.IOException: The system cannot open the device or file specified. : 'NuGet-Migrations'
         at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
         at System.Threading.Mutex..ctor(Boolean initiallyOwned, String name)
         at NuGet.Common.Migrations.MigrationRunner.Run()
         at Microsoft.DotNet.Configurer.DotnetFirstTimeUseConfigurer.Configure()
         at Microsoft.DotNet.Cli.Program.ConfigureDotNetForFirstTimeUse(IFirstTimeUseNoticeSentinel firstTimeUseNoticeSentinel, IAspNetCertificateSentinel aspNetCertificateSentinel, IFileSentinel toolPathSentinel, Boolean isDotnetBeingInvokedFromNativeInstaller, DotnetFirstRunConfiguration dotnetFirstRunConfiguration, IEnvironmentProvider environmentProvider, Dictionary`2 performanceMeasurements)
         at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
         at Microsoft.DotNet.Cli.Program.Main(String[] args)
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/B3D809D6/w/C9020AAE/uploads/Reports/JIT.HardwareIntrinsics/X86/Aes/Aes_r/Aes_r.output.txt
      Raw output:
      BEGIN EXECUTION
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/B3D809D6/p
      > /datadisks/disk1/work/B3D809D6/w/C9020AAE/e/JIT/HardwareIntrinsics/X86/Aes/Aes_r/Aes_r.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_HardwareIntrinsics._X86_Aes_Aes_r_Aes_r_._X86_Aes_Aes_r_Aes_r_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        System.IO.IOException: The system cannot open the device or file specified. : 'NuGet-Migrations'
           at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
           at System.Threading.Mutex..ctor(Boolean initiallyOwned, String name)
           at NuGet.Common.Migrations.MigrationRunner.Run()
           at Microsoft.DotNet.Configurer.DotnetFirstTimeUseConfigurer.Configure()
           at Microsoft.DotNet.Cli.Program.ConfigureDotNetForFirstTimeUse(IFirstTimeUseNoticeSentinel firstTimeUseNoticeSentinel, IAspNetCertificateSentinel aspNetCertificateSentinel, IFileSentinel toolPathSentinel, Boolean isDotnetBeingInvokedFromNativeInstaller, DotnetFirstRunConfiguration dotnetFirstRunConfiguration, IEnvironmentProvider environmentProvider, Dictionary`2 performanceMeasurements)
           at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
           at Microsoft.DotNet.Cli.Program.Main(String[] args)
        
        Return code:      1
        Raw output file:      /datadisks/disk1/work/B3D809D6/w/C9020AAE/uploads/Reports/JIT.HardwareIntrinsics/X86/Aes/Aes_r/Aes_r.output.txt
        Raw output:
        BEGIN EXECUTION
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/B3D809D6/p
        > /datadisks/disk1/work/B3D809D6/w/C9020AAE/e/JIT/HardwareIntrinsics/X86/Aes/Aes_r/Aes_r.sh
  Finished:    JIT.HardwareIntrinsics.XUnitWrapper
=== TEST EXECUTION SUMMARY ===

[kouvel]

My understanding was that some of these tests in WASM have a build portion that runs as part of the test run. The exception is being thrown under Microsoft.DotNet.Cli.Program.ConfigureDotNetForFirstTimeUse. It's a decent suspicion currently that it's the same issue, and once the SDK used in the CI is updated to patched versions hopefully these kind of issues would disappear.