[jitstress] HardwareIntrinsics_ro fails with "process cannot access the file" error

[jakobbotsch]

Seems to be failing widely.

01:45:37.661 Passed test: global::JIT.HardwareIntrinsics.X86._Ssse3.Program.SignInt32()
01:45:37.661 Running test: JIT/HardwareIntrinsics/General/Vector256_1/Vector256_1_ro/Vector256_1_ro.cmd
App Exit Code: 259
Expected: 100
Actual: 259
END EXECUTION - FAILED
FAILED
Unhandled exception. System.IO.IOException: The process cannot access the file 'C:\h\w\AE5609E1\w\B7AD09EA\e\JIT\HardwareIntrinsics\HardwareIntrinsics_ro\HardwareIntrinsics_ro.testStats.csv' because it is being used by another process.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
   at System.IO.File.ReadLines(String path)
   at XUnitLogChecker.Main(String[] args) in D:\a\_work\1\s\src\tests\Common\XUnitLogChecker\XUnitLogChecker.cs:line 77

@ivdiazsa Could this be related to #78742? Could also likely be #83261.

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=201133
Build error leg or test failing: HardwareIntrinsics_ro.WorkItemExecution
Pull request: #83005

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "csv' because it is being used by another process",
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
204788	dotnet/runtime	Send tests to Helix (Unix)	Log	#83427
204784	dotnet/runtime	Send tests to Helix (Unix)	Log	#83050
204543	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	#82970
204345	dotnet/runtime	Send tests to Helix (Unix)	Log
204427	dotnet/runtime	Send tests to Helix (Unix)	Log	#83413
204428	dotnet/runtime	Send tests to Helix (Unix)	Log
204376	dotnet/runtime	Send tests to Helix (Unix)	Log	#83412
204340	dotnet/runtime	Send tests to Helix (Unix)	Log	#83411
204044	dotnet/runtime	Send tests to Helix (Unix)	Log	#83354
203973	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	#79283
203855	dotnet/runtime	Send tests to Helix (Unix)	Log	#83050
203876	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	#83337
203547	dotnet/runtime	Send tests to Helix (Unix)	Log	#83380
203800	dotnet/runtime	coreclr Pri0 Runtime Tests Run linux arm checked	Log	#83388
203765	dotnet/runtime	Send tests to Helix (Unix)	Log	#83299
203621	dotnet/runtime	coreclr Pri0 Runtime Tests Run linux arm checked	Log	#80154
203583	dotnet/runtime	coreclr Pri1 Runtime Tests Run linux x64 release	Log
203537	dotnet/runtime	Send tests to Helix (Unix)	Log	#82827
203341	dotnet/runtime	Send tests to Helix (Unix)	Log	#83354
203507	dotnet/runtime	Send tests to Helix (Unix)	Log	#82954
203473	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	#80154
203478	dotnet/runtime	Send tests to Helix (Unix)	Log	#83371
203421	dotnet/runtime	Send tests to Helix (Unix)	Log
203382	dotnet/runtime	coreclr Pri1 Runtime Tests Run linux x64 release	Log
203384	dotnet/runtime	coreclr Pri0 Runtime Tests Run Composite osx x64 checked	Log
203348	dotnet/runtime	Send tests to Helix (Unix)	Log
203352	dotnet/runtime	coreclr Pri0 Runtime Tests Run linux arm checked	Log	#80154
203328	dotnet/runtime	coreclr Pri0 Runtime Tests Run linux arm64 checked	Log	#79334
203275	dotnet/runtime	Send tests to Helix (Unix)	Log	#82045
203108	dotnet/runtime	Send tests to Helix (Unix)	Log	#82779
202951	dotnet/runtime	Build Tests	Log	#83356
202862	dotnet/runtime	Send tests to Helix (Unix)	Log	#82045
202821	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	#80154
202706	dotnet/runtime	Send tests to Helix (Unix)	Log
202648	dotnet/runtime	Send tests to Helix (Unix)	Log	#83334
202622	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	#82045
202588	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	#79283
202432	dotnet/runtime	Build Tests	Log	#79169
202394	dotnet/runtime	Send tests to Helix (Unix)	Log	#79169
202373	dotnet/runtime	Send tests to Helix (Unix)	Log	#83335
202281	dotnet/runtime	Send tests to Helix (Unix)	Log	#83300
202216	dotnet/runtime	Send tests to Helix (Unix)	Log	#83253
202211	dotnet/runtime	Send tests to Helix (Unix)	Log	#82049
202132	dotnet/runtime	Send tests to Helix (Unix)	Log	#83094
202065	dotnet/runtime	Send tests to Helix (Unix)	Log
202081	dotnet/runtime	Send tests to Helix (Unix)	Log
202049	dotnet/runtime	Send tests to Helix (Windows)	Log
202048	dotnet/runtime	coreclr Pri1 Runtime Tests Run linux arm64 release	Log
202046	dotnet/runtime	coreclr Pri0 Runtime Tests Run Composite osx arm64 checked	Log
202034	dotnet/runtime	Send tests to Helix (Unix)	Log

Build	Definition	Test	Pull Request
205173	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
205061	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
205036	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
204901	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
204829	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution	#83402
204444	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution	#83413
204050	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution	#83354
204232	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
204227	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution	#83402
203486	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
203487	dotnet/runtime	HardwareIntrinsics_r.WorkItemExecution
203369	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution	#83372
203349	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
203347	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution	#83354
203308	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
203201	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
203096	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution	#83354
203089	dotnet/runtime	global::JIT.HardwareIntrinsics.X86._Avx512F.Program.XorInt64()	#83354
202848	dotnet/runtime	HardwareIntrinsics_r.WorkItemExecution
202653	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
202124	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
202123	dotnet/runtime	HardwareIntrinsics_r.WorkItemExecution
202035	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
202030	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
201969	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
201955	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
201953	dotnet/runtime	HardwareIntrinsics_ro.WorkItemExecution
201952	dotnet/runtime	HardwareIntrinsics_r.WorkItemExecution	#83250
201883	dotnet/runtime	HardwareIntrinsics_r.WorkItemExecution	#82744

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
18	79	79

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This seems to be failing widely in jitsress runs now: https://dev.azure.com/dnceng-public/public/_build/results?buildId=201133&view=ms.vss-test-web.build-test-results-tab

Author:	jakobbotsch
Assignees:	-
Labels:	tenet-performance, area-CodeGen-coreclr, untriaged
Milestone:	-

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Seems to be failing widely.

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=201133
Build error leg or test failing: HardwareIntrinsics_ro.WorkItemExecution
Pull request: #83005

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "HardwareIntrinsics_",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

01:45:37.661 Passed test: global::JIT.HardwareIntrinsics.X86._Ssse3.Program.SignInt32()
01:45:37.661 Running test: JIT/HardwareIntrinsics/General/Vector256_1/Vector256_1_ro/Vector256_1_ro.cmd
App Exit Code: 259
Expected: 100
Actual: 259
END EXECUTION - FAILED
FAILED
Unhandled exception. System.IO.IOException: The process cannot access the file 'C:\h\w\AE5609E1\w\B7AD09EA\e\JIT\HardwareIntrinsics\HardwareIntrinsics_ro\HardwareIntrinsics_ro.testStats.csv' because it is being used by another process.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
   at System.IO.File.ReadLines(String path)
   at XUnitLogChecker.Main(String[] args) in D:\a\_work\1\s\src\tests\Common\XUnitLogChecker\XUnitLogChecker.cs:line 77

@ivdiazsa Could this be related to #78742?

Author:	jakobbotsch
Assignees:	-
Labels:	tenet-performance, area-CodeGen-coreclr, area-Infrastructure-coreclr, untriaged
Milestone:	-

[EgorBo]

@ivdiazsa could you please take a look if that was your PR, this broke many outerloop pipelines

[ivdiazsa]

I just checked the failures and the Linux ones seem to be because of the test taking too long. Do we expect these tests to last more than 5 minutes? If yes, then that's an adjustment to make to the watcher. If not, then we can confirm the watcher is doing its job of preventing potential freezes.

As for the Windows ones, I have no idea. Seems like potentially a problem with Windows liking to hold on to resources for whatever reason. However, it seems to be working on my machine so far. I will continue trying to repro it.

[JulieLeeMSFT]

CC @trylek @hoyosjs PTAL.