Intermittent failure in JIT/Directed/coverage/oldtests/tlstest_il_r/tlstest.dll

[markples]

Not much information in the logs:

Output:
08:16:17.751 Running test: JIT/Directed/coverage/oldtests/tlstest_il_r/tlstest.dll
08:16:18.001 Failed test: JIT/Directed/coverage/oldtests/tlstest_il_r/tlstest.dll

Comments:
{ "HelixJobId": "517d00ed-60bb-4043-9a15-6268bcfef42a", "HelixWorkItemName": "Directed_r" }

Error message:
Object reference not set to an instance of an object.

Stack trace:
at tls.tlstest.Main()
at Program.<<Main>$>g__TestExecutor2|0_1(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

Instances from refs/heads/main:
#20230411.8 • [browser][non-icu] HybridGlobalization not supported on WASI (#84629) coreclr windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open
#20230412.1 • [browser][non-icu] HybridGlobalization not supported on WASI (#84629) coreclr windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open
#20230412.3 • JIT: Add an in-release flag to enable physical promotion (#84689) coreclr windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open
#20230412.12 • Zlib: Update zlib to v1.2.13, intel-zlib to v1.2.13_jtk (#84602) coreclr windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open
#20230413.2 • Revert "[browser] Wasm SDK packed as a nuget package (#84082)" (#84743) coreclr windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open

I started looking at this upon hitting the failure in
#20230413.1 • Fix type initialization ordering under R2R
and finding it unlikely to be from that PR.

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Not much information in the logs:

Output:
08:16:17.751 Running test: JIT/Directed/coverage/oldtests/tlstest_il_r/tlstest.dll
08:16:18.001 Failed test: JIT/Directed/coverage/oldtests/tlstest_il_r/tlstest.dll

Comments:
{ "HelixJobId": "517d00ed-60bb-4043-9a15-6268bcfef42a", "HelixWorkItemName": "Directed_r" }

Error message:
Object reference not set to an instance of an object.

Stack trace:
at tls.tlstest.Main()
at Program.<<Main>$>g__TestExecutor2|0_1(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

Instances from refs/heads/main:
#20230411.8 • [browser][non-icu] HybridGlobalization not supported on WASI (#84629) coreclr windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open
#20230412.1 • [browser][non-icu] HybridGlobalization not supported on WASI (#84629) coreclr windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open
#20230412.3 • JIT: Add an in-release flag to enable physical promotion (#84689) coreclr windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open
#20230412.12 • Zlib: Update zlib to v1.2.13, intel-zlib to v1.2.13_jtk (#84602) coreclr windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open
#20230413.2 • Revert "[browser] Wasm SDK packed as a nuget package (#84082)" (#84743) coreclr windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open

I started looking at this upon hitting the failure in
#20230413.1 • Fix type initialization ordering under R2R
and finding it unlikely to be from that PR.

Author:	markples
Assignees:	-
Labels:	area-CodeGen-coreclr, blocking-outerloop
Milestone:	-

[EgorBo]

Can be the TLS PR? cc @kunalspathak

[trylek]

Apparently we have several non-deterministically failing tests on Windows arm; all that I saw fail with a nullref, either in the runtime when executing the test, or in Crossgen2 when compiling it. The Crossgen2 determinism tests frequently hit this but they are not the only ones, it seems to be sporadically affecting other random tests. I'm wondering whether this may indicate something like an arm-specific GC hole, I find it hard to imagine how this could be caused by a codegen bug.

[kunalspathak]

or in Crossgen2 when compiling it.

Can you share some links to them?

I'm wondering whether this may indicate something like an arm-specific GC hole

This test is clearly failing because we are not encoding the information of TLS indirection correctly, leading to something like this:

00010  202C           movs    r0, 44   ; <--- this should be TLS[44] 
000012  6800           ldr     r0, [r0]
000014  6900           ldr     r0, [r0+0x10]
000016  6C03           ldr     r3, [r0+0x40]

I will work on the fix.

[trylek]

Example run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=241282&view=results

Failed tests:

• crossgen2determinism
• coreroot_determinism
• BasicTestWithMcj
• runtimeeventsource
• BinderTracingTest.ResolutionFlow

[trylek]

Even though at the first glance some of the Crossgen2 tests fail with an AV, when you look at the detailed log, you can see the underlying nullref thrown somewhere inside the compiler.