Instrumented clrjit causes random msbuild failures

[EgorBo]

dotnet-optimization pipeline is currently blocked on this issue, I managed to make a minimal standalone repro (Linux-x64):

Create a repro.sh script with the following content:

#!/bin/bash
set -e

rm -rf repro
#SDK_TYPE=-pgo

# Step 1: download the latest dotnet SDK

mkdir repro
cd repro

wget -O dotnet-sdk.tar.gz https://dotnetbuilds.azureedge.net/public/Sdk/7.0.100-preview.4.22204.1/dotnet-sdk$SDK_TYPE-7.0.100-preview.4.22204.1-linux-x64.tar.gz
mkdir dotnet-sdk && tar -xvzf dotnet-sdk.tar.gz -C dotnet-sdk

# Step 2: download OrchardCore project 

git clone https://github.com/dotnet-perf-bot/orchardcore.git
cd orchardcore/src/OrchardCore.Cms.Web

# Step 3: Try to build OrchardCore with instrumented SDK:
DOTNET_MULTILEVEL_LOOKUP=0 DOTNET_SKIP_FIRST_TIME_EXPERIENC=1 DOTNET_CLI_TELEMETRY_OPTOUT=1 ../../../dotnet-sdk/dotnet build -c Release

and run it - it should complete without errors.

Now uncomment #SDK_TYPE=-pgo line to use natively instrumented SDK and run again - observe random meaningless msbuild failures. e.g.

/home/egorbo/prj/repro/dotnet-sdk/sdk/7.0.100-preview.4.22204.1/Current/Microsoft.Common.props(29,5): error MSB4186: 
Invalid static method invocation syntax: "[MSBuild]::GetDirectoryNameOfFileAbove($(MSBuildProjectDirectory), 
'$(_DirectoryBuildPropsFile)')". Method '[MSBuild]::GetDirectoryNameOfFileAbove' not found. Static method invocation should 
be of the form: $([FullTypeName]::Method()), e.g. $([System.IO.Path]::Combine(`a`, `b`)). Check that all parameters are defined,
 are of the correct type, and are specified in the right order. 
[/home/egorbo/prj/repro/orchardcore/src/OrchardCore/OrchardCore/OrchardCore.csproj]

After some experiments I came to conclusion that it's libclrjit.so binary that causes it - I downloaded both instrumented and normal sdk and only copied libclrjit.so from the pgo to normal one and removed pgo:

Replace clrjit from pgo

#!/bin/bash
set -e

rm -rf repro
# Step 1: download the latest dotnet SDK

mkdir repro
cd repro

wget -O dotnet-sdk-pgo.tar.gz https://dotnetbuilds.azureedge.net/public/Sdk/7.0.100-preview.4.22204.1/dotnet-sdk-pgo-7.0.100-preview.4.22204.1-linux-x64.tar.gz
wget -O dotnet-sdk.tar.gz https://dotnetbuilds.azureedge.net/public/Sdk/7.0.100-preview.4.22204.1/dotnet-sdk-7.0.100-preview.4.22204.1-linux-x64.tar.gz
mkdir dotnet-sdk && tar -xvzf dotnet-sdk.tar.gz -C dotnet-sdk
mkdir dotnet-sdk-pgo && tar -xvzf dotnet-sdk-pgo.tar.gz -C dotnet-sdk-pgo

cp -f dotnet-sdk-pgo/shared/Microsoft.NETCore.App/7.0.0-preview.4.22201.3/libclrjit.so dotnet-sdk/shared/Microsoft.NETCore.App/7.0.0-preview.4.22201.3/libclrjit.so
rm -rf dotnet-sdk-pgo

# Step 2: download OrchardCore project 

git clone https://github.com/dotnet-perf-bot/orchardcore.git
cd orchardcore/src/OrchardCore.Cms.Web

# Step 3: Try to build OrchardCore with instrumented SDK:
DOTNET_MULTILEVEL_LOOKUP=0 DOTNET_SKIP_FIRST_TIME_EXPERIENC=1 DOTNET_CLI_TELEMETRY_OPTOUT=1 ../../../dotnet-sdk/dotnet build -c Release orchardcore/src/OrchardCore.Cms.Web

cc @dotnet/jit-contrib @jkotas @davidwrighton while I try to diagnose it further maybe any advices what to check - can instrumentation inside the jit cause gc holes?

I tried to run with non-instrumented sdk and GCStress=0xC - no issues

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[AndyAyersMS]

can instrumentation inside the jit cause gc holes?

Seems unlikely that instrumentation itself could cause this, but instrumentation might expose latent bugs in the jit because of uninitialized references or something. If you run this version of the jit and the normal one under SPMI do you see any diffs?

Can you do partial instrumentation to see if you can pin down which part of the jit needs to be instrumented to cause the failures? This is possible (if a bit painful) with MSVC so maybe it's viable with LLVM too? (I presume we're talking about LLVM, right?)

Or perhaps instrumentation is altering timing somehow and exposing a latent bug or race either in the msbuild or libraries or even perhaps expose an existing gc hole.

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

---

dotnet-optimization pipeline is currently blocked on this issue, I managed to make a minimal standalone repro (Linux-x64):

Create a repro.sh script with the following content:

#!/bin/bash
set -e

rm -rf repro
#SDK_TYPE=-pgo

# Step 1: download the latest dotnet SDK

mkdir repro
cd repro

wget -O dotnet-sdk.tar.gz https://dotnetbuilds.azureedge.net/public/Sdk/7.0.100-preview.4.22204.1/dotnet-sdk$SDK_TYPE-7.0.100-preview.4.22204.1-linux-x64.tar.gz
mkdir dotnet-sdk && tar -xvzf dotnet-sdk.tar.gz -C dotnet-sdk

# Step 2: download OrchardCore project 

git clone https://github.com/dotnet-perf-bot/orchardcore.git
cd orchardcore/src/OrchardCore.Cms.Web

# Step 3: Try to build OrchardCore with instrumented SDK:
DOTNET_MULTILEVEL_LOOKUP=0 DOTNET_SKIP_FIRST_TIME_EXPERIENC=1 DOTNET_CLI_TELEMETRY_OPTOUT=1 ../../../dotnet-sdk/dotnet build -c Release

and run it - it should complete without errors.

Now uncomment #SDK_TYPE=-pgo line to use natively instrumented SDK and run again - observe random meaningless msbuild failures. e.g.

/home/egorbo/prj/repro/dotnet-sdk/sdk/7.0.100-preview.4.22204.1/Current/Microsoft.Common.props(29,5): error MSB4186: 
Invalid static method invocation syntax: "[MSBuild]::GetDirectoryNameOfFileAbove($(MSBuildProjectDirectory), 
'$(_DirectoryBuildPropsFile)')". Method '[MSBuild]::GetDirectoryNameOfFileAbove' not found. Static method invocation should 
be of the form: $([FullTypeName]::Method()), e.g. $([System.IO.Path]::Combine(`a`, `b`)). Check that all parameters are defined,
 are of the correct type, and are specified in the right order. 
[/home/egorbo/prj/repro/orchardcore/src/OrchardCore/OrchardCore/OrchardCore.csproj]

After some experiments I came to conclusion that it's libclrjit.so binary that causes it - I downloaded both instrumented and normal sdk and only copied libclrjit.so from the pgo to normal one and removed pgo:

Replace clrjit from pgo

#!/bin/bash
set -e

rm -rf repro
# Step 1: download the latest dotnet SDK

mkdir repro
cd repro

wget -O dotnet-sdk-pgo.tar.gz https://dotnetbuilds.azureedge.net/public/Sdk/7.0.100-preview.4.22204.1/dotnet-sdk-pgo-7.0.100-preview.4.22204.1-linux-x64.tar.gz
wget -O dotnet-sdk.tar.gz https://dotnetbuilds.azureedge.net/public/Sdk/7.0.100-preview.4.22204.1/dotnet-sdk-7.0.100-preview.4.22204.1-linux-x64.tar.gz
mkdir dotnet-sdk && tar -xvzf dotnet-sdk.tar.gz -C dotnet-sdk
mkdir dotnet-sdk-pgo && tar -xvzf dotnet-sdk-pgo.tar.gz -C dotnet-sdk-pgo

cp -f dotnet-sdk-pgo/shared/Microsoft.NETCore.App/7.0.0-preview.4.22201.3/libclrjit.so dotnet-sdk/shared/Microsoft.NETCore.App/7.0.0-preview.4.22201.3/libclrjit.so
rm -rf dotnet-sdk-pgo

# Step 2: download OrchardCore project 

git clone https://github.com/dotnet-perf-bot/orchardcore.git
cd orchardcore/src/OrchardCore.Cms.Web

# Step 3: Try to build OrchardCore with instrumented SDK:
DOTNET_MULTILEVEL_LOOKUP=0 DOTNET_SKIP_FIRST_TIME_EXPERIENC=1 DOTNET_CLI_TELEMETRY_OPTOUT=1 ../../../dotnet-sdk/dotnet build -c Release orchardcore/src/OrchardCore.Cms.Web

cc @dotnet/jit-contrib @jkotas @davidwrighton while I try to diagnose it further maybe any advices what to check - can instrumentation inside the jit cause gc holes?

I tried to run with non-instrumented sdk and GCStress=0xC - no issues

Author:	EgorBo
Assignees:	-
Labels:	area-CodeGen-coreclr, untriaged
Milestone:	-

[EgorBo]

Update:
The issue reproduces only on Linux-x64 with Release version of instrumented clrjit (built locally with ./build.sh Clr.Jit -c Release /p:PgoInstrument=true on various version of clang (last tested is 12.0)
Various stress test modes didn't help.
I tried partial instrumentation but with no luck so far - e.g. -fprofile-list https://clang.llvm.org/docs/UsersManual.html#cmdoption-fprofile-list

Trying spmi now...

[EgorBo]

With help of @jakobbotsch we managed to locate the problem - it's in importer_vectorization.cpp for

internal static bool IsWindowsComponent(AssemblyIdentity identity)
{
    return (identity.ContentType == AssemblyContentType.WindowsRuntime) &&
           identity.Name.StartsWith("windows.", StringComparison.OrdinalIgnoreCase);
}

it produces different codegen for JIT-Release vs Jit-Release-Instrumented: https://www.diffchecker.com/n7jnCp3y