Breakpoint changes the output of the program on x64

[tommcdon]

The output for the execution of a program may change if you add a breakpoint on a specified line.

Repro Steps:

static void Main(string[] args)
{
    Console.WriteLine(WrapDegrees(-722.5f));
    Console.WriteLine();
}

private static float WrapDegrees(float degrees)
{
    float remainder = Math.Abs(degrees);
    remainder %= 360; // breakpoint here
    return degrees < 0 ? 360 - remainder : remainder;
}

Code from: https://gist.github.com/tannergooding/af91efb30215ab360dfb3fe18e58fe36

1. Add breakpoint at specified line;
2. Target .NET Core 3.0 Preview 4 for x64;
3. Start debugging;
4. Hit breakpoint and continue execution.

Expected output
357.5

Actual output
357.51007

Remarks

• If you don't add any breakpoint (or add any breakpoint as long as the specified line does not have a breakpoint), the output will be the same as expected;
• Unable to reproduce the issue with v3.0.0-preview-27122-01 or older versions (regression);
• Unable to reproduce if targeting x86.

[tannergooding]

@tommcdon, is this just a duplicate of the internal DevDiv bug (853718)?

[cshung]

@tannergooding Yes

[cshung]

tl;dr - I narrowed this issue down to the fact that the debugger is not decoding the AVX instructions correctly. Below is my experiment that proves that fact.

1. Checkout my branch, build it for debug, x64

2. Create a .NET Core 3 console application with the repro code, publish it in self contained mode and patch the coreclr with the built binaries.

3. Debug the built exe using Visual Studio with breakpoint set on the indicated line and the Console.WriteLine(); line so that we can inspect the result.

4. While the 1st breakpoint hit, non-invasive attach WinDBG to the debuggee process and turn on inspection by running ?? coreclr!g_inspect = 1, and then detach WinDBG to allow further execution.

5. Continue the process in Visual Studio, the process should be infinitely looping on the first loop.

6. Non-invasive attach WinDBG to the debuggee process, Inspecting the code around the infinite loop, it should be clear that the debuggee understand this is an AVX instruction, but it deliberately setting opcode0 to 0xf so that the code will run into the 2 op-code branch (which is questionable ... why?). After inspecting, change RIP to get out the loop. (Some disassembly reading is required here), and then detach WinDBG to allow further execution, it should stuck on the second infinite loop.

7. Non-invasive attach WinDBG to the debuggee process, change note that the code is setting cbImmed to 1, but the instruction doesn't have an immediate value field, so this is wrong. Undo the effect of the wrong code by running ?? cbImmed = 0, stop inspection by running ?? coreclr!g_inspect = 0, and then detach WInDBG to allow further execution.

8. The process should hit the Console.WriteLine(); breakpoint and print out the correct result because we fixed the error due to the instruction decoding bug.