Fix `System.InvalidCastException` in Grpc during Header Removal#7975

bouwkast · 2025-12-18T16:06:18Z

Summary of changes

This resolves a System.InvalidCastException that was happening in the GRPC client instrumentation (I think just GrpcLegacy) when we attempted to remove headers.

Reason for change

Errors in Error Tracking

Implementation details

before

public void Remove(string name)
{
    var entry = _headers.Get(name);
    if (entry is not null)
    {
        _headers.Remove(name);
    }
}

after

public void Remove(string name)
{
    var entry = _headers.Get(name);
    if (entry is not null)
    {
        _headers.Remove(entry);
    }
}

Test coverage

Updated the sample application to detect this and if it is detected it adds a tag which causes the tests to fail (and they did fail before I applied the fix).

Other details

Used #7961 /analyze-error command based on the reported error in Error Tracking. Seemed to work okay, it identified the issue, follow up was able to get a reproduction quickly, but it really struggled to get the tests updated.

Helpfully, the command output a document that linked to and copied the actual Grpc.Core.Api code that we hit 👍
Saves some time searching on GitHub ourselves I guess.

This is a bit of an edge case here so I don't think very wide spread.

Reproduced Error:

2025-12-18 10:03:42.678 -05:00 [ERR] Error creating inactive client span for GRPC call System.InvalidCastException: Unable to cast object of type 'System.String' to type 'Entry'.
   at Grpc_Core_Api__D754F35622E28BAD.Grpc_Core_Metadata.Datadog_Trace_ClrProfiler_AutoInstrumentation_Grpc_IMetadata_3.Remove(Object )
   at Datadog.Trace.ClrProfiler.AutoInstrumentation.Grpc.MetadataHeadersCollection.Remove(String name)
   at Datadog.Trace.ClrProfiler.AutoInstrumentation.Grpc.MetadataHeadersCollection.Set(String name, String value)
   at Datadog.Trace.Headers.HeadersCollectionAccesor`1.Set(TCarrier carrier, String key, String value)
   at Datadog.Trace.Propagators.W3CTraceContextPropagator.Inject[TCarrier,TCarrierSetter](PropagationContext context, TCarrier carrier, TCarrierSetter carrierSetter)
   at Datadog.Trace.Propagators.SpanContextPropagator.Inject[TCarrier,TCarrierSetter](PropagationContext context, TCarrier carrier, TCarrierSetter carrierSetter)
   at Datadog.Trace.Propagators.SpanContextPropagator.Inject[TCarrier](PropagationContext context, TCarrier headers)
   at Datadog.Trace.ClrProfiler.AutoInstrumentation.Grpc.GrpcLegacy.Client.GrpcLegacyClientCommon.InjectHeaders[TMethod,TCallOptions](Tracer tracer, TMethod method, TCallOptions& callOptionsInstance)
 { MachineName: ".", Process: "[84684 Samples.GrpcLegacy]", AppDomain: "[1 Samples.GrpcLegacy.exe]", TracerVersion: "3.34.0.0" }

andrewlock

Nice, thanks!

tracer/test/test-applications/integrations/Samples.GrpcLegacy/ClientWorker.cs

NachoEchevarria

Nice!

.Get isn't available in ealier versions of Grpc

Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

dd-trace-dotnet-ci-bot · 2025-12-19T15:05:43Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (7975) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	68.24 ± (68.22 - 68.52) ms	68.42 ± (68.42 - 68.61) ms	+0.3%	✅⬆️
.NET Framework 4.8 - Bailout
duration	72.06 ± (71.95 - 72.17) ms	72.08 ± (72.01 - 72.23) ms	+0.0%	✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1004.50 ± (1003.46 - 1008.79) ms	1005.64 ± (1007.23 - 1013.75) ms	+0.1%	✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms	21.91 ± (21.87 - 21.94) ms	22.07 ± (22.03 - 22.10) ms	+0.7%	✅⬆️
process.time_to_main_ms	78.62 ± (78.49 - 78.75) ms	79.33 ± (79.17 - 79.50) ms	+0.9%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.94 ± (10.94 - 10.95) MB	10.91 ± (10.91 - 10.91) MB	-0.3%	✅
runtime.dotnet.threads.count	12 ± (12 - 12)	12 ± (12 - 12)	+0.0%	✅
.NET Core 3.1 - Bailout
process.internal_duration_ms	21.78 ± (21.75 - 21.81) ms	22.00 ± (21.97 - 22.03) ms	+1.0%	✅⬆️
process.time_to_main_ms	79.86 ± (79.77 - 79.96) ms	80.38 ± (80.27 - 80.50) ms	+0.7%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.96 ± (10.95 - 10.96) MB	10.94 ± (10.93 - 10.94) MB	-0.2%	✅
runtime.dotnet.threads.count	13 ± (13 - 13)	13 ± (13 - 13)	+0.0%	✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	212.79 ± (210.56 - 215.02) ms	214.42 ± (213.13 - 215.72) ms	+0.8%	✅⬆️
process.time_to_main_ms	472.07 ± (471.56 - 472.58) ms	478.67 ± (477.71 - 479.63) ms	+1.4%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	48.05 ± (48.03 - 48.07) MB	48.09 ± (48.07 - 48.11) MB	+0.1%	✅⬆️
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	+0.0%	✅
.NET 6 - Baseline
process.internal_duration_ms	20.73 ± (20.70 - 20.76) ms	20.87 ± (20.84 - 20.89) ms	+0.7%	✅⬆️
process.time_to_main_ms	68.31 ± (68.19 - 68.42) ms	70.07 ± (69.91 - 70.22) ms	+2.6%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.61 ± (10.60 - 10.61) MB	10.64 ± (10.64 - 10.64) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 6 - Bailout
process.internal_duration_ms	20.57 ± (20.54 - 20.60) ms	20.86 ± (20.84 - 20.88) ms	+1.4%	✅⬆️
process.time_to_main_ms	69.00 ± (68.93 - 69.07) ms	70.85 ± (70.69 - 71.01) ms	+2.7%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.65 ± (10.65 - 10.66) MB	10.68 ± (10.68 - 10.68) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	199.93 ± (198.97 - 200.89) ms	204.08 ± (202.95 - 205.22) ms	+2.1%	✅⬆️
process.time_to_main_ms	439.18 ± (438.69 - 439.67) ms	449.08 ± (447.82 - 450.33) ms	+2.3%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	48.46 ± (48.41 - 48.51) MB	48.38 ± (48.32 - 48.43) MB	-0.2%	✅
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	+0.0%	✅⬆️
.NET 8 - Baseline
process.internal_duration_ms	18.81 ± (18.78 - 18.84) ms	19.65 ± (19.59 - 19.71) ms	+4.4%	✅⬆️
process.time_to_main_ms	67.14 ± (67.03 - 67.25) ms	74.68 ± (74.16 - 75.19) ms	+11.2%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.69 ± (7.68 - 7.70) MB	7.68 ± (7.68 - 7.69) MB	-0.1%	✅
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 8 - Bailout
process.internal_duration_ms	18.76 ± (18.74 - 18.79) ms	19.55 ± (19.48 - 19.62) ms	+4.2%	✅⬆️
process.time_to_main_ms	67.98 ± (67.92 - 68.05) ms	75.30 ± (74.66 - 75.94) ms	+10.8%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.71 ± (7.70 - 7.71) MB	7.75 ± (7.74 - 7.76) MB	+0.5%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	179.22 ± (178.33 - 180.11) ms	194.45 ± (193.29 - 195.61) ms	+8.5%	✅⬆️
process.time_to_main_ms	424.06 ± (423.44 - 424.68) ms	461.11 ± (459.43 - 462.80) ms	+8.7%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	36.22 ± (36.19 - 36.26) MB	36.58 ± (36.54 - 36.62) MB	+1.0%	✅⬆️
runtime.dotnet.threads.count	27 ± (27 - 27)	27 ± (27 - 27)	-0.2%	✅

HttpMessageHandler

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	193.63 ± (193.67 - 194.58) ms	194.14 ± (193.90 - 194.92) ms	+0.3%	✅⬆️
.NET Framework 4.8 - Bailout
duration	196.83 ± (196.81 - 197.46) ms	197.49 ± (197.37 - 197.77) ms	+0.3%	✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1110.81 ± (1113.29 - 1120.74) ms	1113.26 ± (1122.52 - 1132.61) ms	+0.2%	✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms	188.64 ± (188.22 - 189.05) ms	188.20 ± (187.85 - 188.55) ms	-0.2%	✅
process.time_to_main_ms	81.10 ± (80.86 - 81.34) ms	80.98 ± (80.75 - 81.21) ms	-0.1%	✅
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.10 ± (16.07 - 16.12) MB	16.14 ± (16.11 - 16.16) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	20 ± (20 - 20)	20 ± (20 - 20)	-0.7%	✅
.NET Core 3.1 - Bailout
process.internal_duration_ms	187.59 ± (187.30 - 187.88) ms	188.08 ± (187.71 - 188.45) ms	+0.3%	✅⬆️
process.time_to_main_ms	82.08 ± (81.93 - 82.24) ms	82.65 ± (82.50 - 82.81) ms	+0.7%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.11 ± (16.09 - 16.14) MB	16.11 ± (16.08 - 16.14) MB	-0.0%	✅
runtime.dotnet.threads.count	21 ± (20 - 21)	21 ± (20 - 21)	+0.3%	✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	395.02 ± (392.66 - 397.38) ms	401.68 ± (398.85 - 404.51) ms	+1.7%	✅⬆️
process.time_to_main_ms	475.17 ± (474.45 - 475.89) ms	476.49 ± (475.85 - 477.13) ms	+0.3%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	58.51 ± (58.36 - 58.66) MB	58.52 ± (58.37 - 58.66) MB	+0.0%	✅⬆️
runtime.dotnet.threads.count	29 ± (29 - 29)	29 ± (29 - 29)	-0.0%	✅
.NET 6 - Baseline
process.internal_duration_ms	192.69 ± (192.30 - 193.08) ms	192.45 ± (192.09 - 192.80) ms	-0.1%	✅
process.time_to_main_ms	70.17 ± (70.00 - 70.35) ms	70.40 ± (70.22 - 70.59) ms	+0.3%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.04 ± (15.90 - 16.18) MB	16.07 ± (15.93 - 16.22) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	19 ± (18 - 19)	19 ± (19 - 19)	+0.4%	✅⬆️
.NET 6 - Bailout
process.internal_duration_ms	191.54 ± (191.26 - 191.81) ms	191.39 ± (191.08 - 191.71) ms	-0.1%	✅
process.time_to_main_ms	70.92 ± (70.80 - 71.03) ms	70.96 ± (70.86 - 71.07) ms	+0.1%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.05 ± (15.89 - 16.21) MB	16.20 ± (16.07 - 16.33) MB	+1.0%	✅⬆️
runtime.dotnet.threads.count	19 ± (19 - 19)	20 ± (19 - 20)	+2.6%	✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	407.50 ± (405.48 - 409.52) ms	412.44 ± (410.03 - 414.85) ms	+1.2%	✅⬆️
process.time_to_main_ms	445.84 ± (445.25 - 446.44) ms	445.99 ± (445.36 - 446.62) ms	+0.0%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	58.97 ± (58.83 - 59.12) MB	59.39 ± (59.28 - 59.49) MB	+0.7%	✅⬆️
runtime.dotnet.threads.count	30 ± (29 - 30)	30 ± (29 - 30)	-0.1%	✅
.NET 8 - Baseline
process.internal_duration_ms	191.03 ± (190.64 - 191.41) ms	190.52 ± (190.15 - 190.90) ms	-0.3%	✅
process.time_to_main_ms	70.14 ± (69.92 - 70.35) ms	69.70 ± (69.53 - 69.87) ms	-0.6%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.69 ± (11.66 - 11.71) MB	11.73 ± (11.70 - 11.76) MB	+0.4%	✅⬆️
runtime.dotnet.threads.count	18 ± (18 - 18)	18 ± (18 - 18)	+0.4%	✅⬆️
.NET 8 - Bailout
process.internal_duration_ms	190.15 ± (189.87 - 190.43) ms	190.55 ± (190.24 - 190.86) ms	+0.2%	✅⬆️
process.time_to_main_ms	70.76 ± (70.64 - 70.89) ms	70.87 ± (70.75 - 70.99) ms	+0.1%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.79 ± (11.76 - 11.82) MB	11.82 ± (11.80 - 11.85) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	19 ± (19 - 19)	19 ± (19 - 19)	-0.3%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	365.92 ± (364.30 - 367.54) ms	365.85 ± (364.51 - 367.18) ms	-0.0%	✅
process.time_to_main_ms	428.91 ± (428.12 - 429.70) ms	431.17 ± (430.50 - 431.84) ms	+0.5%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	47.90 ± (47.86 - 47.93) MB	47.98 ± (47.96 - 48.01) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	29 ± (29 - 29)	29 ± (29 - 29)	+0.1%	✅⬆️

Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts

FakeDbCommand (.NET Framework 4.8)

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7975) - mean (69ms)  : 67, 70
    master - mean (68ms)  : 66, 70

    section Bailout
    This PR (7975) - mean (72ms)  : 71, 73
    master - mean (72ms)  : 71, 73

    section CallTarget+Inlining+NGEN
    This PR (7975) - mean (1,010ms)  : 963, 1058
    master - mean (1,006ms)  : 968, 1044

FakeDbCommand (.NET Core 3.1)

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7975) - mean (107ms)  : 103, 110
    master - mean (106ms)  : 103, 108

    section Bailout
    This PR (7975) - mean (108ms)  : 106, 109
    master - mean (107ms)  : 105, 108

    section CallTarget+Inlining+NGEN
    This PR (7975) - mean (723ms)  : 690, 756
    master - mean (712ms)  : 678, 746

FakeDbCommand (.NET 6)

gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7975) - mean (96ms)  : 93, 99
    master - mean (94ms)  : 92, 96

    section Bailout
    This PR (7975) - mean (97ms)  : 94, 99
    master - mean (94ms)  : 93, 95

    section CallTarget+Inlining+NGEN
    This PR (7975) - mean (681ms)  : 646, 716
    master - mean (667ms)  : 643, 690

FakeDbCommand (.NET 8)

gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7975) - mean (103ms)  : 94, 113
    master - mean (92ms)  : 90, 94

    section Bailout
    This PR (7975) - mean (104ms)  : crit, 89, 119
    master - mean (93ms)  : 92, 94

    section CallTarget+Inlining+NGEN
    This PR (7975) - mean (690ms)  : crit, 632, 749
    master - mean (630ms)  : 614, 646

HttpMessageHandler (.NET Framework 4.8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7975) - mean (194ms)  : 189, 200
    master - mean (194ms)  : 190, 199

    section Bailout
    This PR (7975) - mean (198ms)  : 196, 199
    master - mean (197ms)  : 194, 200

    section CallTarget+Inlining+NGEN
    This PR (7975) - mean (1,128ms)  : 1049, 1206
    master - mean (1,117ms)  : 1064, 1170

HttpMessageHandler (.NET Core 3.1)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7975) - mean (278ms)  : 274, 282
    master - mean (278ms)  : 272, 284

    section Bailout
    This PR (7975) - mean (279ms)  : 275, 283
    master - mean (278ms)  : 274, 282

    section CallTarget+Inlining+NGEN
    This PR (7975) - mean (917ms)  : 873, 960
    master - mean (909ms)  : 867, 950

HttpMessageHandler (.NET 6)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7975) - mean (272ms)  : 267, 277
    master - mean (271ms)  : 266, 277

    section Bailout
    This PR (7975) - mean (271ms)  : 266, 275
    master - mean (270ms)  : 267, 274

    section CallTarget+Inlining+NGEN
    This PR (7975) - mean (894ms)  : 850, 937
    master - mean (888ms)  : 847, 929

HttpMessageHandler (.NET 8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7975) - mean (270ms)  : 263, 277
    master - mean (271ms)  : 265, 277

    section Bailout
    This PR (7975) - mean (271ms)  : 267, 274
    master - mean (270ms)  : 267, 274

    section CallTarget+Inlining+NGEN
    This PR (7975) - mean (828ms)  : 807, 849
    master - mean (825ms)  : 808, 842

Fix GrpcLegacy Header Removal bug

d7fc905

bouwkast requested review from a team as code owners December 18, 2025 16:06

bouwkast added type:bug identified-by:telemetry AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos labels Dec 18, 2025

andrewlock approved these changes Dec 19, 2025

View reviewed changes

tracer/test/test-applications/integrations/Samples.GrpcLegacy/ClientWorker.cs Outdated Show resolved Hide resolved

NachoEchevarria approved these changes Dec 19, 2025

View reviewed changes

bouwkast and others added 2 commits December 19, 2025 09:13

Replace Get with a LINQ call

c1257c6

.Get isn't available in ealier versions of Grpc

Update log message

13d5b19

Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

bouwkast merged commit 36ac459 into master Dec 19, 2025
151 checks passed

bouwkast deleted the steven/fix-grcp-header-removal branch December 19, 2025 18:54

github-actions bot added this to the vNext-v3 milestone Dec 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `System.InvalidCastException` in Grpc during Header Removal#7975