NotSupportedException & InvalidOperationException for unknown reasons #459
Description
I'm lumping these two bugs into one, because I suspect they have the same root cause.
.NET Standard 2.1
Running in Azure AKS (linux)
System.Data.SqlClient version 4.8.1
Connecting to a tiny little sql azure database instance
We've run into two issues:
Issue 1
InvalidOperationException: Invalid operation. The connection is closed.
^-- FOREVER!
Our code is exceedingly simple (flattened):
using var connection = new SqlConnection(ConnectionString);
await connection.OpenAsync(cancellationToken).ConfigureAwait(false);
using var cmd = connection.CreateCommand();
cmd.CommandText = command;
using var r = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
// consume reader results
// end of method
Stack trace:
System.InvalidOperationException:
at System.Data.SqlClient.TdsParserStateObject.WritePacket (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.TdsParserStateObject.WriteBytes (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.TdsParser.WriteString (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.TdsParser.TdsExecuteSQLBatch (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.SqlCommand.BeginExecuteReader (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Data.SqlClient.SqlCommand.ExecuteReaderAsync (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at [our code]
Sometimes, it gets stuck in an infinite loop of doing that. In other words, every attempt to execute this code would throw that same exception.
We fixed it by wrapping it like this:
try
{
...
}
catch (InvalidOperationException)
{
SqlConnection.ClearAllPools();
throw;
}
To me, it looks like a SqlConnection gets into an unstable state, gets put back into the pool and then explodes every time it's taken from the pool.
I'd say it looks like a race condition with a forgotten using statement or something... but... no. There aren't even any concurrent calls happening.
Issue 2
NotSupportedException: A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 35 - An internal exception was caught) The WriteAsync method cannot be called when another write operation is pending.
Stack trace:
System.Data.SqlClient.SqlException:
at System.Data.SqlClient.TdsParserStateObject.SNIWritePacket (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.TdsParserStateObject.WriteSni (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.TdsParserStateObject.WritePacket (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.TdsParserStateObject.ExecuteFlush (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.TdsParser.TdsExecuteSQLBatch (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Data.SqlClient.SqlCommand.BeginExecuteReader (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Data.SqlClient.SqlCommand.ExecuteReaderAsync (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Symetria.Sql.Core.SqlBase+<ExecuteQueryAsync>d__9.MoveNext (Symetria.Sql.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullSymetria.Sql.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /src/Symetria.Sql.Core/SqlBase.csSymetria.Sql.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 96)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at [our code ... same as above]
Our application insights shows 183 exceptions of Issue 1 and only 4 exceptions of this Issue 2 over the past 24 hours.
I'm somewhat at a loss on how to debug this or provide more useful information. Open to suggestions.
