TLS handshake failures are not clear with SChannel

[vcsjones]

Given that you are using SslStream on Windows with .NET Core 3.1 (or .NET Framework), a TLS handshake failure presents itself as "The message received was unexpected or badly formatted.":

System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception.

---> System.ComponentModel.Win32Exception: The message received was unexpected or badly formatted
   --- End of inner exception stack trace ---
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar,
Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()

In actuality, this was a fatal TLS alert 40 that occurred because the client and server did not have any cipher suites in common.

This makes troubleshooting a little bit difficult and requires understanding how SChannel maps TLS alerts to SEC_E error codes. This is also complicated by SChannel mapping multiple TLS alerts to the same error code.

It would be very helpful if the exception included, or exposed, the actual handshake alert to make troubleshooting easier. Ideally, something along the lines of:

"TLS failed with fatal alert 40 (handshake_failure)"

---

I'm not clear at the moment what OpenSSL / SecureTransport does at the moment, but ideally troubleshooting TLS handshake issues is roughly the same on all platforms.

I'm also not clear if this is possible, if SChannel gives you the ability to get the raw alert. If not, perhaps there is still a way to improve the exception so people have a better time determining the root cause of the problem.

[davidsh]

@wfurt

[wfurt]

What version do you use @vcsjones?
Improving diagnostics is a goal for 5.0

[vcsjones]

@wfurt this was in 3.1, let me see what happens in a nightly.

[wfurt]

I was primarily thinking about dotnet/corefx#41967. That should make sure we at least process and send alerts. The exception still may be vague but the behavior may show more what is wrong. The next phase is to add more tracing and a description how to use it. If you can share repro I can turn it into test case.

[vcsjones]

@wfurt this is what I get with 5.0.100-preview.2.20154.8 on Windows Server 2012 R2 (yes, a bit old). Would you expect the behavior to vary much by Windows version?

System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.ComponentModel.Win32Exception (0x80090326): The message received was unexpected or badly formatted.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at Program.Main(String[] args)

If you can share repro I can turn it into test case.

The story behind this error is that we had a client with no AES-GCM ciphers enabled trying to talk to a server that had only AES-GCM ciphers enabled. The code is simple enough:

using System;
using System.Net.Security;
using System.Net.Sockets;
using System.Threading.Tasks;

public class Program {
    private static async Task Main(string[] args) {
        using (var client = new TcpClient("<domain>", 443))
        using (var stream = new SslStream(client.GetStream(), false)) {
            await stream.AuthenticateAsClientAsync("<domain>");
            Console.WriteLine(stream.RemoteCertificate.Subject);
        }
    }
}

The trick is running on a client that has no common cipher suite with the remote server. I'm not sure how easy it is to have a test set up like that which is very dependent on the server and client configuration.