Fix ALPN protocol list size field type and add boundary tests

[rzikm]

Summary

Fix Sec_Application_Protocols.ProtocolListSize field type from short to ushort to match the native Windows SEC_APPLICATION_PROTOCOL_LIST struct (USHORT), and add tests for ALPN list size validation.

Changes

Bug fix

• Interop.Sec_Application_Protocols.cs: Changed ProtocolListSize from short to ushort and updated the aggregate size limit from short.MaxValue (32,767) to ushort.MaxValue (65,535), aligning with both the native Windows API and the RFC 7301 TLS wire format.

Tests

• SslStreamAlpnTests.cs: Added SslStream_StreamToStream_AlpnListTotalSizeExceedsLimit_Throws — verifies that exceeding the 65,535-byte aggregate ALPN list limit throws:
  • ArgumentException on Windows (managed validation in GetProtocolLength())
  • AuthenticationException on Linux/FreeBSD (OpenSSL fails during ClientHello construction)
• SslApplicationProtocolTests.cs: Added boundary tests for individual protocol sizes (0, 1, 254, 255, 256, 512 bytes) via both byte[] and string constructors.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR fixes a type mismatch in the Windows SChannel interop layer where Sec_Application_Protocols.ProtocolListSize was incorrectly defined as short instead of ushort, aligning it with the native Windows SEC_APPLICATION_PROTOCOL_LIST structure and RFC 7301 specifications. The fix increases the maximum aggregate ALPN list size from 32,767 to 65,535 bytes and adds comprehensive tests to validate both individual protocol size limits and aggregate list size limits.

Changes:

• Fixed ProtocolListSize field type from short to ushort in the Windows SChannel interop struct
• Updated aggregate ALPN list size validation limit from short.MaxValue (32,767) to ushort.MaxValue (65,535)
• Added unit tests for individual protocol size boundaries (0, 1, 254, 255, 256, 512 bytes)
• Added functional test verifying aggregate ALPN list size limit enforcement with platform-specific exception handling

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
Interop.Sec_Application_Protocols.cs	Changed ProtocolListSize field from short to ushort and updated all casts and comparisons to use ushort.MaxValue instead of short.MaxValue
SslApplicationProtocolTests.cs	Added two theory-based boundary tests for individual protocol size validation via both byte array and string constructors
SslStreamAlpnTests.cs	Added functional test for aggregate ALPN list size limit, verifying ArgumentException on Windows and AuthenticationException on Linux/FreeBSD

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:285

• The explicit server.Dispose() on line 285 will be followed by an implicit Dispose when the using statement exits, resulting in double disposal. While Dispose is idempotent and should be safe, this is redundant and could mask issues.

Since the Dispose is being used to force the server task to complete (by closing the underlying stream), consider restructuring the test to avoid the double disposal. One approach is to move the Dispose call outside the using block, or to rely on the using statement's implicit disposal after awaiting the server task. Alternatively, document why the explicit Dispose is necessary (to unblock the server that's waiting for client data that will never arrive).

                server.Dispose();

src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:284

• The PR description states that Linux/FreeBSD will throw AuthenticationException when OpenSSL fails during ClientHello construction. However, the code changes add managed validation to all platforms (Unix in Interop.Ssl.cs lines 256-259, macOS in both SafeDeleteSslContext.cs and SafeDeleteNwContext.cs, and Android in SafeDeleteSslContext.cs lines 326-337). All platforms now throw ArgumentException before calling native APIs, making the behavior consistent across all platforms.

The test correctly expects ArgumentException, but the PR description should be updated to reflect that all platforms now throw ArgumentException due to managed validation, not just Windows.

        [ConditionalFact(nameof(BackendSupportsAlpn))]
        public async Task SslStream_StreamToStream_AlpnListTotalSizeExceedsLimit_Throws()
        {
            // Each protocol is 255 bytes, serialized with a 1-byte length prefix = 256 bytes each.
            // Per RFC 7301, TLS wire format limits ProtocolNameList to 2^16-1 (65,535) bytes.
            // All platforms enforce this via managed validation before calling native APIs.
            // 256 * 256 = 65,536 > 65,535
            const int protocolCount = 256;
            List<SslApplicationProtocol> oversizedProtocols = new List<SslApplicationProtocol>();
            for (int i = 0; i < protocolCount; i++)
            {
                byte[] proto = new byte[255];
                proto.AsSpan().Fill((byte)'a');
                proto[0] = (byte)((i >> 8) + 1);
                proto[1] = (byte)(i & 0xFF);
                oversizedProtocols.Add(new SslApplicationProtocol(proto));
            }

            using X509Certificate2 certificate = Configuration.Certificates.GetServerCertificate();

            SslClientAuthenticationOptions clientOptions = new SslClientAuthenticationOptions
            {
                ApplicationProtocols = oversizedProtocols,
                RemoteCertificateValidationCallback = delegate { return true; },
                TargetHost = Guid.NewGuid().ToString("N"),
            };

            SslServerAuthenticationOptions serverOptions = new SslServerAuthenticationOptions
            {
                ApplicationProtocols = new List<SslApplicationProtocol> { SslApplicationProtocol.Http2 },
                ServerCertificateContext = SslStreamCertificateContext.Create(certificate, null)
            };

            (Stream clientStream, Stream serverStream) = TestHelper.GetConnectedStreams();
            using (clientStream)
            using (serverStream)
            using (var client = new SslStream(clientStream, false))
            using (var server = new SslStream(serverStream, false))
            {
                Task serverTask = server.AuthenticateAsServerAsync(TestAuthenticateAsync, serverOptions);
                await Assert.ThrowsAsync<ArgumentException>(() => client.AuthenticateAsClientAsync(TestAuthenticateAsync, clientOptions));