Tracking Issue: QUIC testing

[jasnell]

QUIC Testing Check-list.. set up as a tracking issue so that folks can follow along / help if they want. This should also give folks a good idea of the size of the work required still to finish.

When discussing any particular item, please refer to the number for easier reference.

• 1.0.0 Core Protocol Implementation
  • 1.1.0 Endpoint Testing
    • 1.1.1 Endpoint creation with various configurations
    • 1.1.2 IPv4 binding
    • 1.1.3 IPv6 binding
    • 1.1.4 IPv6-only mode
      • TODO: The ergonomics of configuring for ipv6 are... not great. Need to revisit this part of the JS API.
    • 1.1.5 Dual-stack mode
    • 1.1.6 Random port allocation
    • 1.1.7 Specific port binding
    • 1.1.8 Port reuse scenarios
    • 1.1.9 Endpoint can be use for both client and server
  • 1.2.0 Endpoint lifecycle
    • 1.2.1 Graceful shutdown with active sessions
    • 1.2.2 Forced shutdown/destroy
      • 1.2.2.1 With no sessions
      • 1.2.2.2 With sessions, no streams
      • 1.2.2.3 With sessions and streams
    • 1.2.3 Cleanup of sessions on endpoint graceful close
    • 1.2.4 Async.dispose implementation
  • 1.3.0 Endpoint options validation
    • 1.3.1 UDP receive/send buffer sizes
    • 1.3.2 UDP TTL configuration
    • 1.3.3 Token expiration settings
    • 1.3.4 Retry token expiration
    • 1.3.5 Connection limits (per-host, total)
    • 1.3.6 Stateless reset limits
    • 1.3.7 Address LRU cache sizing
    • 1.3.8 Reset token secret handling
    • 1.3.9 Token secret handling
  • 1.4.0 Server mode
    • 1.4.1 Listen configuration
    • 1.4.2 Busy state handling
    • 1.4.3 Rejection of new connections when busy
    • 1.4.4 Multiple session handling
    • 1.4.5 Session acceptance callback
  • 1.5.0 Client mode
    • 1.5.1 Connect to remote endpoint
    • 1.5.2 Multiple concurrent connections
    • 1.5.3 Connection to same host/port
    • 1.5.4 Endpoint reuse across connections
  • 1.6.0 Packet handling
    • 1.6.1 Send packet queuing
    • 1.6.2 Receive packet processing
    • 1.6.3 Packet validation
    • 1.6.4 Malformed packet handling
    • 1.6.5 Version negotiation packets
    • 1.6.6 Retry packets
    • 1.6.7 Stateless reset packets
    • 1.6.8 Immediate connection close
  • 1.7.0 Statistics collection
    • 1.7.1 Bytes sent/received
    • 1.7.2 Packets sent/received
    • 1.7.3 Session counts (client/server)
    • 1.7.4 Server busy count
    • 1.7.5 Timestamp tracking
• 2.0.0 Session Testing
  • 2.1.0 Session establishment
    • 2.1.1 Client-initiated session
    • 2.1.2 Server-side session acceptance
    • 2.1.3 TLS handshake completion
    • 2.1.4 Handshake timeout enforcement
    • 2.1.5 Certificate validation
    • 2.1.6 ALPN negotiation
    • 2.1.7 SNI handling
    • 2.1.8 Early data (0-RTT) support
  • 2.2.0 Session configuration
    • 2.2.1 QUIC version selection
    • 2.2.2 Minimum version enforcement
    • 2.2.3 Transport parameters negotiation
    • 2.2.4 Congestion control algorithm
      • 2.2.4.1 Reno
      • 2.2.4.2 Cubic
      • 2.2.4.3 BBR
    • 2.2.5 Flow control windows
    • 2.2.6 Max payload size
    • 2.2.7 Unacknowledged packet threshold
    • 2.2.8 QLog enablement
  • 2.3.0 Session lifecycle
    • 2.3.1 Normal closure
    • 2.3.2 Graceful shutdown
    • 2.3.3 Immediate destruction
    • 2.3.4 Idle timeout
    • 2.3.5 Closing period behavior
    • 2.3.6 Draining period behavior
    • 2.3.7 Async.dispose implementation
  • 2.4.0 Connection migration
    • 2.4.1 Path validation
    • 2.4.2 Preferred address handling
      • 2.4.2.1 ignore
      • 2.4.2.2 use
      • 2.4.2.3 default
      • 2.4.2.4 advertising preferred address
    • 2.4.3 Active connection ID management
    • 2.4.4 NAT rebinding detection
  • 2.5.0 Session tickets
    • 2.5.1 Ticket generation (server)
    • 2.5.2 Ticket storage (client)
    • 2.5.3 Session resumption with ticket
    • 2.5.4 Ticket expiration
    • 2.5.5 Application data in tickets
  • 2.6.0 Key updates
    • 2.6.1 Initiating key update
    • 2.6.2 Responding to key update
    • 2.6.3 Key logging (for debugging)
    • 2.6.4 TLS tracing
  • 2.7.0 Error handling
    • 2.7.1 Transport errors
    • 2.7.2 Application errors
    • 2.7.3 Version negotiation errors
    • 2.7.4 Connection failures
    • 2.7.5 TLS errors
    • 2.7.6 Certificate verification errors
  • 2.8.0 Statistics collection
    • 2.8.1 Bytes sent/received
    • 2.8.2 Packets sent/received/lost
    • 2.8.3 Stream counts
    • 2.8.4 RTT measurements
    • 2.8.5 Handshake timing
    • 2.8.6 Congestion window tracking
  • 2.9.0 Stateless reset
• 3.0.0 Stream Testing
  • 3.1.0 Stream creation
    • 3.1.1 Client-initiated streams
      • 3.1.1.1 Bidirectional
      • 3.1.1.2 Unidirectional
    • 3.1.2 Server-initiated streams
      • 3.1.2.1 Bidirectional
      • 3.1.2.2 Unidirectional
    • 3.1.3 Pending stream handling
    • 3.1.4 Stream concurrency limits
    • 3.1.5 Opening streams before handshake completion
    • 3.1.6 Sending early data
    • 3.1.7 Receiving early data
  • 3.2.0 Stream directionality
    • 3.2.1 Bidirectional: both sides can send
    • 3.2.2 Unidirectional: single sender
    • 3.2.3 Read/write state combinations
    • 3.2.4 Local vs remote initiated streams
  • 3.3.0 Data transfer
    • 3.3.1 Sending data via outbound queue
    • 3.3.2 Receiving data via inbound queue
    • 3.3.3 In-memory data sources
      • 3.3.3.1 ArrayBuffer/SharedArrayBuffer data sources
      • 3.3.3.2 ArrayBufferView data sources
      • 3.3.3.3 Blob data sources
      • 3.3.6.4 String data sources
    • 3.3.4 Stream data sources
      • 3.3.7.1 Node.js stream data sources
      • 3.3.7.2 WHATWG stream data sources
      • 3.3.7.3 Sync/Async iterator data sources
      • 3.3.7.4 File System / FD data sources
    • 3.3.5 Zero-length writes
    • 3.3.6 Large data transfers
      • 3.3.6.1 Individual
      • 3.3.6.2 Concurrent
    • 3.3.7 Backpressure handling
  • 3.4.0 Flow control
    • 3.4.1 Stream-level flow control
    • 3.4.2 Session-level flow control
    • 3.4.3 Window updates
    • 3.4.4 Blocked stream handling
    • 3.4.5 High watermark behavior
  • 3.5.0 Stream closure
    • 3.5.1 Normal stream end (FIN)
    • 3.5.2 Stream reset by peer
    • 3.5.3 Stop sending requests
    • 3.5.4 Graceful vs forceful close
    • 3.5.5 Bidirectional shutdown
    • 3.5.6 Half-closed states
  • 3.6.0 Stream priority (HTTP/3)
    • 3.6.1 Priority levels (high/default/low)
    • 3.6.2 Incremental vs non-incremental
    • 3.6.3 Priority updates
    • 3.6.4 Priority enforcement
  • 3.7.0 Stream states
    • 3.7.1 Pending state
    • 3.7.2 Open state
    • 3.7.3 Readable/writable states
    • 3.7.4 End-of-stream detection
    • 3.7.5 Final size tracking
  • 3.8.0 Error scenarios
    • 3.8.1 Stream errors
    • 3.8.2 Reset with error codes
    • 3.8.3 Data after FIN
    • 3.8.4 Protocol violations
    • 3.8.5 Timeouts
    • 3.8.6 Loss recovery
  • 3.9.0 Statistics collection
    • 3.9.1 Bytes sent/received
    • 3.9.2 Acknowledgments
    • 3.9.3 Created/destroyed timestamps
• 4.0.0 Datagram Support
  • 4.1.0 Datagram transmission
    • 4.1.1 Send unreliable datagrams
    • 4.1.2 String data encoding
    • 4.1.3 ArrayBufferView data
    • 4.1.4 Size limits
    • 4.1.5 Datagram ID generation
    • 4.1.6 Datagram not supported
  • 4.2.0 Datagram reception
    • 4.2.1 Receive callback
    • 4.2.2 Early datagram handling
    • 4.2.3 Out-of-order delivery
  • 4.3.0 Datagram status
    • 4.3.1 Acknowledgment notification
    • 4.3.2 Loss notification
    • 4.3.3 Status callback
• 5.0.0 Transport Parameters Testing
  • 5.1.0 Local transport params
    • 5.1.1 Initial max stream data (bidi local/remote, uni)
    • 5.1.2 Initial max data
    • 5.1.3 Initial max streams (bidi/uni)
    • 5.1.4 Max idle timeout
    • 5.1.5 Active connection ID limit
    • 5.1.6 ACK delay exponent
    • 5.1.7 Max ACK delay
    • 5.1.8 Max datagram frame size
    • 5.1.9 Preferred address (IPv4/IPv6)
    • 5.1.10 Disable active migration
    • 5.1.11 Max UDP payload size
  • 5.2.0 Remote transport params
    • 5.2.1 Parsing received params
    • 5.2.2 Validation of received params
    • 5.2.3 Incompatible parameters
    • 5.2.4 Parameter negotiation
    • 5.2.5 Client-server agreement
    • 5.2.6 Minimum value selection
    • 5.2.7 Default value fallbacks
• 6.0.0 TLS Integration Testing
  • 6.1.0 Certificate handling
    • 6.1.1 Single certificate
    • 6.1.2 Multiple certificates
    • 6.1.3 Certificate chains
    • 6.1.4 Certificate validation
    • 6.1.5 Self-signed certificates
    • 6.1.6 Expired certificates
    • 6.1.7 Revoked certificates (CRL)
  • 6.2.0 Key management
    • 6.2.1 KeyObject support
    • 6.2.2 CryptoKey support
    • 6.2.3 Multiple keys
    • 6.2.4 Private key verification
    • 6.2.5 Key type validation
  • 6.3.0 CA and trust
    • 6.3.1 Custom CA certificates
    • 6.3.2 CA chain validation
    • 6.3.3 Client certificate verification
  • 6.4.0 Cipher configuration
    • 6.4.1 Default ciphers
    • 6.4.2 Custom cipher strings
    • 6.4.3 Cipher suite selection
    • 6.4.4 Groups/curves configuration
  • 6.5.0 Session resumption
    • 6.5.1 Session ticket generation
    • 6.5.2 Session ticket reuse
    • 6.5.3 0-RTT data with resumption
    • 6.5.4 Ticket rotation
  • 6.6.0 ALPN
    • 6.6.1 Protocol negotiation
    • 6.6.2 Multiple protocols
    • 6.6.3 No common protocol handling
  • 6.7.0 SNI
    • 6.7.1 Server name indication
    • 6.7.2 SNI-based routing
    • 6.7.3 Virtual hosting
  • 6.8.0 Tokens
    • 6.8.1 Retry token validity / timeouts
      • TODO: Works for ipv4, appears to have issues with some ipv6 paths
    • 6.8.2 Regular token validity /timeouts
    • 6.8.3 Stateless reset token validity / timeouts
• 7.0.0 HTTP/3 Application Layer Testing
  • 7.1.0 HTTP/3 basics
    • 7.1.1 Request/response flow
    • 7.1.2 Stream creation for requests
    • 7.1.3 Multiple concurrent requests
  • 7.2.0 Headers
    • 7.2.1 Header encoding/decoding
    • 7.2.2 QPACK compression
    • 7.2.3 Header limits (pairs, length, field section size)
    • 7.2.4 Pseudo-headers (:method, :path, :authority, :scheme)
    • 7.2.5 Early headers / early-hints / 1xx
    • 7.2.6 Regular headers
    • 7.2.7 Trailing headers
    • 7.2.8 Header ordering
    • 7.2.9 Duplicate headers
  • 7.3.0 QPACK
    • 7.3.1 Dynamic table management
    • 7.3.2 Encoder/decoder streams
    • 7.3.3 Table capacity limits
    • 7.3.4 Blocked streams handling
  • 7.4.0 HTTP/3 settings
    • 7.4.1 SETTINGS frame exchange
    • 7.4.2 Max header list size
    • 7.4.3 QPACK settings
    • 7.4.4 Extended CONNECT protocol
  • 7.5.0 Control streams
    • 7.5.1 Bidirectional control stream
    • 7.5.2 Unidirectional control streams
    • 7.5.3 Stream type identification
  • 7.6.0 Priority (extensible priorities)
    • 7.6.1 PRIORITY_UPDATE frames
    • 7.6.2 Urgency parameter
    • 7.6.3 Incremental parameter
• 8.0.0 Error Handling and Edge Cases
  • 8.1.0 Protocol errors
    • 8.1.1 Invalid frames
    • 8.1.2 Stream ID violations
    • 8.1.3 Flow control violations
    • 8.1.4 Connection ID errors
    • 8.1.5 Token validation failures
  • 8.2.0 Resource exhaustion
    • 8.2.1 Maximum streams exceeded
    • 8.2.2 Memory pressure
    • 8.2.3 Connection limits
    • 8.2.4 Packet queue overflow
  • 8.3.0 Network conditions
    • 8.3.1 Packet loss simulation
    • 8.3.2 Packet reordering
    • 8.3.3 Delayed ACKs
    • 8.3.4 Path MTU discovery
    • 8.3.5 ECN marking
  • 8.4.0 Timeout scenarios
    • 8.4.1 Handshake timeout
    • 8.4.2 Idle timeout
    • 8.4.3 Drain timeout
    • 8.4.4 Keep-alive
  • 8.5.0 Malicious input
    • 8.5.1 Oversized packets
    • 8.5.2 Invalid QUIC versions
    • 8.5.3 Replay attacks
    • 8.5.4 Connection ID spoofing
    • 8.5.5 Amplification attack prevention
• 9.0.0 Interoperability Testing
  • 9.1.0 QUIC version compatibility
    • 9.1.1 v1 (RFC 9000)
    • 9.1.2 Version negotiation
    • 9.1.3 Draft version support (if any)
  • 9.2.0 Cross-implementation testing
    • 9.2.1 Against other QUIC implementations
      • 9.2.1.1 quiche
      • 9.2.1.2 msquic
      • 9.2.1.3 picoquic
    • 9.2.2 Public QUIC endpoints
    • 9.2.3 QUIC interop runner participation
  • 9.3.0 HTTP/3 compatibility
    • 9.3.1 Against other HTTP/3 servers/clients
    • 9.3.2 Public HTTP/3 endpoints
    • 9.3.3 Browser compatibility
• 10.0.0 Performance Testing
  • 10.1.0 Throughput
    • 10.1.1 Single stream throughput
    • 10.1.2 Multiple stream throughput
    • 10.1.3 Bidirectional throughput
    • 10.1.4 Large file transfers
    • 10.1.5 Small message latency
• 10.2.0 Scalability
  • 10.2.1 Many concurrent sessions
  • 10.2.2 Many concurrent streams per session
  • 10.2.3 Connection establishment rate
  • 10.2.4 Memory usage under load
• 10.3.0 Congestion control
  • 10.3.1 Algorithm comparison (Reno vs Cubic vs BBR)
  • 10.3.2 Recovery from loss
  • 10.3.3 Bandwidth utilization
  • 10.3.4 Fairness
• 10.4.0 0-RTT performance
  • 10.4.1 Resumption latency reduction
  • 10.4.2 Early data volume
• 10.5.0 [ ] Benchmarks
• 11.0.0 API Testing
  • 11.1.0 JavaScript API surface
    • 11.1.1 quic.connect() function
    • 11.1.2 quic.listen() function
    • 11.1.3 QuicEndpoint class
    • 11.1.4 QuicSession class
    • 11.1.5 QuicStream class
    • 11.1.6 Constants export
  • 11.2.0 Options validation
    • 11.2.1 Type checking
    • 11.2.2 Range validation
    • 11.2.3 Required vs optional
    • 11.2.4 Default values
    • 11.2.5 Invalid combinations
  • 11.3.0 Event handling
    • 11.3.1 Diagnostics channels
    • 11.3.2 Callback invocations
    • 11.3.3 Event ordering
    • 11.3.4 Error propagation
• 12.0.0 Integration Testing
  • 12.1.0 Simple echo server
  • 12.2.0 File transfer
  • 12.3.0 Streaming media
  • 12.4.0 WebTransport-like usage
• 13.0.0 Debugging and Observability
  • 13.1.0 QLog output
    • 13.1.1 Event logging
    • 13.1.2 Format compliance
    • 13.1.3 Visualization compatibility
  • 13.2.0 Key logging
    • 13.2.1 SSLKEYLOGFILE format
    • 13.2.2 Wireshark integration
  • 13.3.0 Statistics
    • 13.3.1 Endpoint stats accuracy
    • 13.3.2 Session stats accuracy
    • 13.3.3 Stream stats accuracy
    • 13.3.4 Timestamp consistency
  • 13.4.0 Diagnostics channels
    • 13.4.1 All events emitted correctly
    • 13.4.2 Event payload validation
    • 13.4.3 Performance overhead
• 14.0.0 Network environments
  • 14.1.0 IPv4 only
  • 14.2.0 IPv6 only
  • 14.3.0 Dual stack
  • 14.4.0 NAT traversal
  • 14.5.0 Firewalls
• 15.0.0 Compliance Testing
  • 15.1.0 RFC 9000 (QUIC v1)
    • 15.1.1 All MUST requirements
    • 15.1.2 All SHOULD requirements
    • 15.1.3 Recommended behaviors
  • 15.2.0 RFC 9001 (QUIC TLS)
    • 15.2.1 TLS 1.3 integration
    • 15.2.2 Key derivation
    • 15.2.3 Handshake messages
  • 15.3.0 RFC 9002 (Loss Detection and Congestion Control)
    • 15.3.1 Loss detection algorithms
    • 15.3.2 Congestion control requirements
    • 15.3.3 Pacing
  • 15.4.0 RFC 9114 (HTTP/3)
    • 15.4.1 Frame types
    • 15.4.2 Stream mapping
    • 15.4.3 Error codes
  • 15.5.0 RFC 9204 (QPACK)
    • 15.5.1 Encoding/decoding
    • 15.5.2 Dynamic table
    • 15.5.3 Stream blocking
  • 15.6.0 RFC 9221 (Unreliable Datagrams)
    • 15.6.1 Datagram frames
    • 15.6.2 Flow control exemption
• 16.0.0 Fuzzing and Security
  • 16.1.0 Packet fuzzing
  • 16.2.0 Frame fuzzing
  • 16.3.0 Input validation fuzzing
  • 16.4.0 Security scenarios
    • 16.4.1 Amplification attack resistance
    • 16.4.2 DoS resistance / Packet flooding
    • 16.4.3 Resource limits enforcement
    • 16.4.4 Timing attack resistance
    • 16.4.5 Slow-loris-type attacks (slow transmission, slow acks)
• 17.0.0 Documentation Testing
  • 17.1.0 Example code validity
    • 17.1.1 All examples run successfully
    • 17.1.2 Examples follow best practices
  • 17.2.0 API documentation accuracy
  • 17.3.0 All public APIs documented
  • 17.4.0 Parameter types correct
  • 17.5.0 Return types correct
  • 17.6.0 Error conditions documented

[martenrichter]

Here is some work (actually not on the list):
https://github.com/nodejs/node/pull/60237/files#diff-ea3f0a30d20c7e97567c28c8ac8ab672fb82138c02b5fa38eb4146649cfd7aa8
includes a test (as a side effect) for 3.1.2.2 .
Other combinations are possible. I am currently debugging it and finding and filing issues in quic, that are uncovered. (I have filed them separately, but I can also try to patch them within the PR.):
#60309

[martenrichter]

Also added 3.1.1.2 to the PR, which is outstanding for review. However, the test does not work currently as it uncovered a race condition (which is the purpose of a test), which is now on the TODO list.

[Ethan-Arrowood]

This is awesome. I’d love to try and help here. Any easier places for me to start and get familiar with first? Thank you for organizing this list!

[github-actions]

This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.