Skip to content

Compression=true: a read that inflates to zero bytes is taken for the end of the stream ("Missing 4 bytes to fill total 4") #1293

Open

Description

@vitalyashcheulov

Version: FirebirdSql.Data.FirebirdClient 10.3.4 (the same code is on master). Seen against Firebird 3.0.

What happens

With Compression=true, a read fails with

System.IO.IOException: Missing 4 bytes to fill total 4.
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadBytesAsync(...)
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadInt32Async(...)
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadOperationAsync(...)

when a chunk returned by the socket inflates to no output. The server is alive and the connection is healthy: the rest of the data is simply still on its way. We met it on fetches of about 10,000 rows over a VPN, where TCP splits the stream in places the server did not choose. The same reads are fine with compression off.

Cause

FirebirdNetworkHandlingWrapper.Read and ReadAsync make one socket read per call and return what inflating it produced. zlib may consume a chunk and produce nothing, because it needs the rest of the block. HandleDecompression then returns 0, nothing is queued, and the method returns 0.

XdrReaderWriter.ReadBytes / ReadBytesAsync read a 0 as the end of the stream:

while (toRead > 0 && currentlyRead != 0)
{
    toRead -= (currentlyRead = _dataProvider.Read(buffer, count - toRead, toRead));
}
if (currentlyRead == 0)
{
    // ...
    throw new IOException($"Missing {toRead} bytes to fill total {count}.");
}

So the two layers disagree about what 0 means: "nothing yet" in the wrapper, "end of stream" in the reader.

Reproduction

The program below puts a small TCP relay between the client and a local server. The relay delivers the server's stream in writes of a fixed size, which is what a link with that segment size does. It reads RDB$RELATION_FIELDS (3,709 rows here). The query, the server and the data are the same in every case. Only the two columns below change.

Compression Delivered in pieces of Result
off as received OK
off 16, 64, 256, 1024, 1460, 4096, 16384 bytes OK
on as received OK
on 256, 536, 1024, 1200, 1400, 2048, 2920, 4096, 8192 bytes OK in 5 of 5 runs each
on 1460 bytes, the usual TCP segment Missing 4 bytes to fill total 4. in 5 of 5 runs, after all 3,709 rows have been read

Where the failure lands depends on where a boundary falls in the compressed stream, so a size that passes with this data set can fail with another. With this one, 64-byte pieces failed in 5 of 5 runs at row 2,600.

Suggested fix

Keep reading until a read produces output or the socket really ends. This is Read with whitespace changes hidden; ReadAsync takes the same change:

@@ -59,8 +59,10 @@ sealed class FirebirdNetworkHandlingWrapper : IDataProvider, ITracksIOFailure
 	{
 		if (_inputBuffer.Count < count)
 		{
-			var readBuffer = _readBuffer;
 			int read;
+			do
+			{
+				var readBuffer = _readBuffer;
 				try
 				{
 					read = _dataProvider.Read(readBuffer, 0, readBuffer.Length);
@@ -76,13 +78,18 @@ sealed class FirebirdNetworkHandlingWrapper : IDataProvider, ITracksIOFailure
 					{
 						_decryptor.ProcessBytes(readBuffer, 0, read, readBuffer, 0);
 					}
+					var available = read;
 					if (_decompressor != null)
 					{
-					read = HandleDecompression(readBuffer, read);
+						available = HandleDecompression(readBuffer, read);
 						readBuffer = _compressionBuffer;
 					}
-				WriteToInputBuffer(readBuffer, read);
+					WriteToInputBuffer(readBuffer, available);
+				}
 			}
+			// A chunk can inflate to nothing: zlib holds it until the rest of the block arrives.
+			// Only a zero from the socket is the end of the stream.
+			while (read != 0 && _inputBuffer.Count == 0);
 		}
 		var dataLength = ReadFromInputBuffer(buffer, offset, count);
 		return dataLength;

Built from the FirebirdClient-10.3.4.0 tag with this change, the same program passes the 64-byte and the 1460-byte cases in 5 of 5 runs each. I can open a pull request if this is the direction you want.

One more thing, not chased

Pieces of 16 bytes and smaller also fail during the handshake, and that failure does not go away with the fix: it becomes Ionic.Zlib.ZlibException: Bad state (invalid stored block lengths), or invalid block type, right after StartEncryption. It is not stable: 16 bytes passed in 1 of 5 runs. My guess, which I have not verified, is that a boundary inside the end of the last unencrypted packet leaves bytes that are then decrypted. Real links should not fragment that small, so it is probably a different and much rarer problem.

Program.cs
// Minimal reproduction: FirebirdClient 10.3.4 with Compression=true over a TCP path that delivers the
// server's stream in small fragments. No Firebird change is needed; the relay stands in for a link that
// fragments the stream. Reads system tables only.
using System.Diagnostics;
using System.Net;
using System.Net.Sockets;
using FirebirdSql.Data.FirebirdClient;

var password = args[0];
var database = args[1];
var sizes = args[2].Split(',').Select(int.Parse).ToArray();

foreach (var compression in new[] { false, true })
{
    foreach (var size in sizes)
    {
        var listener = new TcpListener(IPAddress.Loopback, 0);
        listener.Start();
        var port = ((IPEndPoint)listener.LocalEndpoint).Port;
        using var stop = new CancellationTokenSource();
        var relay = Task.Run(() => RunRelayAsync(listener, size, stop.Token));

        var builder = new FbConnectionStringBuilder
        {
            DataSource = "127.0.0.1",
            Port = port,
            Database = database,
            UserID = "SYSDBA",
            Password = password,
            Pooling = false,
            Compression = compression,
        };

        var clock = Stopwatch.StartNew();
        var rows = 0L;
        string outcome;
        var work = Task.Run(async () =>
        {
            await using var connection = new FbConnection(builder.ToString());
            await connection.OpenAsync();
            await using var command = connection.CreateCommand();
            command.CommandText = "select a.RDB$FIELD_NAME, a.RDB$RELATION_NAME, a.RDB$FIELD_SOURCE from RDB$RELATION_FIELDS a";
            await using var reader = await command.ExecuteReaderAsync();
            while (await reader.ReadAsync())
            {
                Interlocked.Increment(ref rows);
            }
        });

        if (await Task.WhenAny(work, Task.Delay(TimeSpan.FromSeconds(25))) != work)
        {
            outcome = "no answer in 25 s";
        }
        else if (work.IsFaulted)
        {
            var ex = work.Exception!.InnerException!;
            var root = ex;
            while (root.InnerException is not null)
            {
                root = root.InnerException;
            }

            outcome = $"FAILED {ex.GetType().Name} / {root.GetType().Name}: {root.Message}";
        }
        else
        {
            outcome = "OK";
        }

        stop.Cancel();
        listener.Stop();
        Console.WriteLine($"compression={compression,-5} fragment={(size == 0 ? "whole" : size + " B"),-8} rows={Interlocked.Read(ref rows),-6} {clock.ElapsedMilliseconds,6} ms  {outcome}");
    }
}

static async Task RunRelayAsync(TcpListener listener, int fragment, CancellationToken token)
{
    try
    {
        while (!token.IsCancellationRequested)
        {
            var client = await listener.AcceptTcpClientAsync(token);
            client.NoDelay = true;
            var server = new TcpClient { NoDelay = true };
            await server.ConnectAsync(IPAddress.Loopback, 3050, token);
            _ = PumpAsync(client.GetStream(), server.GetStream(), 0, token);
            _ = PumpAsync(server.GetStream(), client.GetStream(), fragment, token);
        }
    }
    catch (OperationCanceledException)
    {
    }
    catch (SocketException)
    {
    }
}

static async Task PumpAsync(NetworkStream from, NetworkStream to, int fragment, CancellationToken token)
{
    var buffer = new byte[65536];
    try
    {
        while (true)
        {
            var read = await from.ReadAsync(buffer, token);
            if (read == 0)
            {
                to.Close();
                return;
            }

            if (fragment == 0)
            {
                await to.WriteAsync(buffer.AsMemory(0, read), token);
                continue;
            }

            for (var offset = 0; offset < read; offset += fragment)
            {
                await to.WriteAsync(buffer.AsMemory(offset, Math.Min(fragment, read - offset)), token);
                await to.FlushAsync(token);

                // Long enough for the client to take the fragment as a read of its own.
                var spin = Stopwatch.StartNew();
                while (spin.Elapsed.TotalMicroseconds < 150)
                {
                    Thread.SpinWait(20);
                }
            }
        }
    }
    catch (Exception ex) when (ex is IOException or OperationCanceledException or ObjectDisposedException)
    {
        to.Close();
    }
}

Run with fbrepro <password> <database path> 0,16,64,1460. Needs a Firebird server on 127.0.0.1:3050; 0 means "as received".

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions