Repository navigation
Description
Version: FirebirdSql.Data.FirebirdClient 10.3.4 (the same code is on master). Seen against Firebird 3.0.
What happens
With Compression=true, a read fails with
System.IO.IOException: Missing 4 bytes to fill total 4.
at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadBytesAsync(...)
at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadInt32Async(...)
at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadOperationAsync(...)
when a chunk returned by the socket inflates to no output. The server is alive and the connection is healthy: the rest of the data is simply still on its way. We met it on fetches of about 10,000 rows over a VPN, where TCP splits the stream in places the server did not choose. The same reads are fine with compression off.
Cause
FirebirdNetworkHandlingWrapper.Read and ReadAsync make one socket read per call and return what inflating it produced. zlib may consume a chunk and produce nothing, because it needs the rest of the block. HandleDecompression then returns 0, nothing is queued, and the method returns 0.
XdrReaderWriter.ReadBytes / ReadBytesAsync read a 0 as the end of the stream:
while (toRead > 0 && currentlyRead != 0)
{
toRead -= (currentlyRead = _dataProvider.Read(buffer, count - toRead, toRead));
}
if (currentlyRead == 0)
{
// ...
throw new IOException($"Missing {toRead} bytes to fill total {count}.");
}So the two layers disagree about what 0 means: "nothing yet" in the wrapper, "end of stream" in the reader.
Reproduction
The program below puts a small TCP relay between the client and a local server. The relay delivers the server's stream in writes of a fixed size, which is what a link with that segment size does. It reads RDB$RELATION_FIELDS (3,709 rows here). The query, the server and the data are the same in every case. Only the two columns below change.
| Compression | Delivered in pieces of | Result |
|---|---|---|
| off | as received | OK |
| off | 16, 64, 256, 1024, 1460, 4096, 16384 bytes | OK |
| on | as received | OK |
| on | 256, 536, 1024, 1200, 1400, 2048, 2920, 4096, 8192 bytes | OK in 5 of 5 runs each |
| on | 1460 bytes, the usual TCP segment | Missing 4 bytes to fill total 4. in 5 of 5 runs, after all 3,709 rows have been read |
Where the failure lands depends on where a boundary falls in the compressed stream, so a size that passes with this data set can fail with another. With this one, 64-byte pieces failed in 5 of 5 runs at row 2,600.
Suggested fix
Keep reading until a read produces output or the socket really ends. This is Read with whitespace changes hidden; ReadAsync takes the same change:
@@ -59,8 +59,10 @@ sealed class FirebirdNetworkHandlingWrapper : IDataProvider, ITracksIOFailure
{
if (_inputBuffer.Count < count)
{
- var readBuffer = _readBuffer;
int read;
+ do
+ {
+ var readBuffer = _readBuffer;
try
{
read = _dataProvider.Read(readBuffer, 0, readBuffer.Length);
@@ -76,13 +78,18 @@ sealed class FirebirdNetworkHandlingWrapper : IDataProvider, ITracksIOFailure
{
_decryptor.ProcessBytes(readBuffer, 0, read, readBuffer, 0);
}
+ var available = read;
if (_decompressor != null)
{
- read = HandleDecompression(readBuffer, read);
+ available = HandleDecompression(readBuffer, read);
readBuffer = _compressionBuffer;
}
- WriteToInputBuffer(readBuffer, read);
+ WriteToInputBuffer(readBuffer, available);
+ }
}
+ // A chunk can inflate to nothing: zlib holds it until the rest of the block arrives.
+ // Only a zero from the socket is the end of the stream.
+ while (read != 0 && _inputBuffer.Count == 0);
}
var dataLength = ReadFromInputBuffer(buffer, offset, count);
return dataLength;Built from the FirebirdClient-10.3.4.0 tag with this change, the same program passes the 64-byte and the 1460-byte cases in 5 of 5 runs each. I can open a pull request if this is the direction you want.
One more thing, not chased
Pieces of 16 bytes and smaller also fail during the handshake, and that failure does not go away with the fix: it becomes Ionic.Zlib.ZlibException: Bad state (invalid stored block lengths), or invalid block type, right after StartEncryption. It is not stable: 16 bytes passed in 1 of 5 runs. My guess, which I have not verified, is that a boundary inside the end of the last unencrypted packet leaves bytes that are then decrypted. Real links should not fragment that small, so it is probably a different and much rarer problem.
Program.cs
// Minimal reproduction: FirebirdClient 10.3.4 with Compression=true over a TCP path that delivers the
// server's stream in small fragments. No Firebird change is needed; the relay stands in for a link that
// fragments the stream. Reads system tables only.
using System.Diagnostics;
using System.Net;
using System.Net.Sockets;
using FirebirdSql.Data.FirebirdClient;
var password = args[0];
var database = args[1];
var sizes = args[2].Split(',').Select(int.Parse).ToArray();
foreach (var compression in new[] { false, true })
{
foreach (var size in sizes)
{
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
using var stop = new CancellationTokenSource();
var relay = Task.Run(() => RunRelayAsync(listener, size, stop.Token));
var builder = new FbConnectionStringBuilder
{
DataSource = "127.0.0.1",
Port = port,
Database = database,
UserID = "SYSDBA",
Password = password,
Pooling = false,
Compression = compression,
};
var clock = Stopwatch.StartNew();
var rows = 0L;
string outcome;
var work = Task.Run(async () =>
{
await using var connection = new FbConnection(builder.ToString());
await connection.OpenAsync();
await using var command = connection.CreateCommand();
command.CommandText = "select a.RDB$FIELD_NAME, a.RDB$RELATION_NAME, a.RDB$FIELD_SOURCE from RDB$RELATION_FIELDS a";
await using var reader = await command.ExecuteReaderAsync();
while (await reader.ReadAsync())
{
Interlocked.Increment(ref rows);
}
});
if (await Task.WhenAny(work, Task.Delay(TimeSpan.FromSeconds(25))) != work)
{
outcome = "no answer in 25 s";
}
else if (work.IsFaulted)
{
var ex = work.Exception!.InnerException!;
var root = ex;
while (root.InnerException is not null)
{
root = root.InnerException;
}
outcome = $"FAILED {ex.GetType().Name} / {root.GetType().Name}: {root.Message}";
}
else
{
outcome = "OK";
}
stop.Cancel();
listener.Stop();
Console.WriteLine($"compression={compression,-5} fragment={(size == 0 ? "whole" : size + " B"),-8} rows={Interlocked.Read(ref rows),-6} {clock.ElapsedMilliseconds,6} ms {outcome}");
}
}
static async Task RunRelayAsync(TcpListener listener, int fragment, CancellationToken token)
{
try
{
while (!token.IsCancellationRequested)
{
var client = await listener.AcceptTcpClientAsync(token);
client.NoDelay = true;
var server = new TcpClient { NoDelay = true };
await server.ConnectAsync(IPAddress.Loopback, 3050, token);
_ = PumpAsync(client.GetStream(), server.GetStream(), 0, token);
_ = PumpAsync(server.GetStream(), client.GetStream(), fragment, token);
}
}
catch (OperationCanceledException)
{
}
catch (SocketException)
{
}
}
static async Task PumpAsync(NetworkStream from, NetworkStream to, int fragment, CancellationToken token)
{
var buffer = new byte[65536];
try
{
while (true)
{
var read = await from.ReadAsync(buffer, token);
if (read == 0)
{
to.Close();
return;
}
if (fragment == 0)
{
await to.WriteAsync(buffer.AsMemory(0, read), token);
continue;
}
for (var offset = 0; offset < read; offset += fragment)
{
await to.WriteAsync(buffer.AsMemory(offset, Math.Min(fragment, read - offset)), token);
await to.FlushAsync(token);
// Long enough for the client to take the fragment as a read of its own.
var spin = Stopwatch.StartNew();
while (spin.Elapsed.TotalMicroseconds < 150)
{
Thread.SpinWait(20);
}
}
}
}
catch (Exception ex) when (ex is IOException or OperationCanceledException or ObjectDisposedException)
{
to.Close();
}
}Run with fbrepro <password> <database path> 0,16,64,1460. Needs a Firebird server on 127.0.0.1:3050; 0 means "as received".
Activity