docker-container: give readiness wait a full startup timeout - #4050
Merged
crazy-max merged 2 commits intoSep 10, 2026
Conversation
A recently started builder should use its StartedAt timestamp to decide whether BuildKit may still be coming up, but not as the absolute deadline for the client readiness wait. When the first client request arrived near the end of that startup window, the wait could inherit only a few milliseconds and fail with DeadlineExceeded even though the builder was still legitimately starting. Give those recent containers a fresh bounded wait from the current time while continuing to skip the wait for established builders. Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Member
Author
|
Encounter this error quite often in actions-toolkit integration tests: https://github.com/docker/actions-toolkit/actions/runs/34329830393/job/102395668163?pr=1309#step:10:752 |
Use the container start time only to decide whether readiness waiting is needed. Start a full timeout immediately before waiting so Docker dial time does not consume the readiness budget. Name the startup window and readiness timeout independently, and cover the exact startup-window boundary. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
tonistiigi
approved these changes
Sep 10, 2026
This was referenced Sep 13, 2026
VirSanghavi
added a commit
to antifailure/antifailure
that referenced
this pull request
Sep 13, 2026
…uildKit the rest of 20 seconds (#425) ## The failure The first attempt of v1.4.0's release run, [34779202432](https://github.com/antifailure/antifailure/actions/runs/34779202432), failed the sidecar job's *Push both architectures* step about 100 ms after it started: ``` ERROR: failed to build: waiting for BuildKit: DeadlineExceeded: context deadline exceeded while waiting for connections to become ready ``` The same builder had finished the `--load` build seconds earlier. The steps between the two builds only inspect an image, run it and log in, and none of them touches the builder. Re-running the failed jobs passed, `publish` included, which is how v1.4.0 shipped. ## The cause, and the evidence The message is buildx's. [docker/buildx#4027](docker/buildx#4027), first released in v0.37.0 and carried by ubuntu-latest, makes the docker-container driver wait for BuildKit readiness while the builder container is less than 20 seconds old. The wait's deadline is the fixed moment `StartedAt + 20s`, not a timeout of its own. A client call that arrives near the end of the window gets whatever is left, and the connection, a `docker exec` of `buildctl dial-stdio`, needs more than a tenth of a second. The job's own timestamps: - **Attempt 1**, job 103783025245: builder container created 19:57:02.38 to 02.60, push started 19:57:22.19, failed 22.29. That is inside the window with roughly 100 to 300 ms left. The `--load` build started 2 s after boot and had 18 s to spare. - **Attempt 2**, job 103783498485: builder container created 20:00:52.89 to 53.04, push started 20:01:14.38. That is 21.4 s after, outside the window, and it passed, as did `publish`. [docker/buildx#4050](docker/buildx#4050) quotes the identical error from buildx's own CI and gives the wait a timeout of its own. It is released in v0.37.1, whose `driver.go` carries `clientWaitReady` and `buildkitdReadyTimeout`. So this is a timing defect rather than a flake: any release whose first buildx call after boot lands just short of 20 seconds fails the same way. ## The fix One step right after `setup-buildx-action`, which gains `id: buildx`: - reads `buildx_buildkit_<builder>0`'s `StartedAt` and sleeps until the container is at least 22 seconds old. `StartedAt` never moves after boot, so every buildx call after the step, the `--load` build included, is outside the window. Cost: at most about 20 seconds a release. - prints `docker buildx version` and the container's age before and after, so a release log shows what the step did. - fails, rather than skipping, when there is no builder name, no container of that name, or a start time that does not parse. A wait that silently does nothing is a green step protecting nothing. - says in its comment to remove it once the runner's `docker buildx version` is v0.37.1 or later. Not taken: pinning `version: v0.37.1`. At the commit pinned here, `setup-buildx-action` downloads a pinned buildx without `verifySignature`, which trades a timing race for an unverified binary, and the pin would outlive the runner image's own upgrade. ## Proof The step was extracted from this branch's `release.yml` by a YAML parser, and run under a throwaway Docker configuration against a real local docker-container builder, with GNU `date`: - **young**: builder 3 s old. `waiting 19s for buildx's 20 second startup window to close`, then `is now 22s old`, exit 0 - **old**: the same builder at 22 s. No wait, exit 0 - **missing container**: a builder name with no container. `::error::could not read the start time of buildx_buildkit_afdemorepo-no-such-builder0`, exit 1 - **empty builder name**: `::error::setup-buildx-action reported no builder name`, exit 1 - **unparseable start time**, from a docker shim answering `not-a-time`: `::error::... reported a start time that does not parse`, exit 1 Every run printed `docker buildx version` first. `actionlint` 1.7.12 on `release.yml` reports nothing new. Its two `SC2035` notes are on `publish`'s *Checksums* step, and main's copy has the same two. ## What this does NOT prove **The race was not reproduced locally, and this step has never been run against the failure it exists to prevent.** Read the proof above as "the step waits, reports, and refuses correctly", not as "the step stops the push failing". The attempts, all with buildx v0.37.0 and v0.37.1 linux-arm64 binaries verified by checksum and by cosign against their `.sigstore.json` bundles, run standalone in a throwaway container against a local docker-container builder: - **9 arms timed on the Mac's clock**, builds issued at +18.96s to +19.93s after the builder container's `StartedAt`: every one passed, on both versions. - **The Docker VM's clock was 0.378s ahead of the Mac's.** `StartedAt` is stamped by the VM, so those 9 arms were aimed at the wrong moment by about a third of a second, and `docker exec` added its own variable delay on top. - **10 arms timed inside the VM**, sleeping and firing on the clock that stamps `StartedAt` with no `docker exec` in between, v0.37.0 builds fired at +19.746s to +20.080s: every one passed. A likely reason, not measured: locally the `dial-stdio` connection is ready in milliseconds, so the slot in which the leftover window is shorter than the connection is tiny, and sleep jitter of around 100 ms steps over it. Repro attempts were then stopped, because more sweeps would cost Docker time other work needs and would not change the evidence below. **The diagnosis rests on three things instead:** 1. buildx v0.37.0's `driver/docker-container/driver.go` `Client()`, which sets the readiness wait's deadline to the fixed moment `StartedAt + 20s`. 2. docker/buildx#4050, which quotes the identical `waiting for BuildKit: DeadlineExceeded: context deadline exceeded while waiting for connections to become ready` from buildx's own CI and fixes it in v0.37.1. 3. Run 34779202432's own timestamps: attempt 1's push inside the window and failing 100 ms in, attempt 2's push at +21.4s outside it and passing. No changelog entry: this is release workflow plumbing, and the published artifacts and what a customer runs do not change. The commit carries a `Changelog-None` trailer saying so. Signed-off-by: Vir Sanghavi <67278851+VirSanghavi@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
relates to https://github.com/docker/buildx/actions/runs/33654939933/job/100331039441#step:7:2387
A recently started builder should use its
StartedAttimestamp to decide whether BuildKit may still be coming up, but not as the absolute deadline for the client readiness wait.When the first client request arrived near the end of that startup window, the wait could inherit only a few milliseconds and fail with DeadlineExceeded even though the builder was still legitimately starting. Give those recent containers a fresh bounded wait from the current time while continuing to skip the wait for established builders.