driver/kubernetes: Add retry logic for transient TLS connection errors#3493
Merged
AkihiroSuda merged 1 commit intodocker:masterfrom Jan 6, 2026
Merged
driver/kubernetes: Add retry logic for transient TLS connection errors#3493AkihiroSuda merged 1 commit intodocker:masterfrom
AkihiroSuda merged 1 commit intodocker:masterfrom
Conversation
Guimove
force-pushed
the
fix-k8s-tls-retry-2668
branch
from
0780f28 to
71d3044
Compare
tonistiigi
reviewed
Oct 29, 2025
3 tasks
AkihiroSuda
reviewed
Dec 24, 2025
AkihiroSuda
previously approved these changes
Dec 24, 2025
Collaborator
|
Please squash the commits |
|
@Guimove - would you be able to continue with merging the PR in the close future, please? |
Contributor
Author
Will resolve the comments from @AkihiroSuda today so as soon it's validated it could be merged. |
Guimove
force-pushed
the
fix-k8s-tls-retry-2668
branch
4 times, most recently
from
9fd0db5 to
225f31d
Compare
Collaborator
|
DCO check seems failing |
Fixes docker#2668 When Kubernetes marks nodes as "Ready" before their Certificate Signing Requests (CSRs) are approved, the buildx kubernetes driver can fail to connect to builder pods with transient TLS errors like: - "tls: internal error" - "context deadline exceeded" - "use of closed network connection" - "i/o timeout" This is particularly problematic on EKS clusters with ARM64 nodes under heavy load, where multiple builders are being spawned simultaneously. This commit adds retry logic with exponential backoff to the Dial() function in the kubernetes driver. The implementation: - Attempts up to 5 connection retries - Uses exponential backoff starting at 500ms, capped at 10s - Only retries on known transient connection errors - Uses errors.Is/errors.As for proper error type checking - Logs retry attempts using logrus for visibility - Respects context cancellation This allows buildx to gracefully handle the race condition where pods are marked as Running before their TLS certificates are fully ready. Signed-off-by: guimove <dasilva.guillaume@live.fr>
Guimove
force-pushed
the
fix-k8s-tls-retry-2668
branch
from
225f31d to
e758a6b
Compare
Contributor
Author
|
@AkihiroSuda it's fixed |
AkihiroSuda
approved these changes
Jan 5, 2026
Contributor
Author
|
@AkihiroSuda Thanks for the validation, who can merge now ? (seems not me) |
|
I'd like to confirm it's been working well now! Thank you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2668
When Kubernetes marks nodes as "Ready" before their Certificate Signing Requests (CSRs) are approved, the buildx kubernetes driver can fail to connect to builder pods with transient TLS errors like:
This is particularly problematic on EKS clusters with ARM64 nodes under heavy load, where multiple builders are being spawned simultaneously.
This commit adds retry logic with exponential backoff to the Dial() function in the kubernetes driver. The implementation:
This allows buildx to gracefully handle the race condition where pods are marked as Running before their TLS certificates are fully ready.