Repository navigation
Docker-Desktop 4.27.0 breaks Istio on Kubernetes #7148
Description
Description
After the upgrade to 4.27.0 Istio running on-top of Docker Desktop Kubernetes stops working on the istio-init fails to start.
Here are the logs of one of the failing init containers:
2024-01-29T09:09:56.603820Z info Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=true
INVALID_DROP=
2024-01-29T09:09:56.603868Z info Istio iptables variables:
IPTABLES_VERSION=
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_OWNER_GROUPS_INCLUDE=*
OUTBOUND_OWNER_GROUPS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=5671,5672,25672
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DUAL_STACK=false
DNS_CAPTURE=true
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[10.96.0.10],[]
NETWORK_NAMESPACE=
CNI_MODE=false
EXCLUDE_INTERFACES=
2024-01-29T09:09:56.603950Z info Running iptables-restore with the following input:
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -p tcp --dport 5671 -j RETURN
-A ISTIO_OUTPUT -p tcp --dport 5672 -j RETURN
-A ISTIO_OUTPUT -p tcp --dport 25672 -j RETURN
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp -m multiport ! --dports 53,15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp ! --dport 53 -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp ! --dport 53 -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -p tcp --dport 53 -d 10.96.0.10/32 -j REDIRECT --to-ports 15053
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A OUTPUT -p udp --dport 53 -m owner --uid-owner 1337 -j RETURN
-A OUTPUT -p udp --dport 53 -m owner --gid-owner 1337 -j RETURN
-A OUTPUT -p udp --dport 53 -d 10.96.0.10/32 -j REDIRECT --to-port 15053
COMMIT
* raw
-A OUTPUT -p udp --dport 53 -m owner --uid-owner 1337 -j CT --zone 1
-A OUTPUT -p udp --sport 15053 -m owner --uid-owner 1337 -j CT --zone 2
-A OUTPUT -p udp --dport 53 -m owner --gid-owner 1337 -j CT --zone 1
-A OUTPUT -p udp --sport 15053 -m owner --gid-owner 1337 -j CT --zone 2
-A OUTPUT -p udp --dport 53 -d 10.96.0.10/32 -j CT --zone 2
-A PREROUTING -p udp --sport 53 -s 10.96.0.10/32 -j CT --zone 1
COMMIT
2024-01-29T09:09:56.604012Z info Running command (with wait lock): iptables-restore --noflush --wait=30
2024-01-29T09:09:56.604934Z error Command error output: xtables other problem: line 2 failed
2024-01-29T09:09:56.605120Z info Running command (without lock): iptables-save
2024-01-29T09:09:56.606686Z info Command output:
# Generated by iptables-save v1.8.7 on Mon Jan 29 09:09:56 2024
*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Mon Jan 29 09:09:56 2024
# Generated by iptables-save v1.8.7 on Mon Jan 29 09:09:56 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A OUTPUT -p udp -m udp --dport 53 -m owner --uid-owner 1337 -j RETURN
-A OUTPUT -p udp -m udp --dport 53 -m owner --gid-owner 1337 -j RETURN
-A OUTPUT -d 10.96.0.10/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m tcp --dport 5671 -j RETURN
-A ISTIO_OUTPUT -p tcp -m tcp --dport 5672 -j RETURN
-A ISTIO_OUTPUT -p tcp -m tcp --dport 25672 -j RETURN
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m multiport ! --dports 53,15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp -m tcp ! --dport 53 -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp -m tcp ! --dport 53 -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 10.96.0.10/32 -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Mon Jan 29 09:09:56 2024
2024-01-29T09:09:56.606706Z error exit status 1
Reproduce
- istioctl install
- Start any pod with the istio sidecar enabled
Expected behavior
Istio works fine like in all previous releases
docker version
Client:
Cloud integration: v1.0.35+desktop.10
Version: 25.0.1
API version: 1.44
Go version: go1.21.6
Git commit: 29cf629
Built: Tue Jan 23 23:06:12 2024
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.27.0 (135262)
Engine:
Version: 25.0.1
API version: 1.44 (minimum version 1.24)
Go version: go1.21.6
Git commit: 71fa3ab
Built: Tue Jan 23 23:09:35 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.27
GitCommit: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc:
Version: 1.1.11
GitCommit: v1.1.11-0-g4bccb38
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client:
Version: 25.0.1
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1-desktop.4
Path: /Users/xvzf/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.3-desktop.1
Path: /Users/xvzf/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container. (Docker Inc.)
Version: 0.0.22
Path: /Users/xvzf/.docker/cli-plugins/docker-debug
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /Users/xvzf/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.21
Path: /Users/xvzf/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.4
Path: /Users/xvzf/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.0.0
Path: /Users/xvzf/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/xvzf/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.3.0
Path: /Users/xvzf/.docker/cli-plugins/docker-scout
Server:
Containers: 127
Running: 68
Paused: 0
Stopped: 59
Images: 47
Server Version: 25.0.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc version: v1.1.11-0-g4bccb38
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.6.12-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 7.756GiB
Name: docker-desktop
ID: b3867f20-46f6-4d96-bb14-2840a648273a
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: falseDiagnostics ID
89814F6E-F5C8-4F6E-BCA8-FE138807A3C4/20240129091521
Additional Info
No response
Reacted by Antonin Bas, Ivan Korolev and Marc Villacorta
Activity
xvzf commentedon Jan 29, 2024
Note: this is also affecting Docker Desktop for Linux.
xvzf commentedon Jan 29, 2024
The culprit is the kernel configuration, which disables a bunch of conntrack features required to run istio and other service meshes:
For the record, the full diff can be found here: https://gist.github.com/xvzf/7a81f881033cab9a930f081929c3c5f0
dgageot commentedon Jan 29, 2024
Thanks a lot @xvzf for fine the root cause. I'll add them back and add a few tests specific to istio.
dgageot commentedon Jan 29, 2024
@xvzf do you know if only those 5 are required? How can I quickly test Istio nowadays? I haven't tried in a long time.
xvzf commentedon Jan 30, 2024
Hi @dgageot, thanks for looking into this!
There's a platform prerequisites section in the Istio documentation but no specific configuration options for the modules (e.g. zone support on conntrack).
As for testing this specific bug, it should be sufficient to:
istioctl()istioctl install --set profile=demoistio-injection=enabledAlso a suggestion form my colleague @jordiclariana:
works on 4.61.1 but fails on 4.27.0. Might be worth adding it to your test-suite
dgageot commentedon Feb 1, 2024
Hi @xvzf! Docker Desktop 4.27.1 is out. Could you tell me if it solves your issue?
dgageot commentedon Feb 1, 2024
@xvzf Sorry for the false information. I just learnt that those patches will ship with 4.27.2
craigbox commentedon Feb 12, 2024
David is back at Docker - hooray!
dgageot commentedon Feb 13, 2024
Indeed :-) Have you tried to start Istio with Docker Desktop 4.27.2, Craig?
craigbox commentedon Feb 13, 2024
jordiclariana commentedon Feb 13, 2024
@dgageot , we did try 4.27.2 with Istio and so far so good. Seems that the problem is solved there. Thanks!
dgageot commentedon Feb 13, 2024
\o/ Thanks @jordiclariana. I'm closing the issue then