Repository navigation
Docker swarm - encrypted network overlay - stops working. #30727
Description
Description
After creating a 3 node swarm (all managers), and then creating an encrypted overlay, we have noticed that the overlay network drops out randomly
Steps to reproduce the issue:
- Create docker swarm cluster of at least 3 nodes
- Create overlay with:
docker network create --attachable --opt encrypted -d overlay networkname"
NOTE: Making it attachable to test easily
- Start an alpine container (easy test) on 2 nodes:
docker run -it --rm --net=networkname alpine /bin/ash
4.) Find the IPs (ifconfig) of each, and ping across.
Describe the results you received:
It works and randomly it stops. Firewall (both IP protocol 50 and the rest of the parts are any/any allowed between the 3 nodes)
Describe the results you expected:
To work all the time :)
Additional information you deem important (e.g. issue happens only occasionally):
Happens randomly almost. If you reboot, it starts working again.
Output of docker version:
Client:
Version: 1.13.0
API version: 1.25
Go version: go1.7.3
Git commit: 49bf474
Built: Tue Jan 17 09:58:26 2017
OS/Arch: linux/amd64
Server:
Version: 1.13.0
API version: 1.25 (minimum version 1.12)
Go version: go1.7.3
Git commit: 49bf474
Built: Tue Jan 17 09:58:26 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 8
Server Version: 1.13.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 34
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: active
NodeID: 4y3mi5goxun18p0rif8hdrt5o
Is Manager: true
ClusterID: vcwzg0mebqw4kp58pz8ynm0cn
Managers: 3
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: PUB#1
Manager Addresses:
PUB#1:2377
PUB#2:2377
PUB#2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 100 GiB
Name: swarmhost01
ID: GVD4:VFPH:ELAN:X2CK:CLFZ:MFDC:C5LT:RLTU:DWKE:KDKY:HT6M:BAC2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
nfs=yes
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
Environment is between physical and virtual systems. We have changed it around to be only virtual and only physical - same results. Systems are located in 3 different regions, on 3 different public IP spaces.
Activity
ventz commentedon Feb 3, 2017
Additional information: the overlay seems to be aware of the other IPs, since when launching new containers on the different swarm subnets, it keeps increasing the last octet by 1 (ex: 10.0.0.3, 10.0.0.4, etc)
Also, so far from a few tests, it seems in some way related specifically to encrypted overlays -- we have not seen this happen on non-encrypted overlays. In fact, if you remove the overlay (when encrypted) and re-created it non-encrypted, it starts working again.
Also, if you remove and re-create the encrypted overlay, it still will not work until the host(s) are rebooted.
aboch commentedon Feb 3, 2017
@ventz Few questions:
tcpdump -i <iface> -p esp)Just to see if the issue is at the tx node or at the rx node.
Thanks
ventz commentedon Feb 3, 2017
@aboch
It seems "random". For example, on the latest cluster it worked for a few days (given no containers on it) before it hit this status again. I have seen it as soon as an hour, and the "average" (if you can even call it that) seems to be a few hours to less than a couple of days. I can start monitoring this more carefully.
I have one container on each node (when I test), but I've noticed that since I first test with just ICMP (easiest), it won't matter which way I am initializing it. If I get both ICMP echo + reply, the overlay works. As soon as I stop getting "replies", that indicates a problem. Once it hits this state, I do open it from all the sides, and test to see which way exactly it fails. I have noticed a few times that the issue seems to be surrounding the leader node. Ex: in this case where it just happened, as soon as I rebooted the leader, it started working again.
This is the interesting scenario -- you can see ESP coming in on the docker host (not container) while pinging, but it will not send ESP back (or originate if you try to originate only). It's almost like some part responsible for this just collapses. No error logs so far in /var/log/system
Here is what a tcpdump looks like
As you can see, the system is not even trying to originate protocol 50 traffic back.
aboch commentedon Feb 3, 2017
Thanks @ventz
It looks like the system is currently in the erroneous state. If so, it would be interesting do some debugging:
Can you please check if the container running on node#3 is actually receiving the icmp echo request packet and that it is generating the responses back (For this either tcpdump from docker exec if the container image has it, or nsenter into the container netns from the host (nsenter --net=<value from docker inspect container | grep xKey">)
ventz commentedon Feb 3, 2017
@aboch Hi - this is sadly the tcpdump I had taken from earlier today when it happened. I have already rebooted that node.
That said, the container on private#3 was generating back the icmp replies, but they didn't show up on the component responsible for receiving/processing/sending ESP.
It's almost like something isolated to the ESP connection is failing.
To give you an example, when this happened, I removed the containers and removed the overlay. Then re-created the overlay (encrypted), and it still didn't work. Tried removing+re-adding one more time. At that point, I re-added it without encryption, and it worked. I removed it, and re-added it with encryption again, nothing. But then rebooting the leader node fixed it.
aboch commentedon Feb 3, 2017
Looks like.
I did it before, but let me run again few nodes on AWS overnight with containers pinging over encrypted network. I will have them be all managers as in your test. I will report back my findings.
ventz commentedon Feb 3, 2017
I did get it to happen on AWS, but if possible - try to do one at a local datacenter or a different network. Noticed it happening a lot more when it's going over public lines -- so it might be a disruption or blip that ends up causing it.
I'll keep an eye for it again too, although for now the latest cluster seems to be working.
I know this sucks for debugging -- it's clearly broken in some way, but capturing what exactly is not easy, especially from someone going "hey it just fails, oh and it's random" :)
ventz commentedon Feb 4, 2017
@aboch I have an environment where this is happening right now.
aboch commentedon Feb 4, 2017
@ventz
Check with
ip -s xfrm stateif the counters are increasing for the proper SAs (check the src and dst IP and the SPI). On each node you will find 4 states for each couple of nodes, 3 RX (dst is this node) states and one TX (src is this node).ventz commentedon Feb 4, 2017
Ok, this is kind of interesting:
The overlay seems to be a 10.0.0.0/24
Trying from PUB#1 -> PUB#3 (out of 5, where the first 3 are managers, and 2 are workers)
The interesting part is that the arp request had the right mac (00:00:0d) of .13 --> which I verified on .13, was the case. But .13 was not seeing any traffic. Same in the reverse direction.
However, I can actually see ESP exchanges in this case. Also, some part of the overlay works -- because I can reach some of the other nodes, so it's not complete shot. It's just the connection between these 2 hosts (even though they are both online)
aboch commentedon Feb 4, 2017
Also please check on the node which is not generating the responses back whether the counter in increasing for the rule the following command returns:
ventz commentedon Feb 4, 2017
@aboch any way we can chat in real time (irc?) - I can also share access into the env via a screen session.
10 remaining items
aboch commentedon Feb 11, 2017
Thanks @ventz I found where the issue is and pushed the fix ^. It affects only the use case where the advertise address is outside of the box (1-1 NAT case) which is way I could not initially reproduce. It happens during key rotations. It is expected a node reload, or removing and restarting the container on the node fixes it, but only until next datapath key rotation happens.
ventz commentedon Feb 11, 2017
@aboch - That's great! Thanks.
What do you think is the eta for the next update that will have this?
[-]Docker swarm - encrypted network overlay - randomly stops working.[/-][+]Docker swarm - encrypted network overlay - stops working.[/+]ventz commentedon Mar 6, 2017
@aboch It looks like a bug was introduced in 17.03 (which was not present in 1.13) related to the encrypted overlay when natting is involved, and when 2+ systems are on different networks can talk internally, but again when they advertise public IPs since they need to communicate with, let's say a 3rd system externally.
Just testing:
When you launch containers on A (10.0.0.2) and B (10.0.0.3) and ping from container on B->A, you get:
On A:
On B:
Nothing has changed config wise/launch wise since upgrading from 1.13.
(otherwise, it seems to work in the other scenarios --> one system per datacenter/non-routable rfc1918 space)
I can confirm that if you create it non-encrypted, everything works.
ventz commentedon Mar 6, 2017
@aboch And possibly related
System A has:
vs
System B has:
^ Note that System B (the joining one) gets the "Remote-addr" as the --advertise-addr (public IP), but System A does not see it.
The commands used to create and join the two are:
System A:
System B:
Just to be exact -- it doesn't work from the moment it is created, and this same setup worked in 1.13.
If you join a 3rd node outside of the datacenter (System C), both A and B can talk with it individually. But A still can't talk to B and vice versa.
ventz commentedon Mar 6, 2017
Last possibly interesting update - on a new swarm create+encrypted overlay, it seems you can generate ESP packets one way (no matter if FIRST you try from A->B or B->A...but there won't be a return. From that point, the other direction does not work. So if I start with container on A pinging container on B -- I'll see the pri(A)->PUB(B) and on B packets coming in. But never an attempt to reply. If I stop and reverse direction, there wont' be anything. If I leave the swarm, and start again and try in the reverse direction - again the traffic will show up, but only in that direction, and the reverse will not work. Seems to be something related to the way the encryption is handling the NAT/advertised IP.
aboch commentedon Mar 6, 2017
Hi @ventz, yes something looks wrong from your logs.
On System A you should see outgoing ESP packets as PRI1 -> PUB2 and incoming as PUB2->PRI1.
Instead from your logs
it looks like system A is aware of private IP of system B.
This is one problem that needs to be investigated. You should not see any ipsec tunnels with the PRI IP of any remote node in a swarm created with that configuration.
I am not aware of other changes in that area which could have caused this. Also, I have a 3 nodes swarm running on AWS with encryption via public IPs and it is running smooth for more than 2 weeks.
That is expected to be seen only on the node that joins the swarm. During the encryption setup, the remote node PUB IP is learned from the nw control plane, not from that parameter.
As of now I cannot think of what could have caused your issue. I'll get back if I get any idea.
emcgee commentedon May 19, 2018
So we're seeing something quite similar to what is described above. It may also be related to #33133
We've noticed that we're unable to send any data traffic through an encrypted overlay to an endpoint behind AWS/GCE NAT on Debian 9, Ubuntu 18.04, or anything with a kernel > 4.4.
The only modern Debian variant that works is 16.04 with the 4.4 kernel.
Simple repro:
node-1on DigitalOcean,node-2on GCE/AWS.Spin up a Debian Stretch host (4.9 kernel) on both DigitalOcean and GCE or AWS with the latest stable Docker-CE (18.03.1)
Initialize the swarm on DigitalOcean
node-1using--advertise-addr external_ipBy default, DigitalOcean has no firewall so
node-1is wide open. Open TCP 2377, TCP/UDP 7946, UDP 4789, Protocol 50 onnode-2in the AWS Security Group/GCP VPC Firewall Rules and join the swarm as a worker using--advertise-addr node-2-external-ipdocker create network --attachable --driver overlay --opt encrypted encryption_testdocker run -ti --rm --network encryption_test debian bashIf you do this with a non-encrypted overlay, traffic flows with no issues. We're able to ping between containers, run iperf3; all is well. But on the encrypted overlay, the traffic simply won't transmit.
ip xfrm stategives:We can see ESP traffic on the remote host (not the one doing the pinging) but there is no return:
There seems to be a change here in how the kernel is processing the traffic in > 4.4 - has anyone else seen this?
/cc @aboch @ventz
thaJeztah commentedon May 19, 2018
@emcgee could you open a new ticket so that it doesn't get lost? (this issue is closed, so comments get easily overlooked