CoreOS can't run a container with "MountFlags=shared" enabled on docker daemon since 2079.3.0

[aboubeta]

Issue Report

CoreOS can't run a container with "MountFlags=shared" enabled on docker daemon since 2079.3.0

Bug

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2079.3.0
VERSION_ID=2079.3.0
BUILD_ID=2019-04-22-2119
PRETTY_NAME="Container Linux by CoreOS 2079.3.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

• AWS

ip-192-168-0-21 ~ # docker version
Client:
 Version:           18.06.3-ce
 API version:       1.38
 Go version:        go1.10.8
 Git commit:        d7080c1
 Built:             Tue Feb 19 23:07:53 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.3-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       d7080c1
  Built:            Tue Feb 19 23:07:53 2019
  OS/Arch:          linux/amd64
  Experimental:     false
ip-192-168-0-21 ~ # docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.06.3-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: a592beb5bc4c4092b1b1bac971afed27687340c5
init version: v0.13.2 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
 seccomp
  Profile: default
 selinux
[Service]
Kernel Version: 4.19.34-coreos
Operating System: Container Linux by CoreOS 2079.3.0 (Rhyolite)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 579.8MiB
Name: ip-192-168-0-21.eu-west-1.compute.internal
ID: 3LGY:MARW:YRNS:P6VO:BM5N:CH25:KFQR:AAWK:DYJV:FDNN:LJGZ:FLWO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Expected Behavior

ip-192-168-0-21 ~ # docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:92695bc579f31df7a63da6922075d0666e565ceccad16b59c3374d2cf4e8e50e
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Actual Behavior

ip-192-168-0-21 ~ # docker run hello-world
/run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/hello\": stat /hello: no such file or directory": unknown.

Reproduction Steps

• On a clean CoreOS machine

ip-192-168-0-21 ~ # docker run hello-world ## Works fine without MountFlags=shared
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:92695bc579f31df7a63da6922075d0666e565ceccad16b59c3374d2cf4e8e50e
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

ip-192-168-0-21 ~ # systemctl edit docker.service ## Add MountFlags=shared
ip-192-168-0-21 ~ # systemctl daemon-reload
ip-192-168-0-21 ~ # systemctl restart docker
ip-192-168-0-21 ~ # docker run hello-world
/run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/hello\": stat /hello: no such file or directory": unknown.

Related

https://docs.docker.com/engine/release-notes/#18063-ce
https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation

[dm0-]

Upstream commit systemd/systemd@37ed15d seems to restore the previous behavior, so I can backport it, but note that Docker recommends not setting MountFlags=shared on docker.service:

https://docs.docker.com/engine/release-notes/#18091

(Container Linux has always had containerd managed by systemd; not just for 18.09.1.)

[aboubeta]

At least a big note in the release notes would be great. What consequences could have deleting the flag in a Kubernetes cluster?

[dm0-]

A fix is included in the alpha and beta channels, if you want to test it.

[bgilbert]

This should be fixed in stable 2079.5.0, due shortly. Thanks for reporting.