All Docker containers fail to start after update to 2983.2.0

[meltonbw]

Description

After an update to 2983.2.0, all Docker containers fail to start with the message: standard_init_linux.go:228: exec user process caused: operation not permitted

Impact

Cannot start any containers. All containers fail to start at boot.

Environment and steps to reproduce

1. Set-up: Flatcar version 2983.2.0
2. Task: Booting up
3. Action(s): None
4. Error: All containers fail with the error: standard_init_linux.go:228: exec user process caused: operation not permitted

Expected behavior

Containers should start without error. I cannot start a basic container with bash from the CL either:

$ docker container run --interactive --tty --rm ubuntu bash
Unable to find image 'ubuntu:latest' locally
docker.io/library/ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c: Pulling from library/ubuntu
da7391352a9b: Pull complete
14428a6d4bcd: Pull complete
2c2d948710f2: Pull complete
Digest: sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c
Status: Downloaded newer image for ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c
Tagging ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c as ubuntu:latest
standard_init_linux.go:228: exec user process caused: operation not permitted

Additional information

$ docker info -f "{{json .}}" | jq ".SecurityOptions"
[
  "name=seccomp,profile=default",
  "name=selinux",
  "name=userns",
  "name=cgroupns"
]

$ systemctl cat docker.service
# /run/systemd/system/docker.service
[Unit]
Requires=torcx.target
After=torcx.target
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=containerd.service docker.socket network-online.target
Wants=network-online.target
Requires=containerd.service docker.socket

[Service]
EnvironmentFile=/run/metadata/torcx
Environment=TORCX_IMAGEDIR=/docker
Type=notify
EnvironmentFile=-/run/flannel/flannel_docker_opts.env
Environment=DOCKER_SELINUX=--selinux-enabled=true

# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/env PATH=${TORCX_BINDIR}:${PATH} ${TORCX_BINDIR}/dockerd --host=fd:// --containerd=/var/run/docker/libcontainerd/docker-containerd.sock $DOCKER_SELINUX $DOCKER_OPTS $DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/docker.service.d/docker-opts.conf
[Service]
Environment="DOCKER_OPTS="

[tormath1]

Hi @meltonbw,

It looks like SELinux related. What's the return of:

$ getenforce

?

[meltonbw]

$ getenforce
Enforcing

Doing setenforce 0 does not seem to help.

[tormath1]

Thanks, then could you try to deactivate the userns security options ?

In the CI, we run every test with selinux mode set to enforce but we don't test things yet with userns security option, that might be the root cause then.

• could you share the content of /etc/docker/daemon.json or any Docker customization you have ?
• do you have custom SELinux policies ?
• do you have an orchestrator like Kubernetes to run your containers ?

[meltonbw]

Disabling userns didn't help unfortunately.

$ cat /etc/docker/daemon.json
{
    "icc": false,
    "log-driver": "journald",
    "live-restore": true,
    "userland-proxy": false,
    "no-new-privileges": true,
    "dns": ["10.0.0.1", "9.9.9.9", "1.1.1.1"]
}

No custom SELinux policies and I don't use an orchestrator.

Here is the full docker info:

$ docker --debug info
Client:
 Context:    default
 Debug Mode: true

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 20.10.10
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 997b1f9905746cccc997ab5c697f838e5be519ba
 runc version: 61ab78b58f2c0c3fbfc63477f2c020e825b9789d
 init version:
 Security Options:
  seccomp
   Profile: default
  selinux
  cgroupns
 Kernel Version: 5.10.77-flatcar
 Operating System: Flatcar Container Linux by Kinvolk 2983.2.0 (Oklo)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.939GiB
 Name: flatcar-lan
 ID: <redacted>
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

[meltonbw]

Note, I am able to start a --privileged Docker container (with and without userns).

[jepio]

I think this is connected to the no-new-privileges setting, there are multiple hits when searching for problems with that and selinux.

[tormath1]

I'm still trying to reproduce it - even with no-new-privileges + SELinux in enforce mode I'm still able to start a container.

[serbaut]

We have the same issue on some hosts. It looks like the selinux policies differs between the hosts. What is the recommended way to install the default policies?

[tormath1]

@serbaut thanks for raising this too - we're currently investigating.

Do you have the same runtime spec: SELinux in enforcing mode, no-new-privileges and so on ?

[serbaut]

The following seems to fix it:

rm -rf /var/lib/selinux
ln -s /usr/lib/selinux/policy /var/lib/selinux
rm -rf /etc/selinux/mcs
ln -s /usr/lib/selinux/mcs /etc/selinux/mcs
semodule -R

[serbaut]

Do you have the same runtime spec: SELinux in enforcing mode, no-new-privileges and so on ?

# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             mcs
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

We dont have any docker config (node is running k8s).

[tormath1]

@serbaut thanks.

Having Current mode: permissive should not raised permission denied issue in the containers since it's in permissive mode. We're continuing to investigating.

[serbaut]

This is how it looks on a bad node

# ls -ld /etc/selinux/* /var/lib/selinux
lrwxrwxrwx. 1 root root   28 Apr 27  2021 /etc/selinux/config -> ../../usr/lib/selinux/config
drwxr-xr-x. 4 root root 4096 May 18 15:55 /etc/selinux/mcs
lrwxrwxrwx. 1 root root   25 Apr 27  2021 /etc/selinux/mls -> ../../usr/lib/selinux/mls
lrwxrwxrwx. 1 root root   35 May 18 13:43 /etc/selinux/semanage.conf -> ../../usr/lib/selinux/semanage.conf
lrwxrwxrwx. 1 root root   30 Apr 27  2021 /etc/selinux/targeted -> ../../usr/lib/selinux/targeted
drwxr-xr-x. 5 root root 4096 May 18 15:55 /var/lib/selinux

vs

# ls -ld /etc/selinux/* /var/lib/selinux
lrwxrwxrwx. 1 root root 28 Apr 27  2021 /etc/selinux/config -> ../../usr/lib/selinux/config
lrwxrwxrwx. 1 root root 25 Apr 27  2021 /etc/selinux/mcs -> ../../usr/lib/selinux/mcs
lrwxrwxrwx. 1 root root 25 Apr 27  2021 /etc/selinux/mls -> ../../usr/lib/selinux/mls
lrwxrwxrwx. 1 root root 35 May 18 08:42 /etc/selinux/semanage.conf -> ../../usr/lib/selinux/semanage.conf
lrwxrwxrwx. 1 root root 30 Apr 27  2021 /etc/selinux/targeted -> ../../usr/lib/selinux/targeted
lrwxrwxrwx. 1 root root 28 Apr 27  2021 /var/lib/selinux -> ../../usr/lib/selinux/policy

So depending on which image you started with you get different selinux configs.

[tormath1]

@serbaut both nodes are on the same Flatcar version ?