Ship docker sysext and rip out torcx [WIP]

[krishjainx]

Ship Docker and Containerd Sysext and deprecate torcx

Solves flatcar/Flatcar#443

Related Issues: flatcar/Flatcar#1092, flatcar/Flatcar#1093

Once we have it building in the CI people can download the image from there and play around already.

How to use

@pothos recommended I file a draft PR for aiding communication

Testing done

When I attempt to use run build_image script and run the launch script the Flatcar provided sysext works.
1.

core@localhost ~ $ sudo systemctl status docker.socket
● docker.socket - Docker Socket for the API
     Loaded: loaded (/usr/lib/systemd/system/docker.socket; disabled; preset: d>
     Active: active (running) since Sat 2023-07-15 20:18:27 UTC; 40s ago
   Triggers: ● docker.service
     Listen: /run/docker.sock (Stream)
      Tasks: 0 (limit: 7293)
     Memory: 0B
        CPU: 438us
     CGroup: /system.slice/docker.socket

Jul 15 20:18:27 localhost systemd[1]: Starting docker.socket - Docker Socket fo>
Jul 15 20:18:27 localhost systemd[1]: Listening on docker.socket - Docker Socke>
core@localhost ~ $ sudo systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/usr/lib/systemd/system/containerd.service; disabled; pres>
     Active: active (running) since Sat 2023-07-15 20:18:27 UTC; 45s ago
       Docs: https://containerd.io
    Process: 1281 ExecStartPre=mkdir -p /run/docker/libcontainerd (code=exited,>
    Process: 1308 ExecStartPre=ln -fs /run/containerd/containerd.sock /run/dock>
   Main PID: 1316 (containerd)
      Tasks: 15
     Memory: 68.6M
        CPU: 253ms
     CGroup: /system.slice/containerd.service
             └─1316 /usr/bin/containerd --config /usr/share/containerd/config.t>

Jul 15 20:18:27 localhost containerd[1316]: time="2023-07-15T20:18:27.957570412>
Jul 15 20:18:27 localhost containerd[1316]: time="2023-07-15T20:18:27.957619335>
Jul 15 20:18:27 localhost containerd[1316]: time="2023-07-15T20:18:27.958557496>
Jul 15 20:18:27 localhost containerd[1316]: time="2023-07-15T20:18:27.958596751>
Jul 15 20:18:27 localhost containerd[1316]: time="2023-07-15T20:18:27.958641068>
Jul 15 20:18:27 localhost containerd[1316]: time="2023-07-15T20:18:27.959347819>
Jul 15 20:18:27 localhost containerd[1316]: time="2023-07-15T20:18:27.959387454>
Jul 15 20:18:27 localhost containerd[1316]: time="2023-07-15T20:18:27.959399283>
Jul 15 20:18:27 localhost containerd[1316]: time="2023-07-15T20:18:27.960310590>
Jul 15 20:18:27 localhost systemd[1]: Started containerd.service - containerd c>
core@localhost ~ $ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
core@localhost ~ $ systemd-sysext status
HIERARCHY EXTENSIONS         SINCE                      
/opt      none               -                          
/usr      containerd-flatcar Sat 2023-07-15 20:18:27 UTC
          docker-flatcar     
          oem-qemu           
core@localhost ~ $ docker run --rm  hello-world
[   70.056820] docker0: port 1(veth5a0bf82) entered blocking state
[   70.057413] docker0: port 1(veth5a0bf82) entered disabled state
[   70.058020] device veth5a0bf82 entered promiscuous mode
[   70.229665] eth0: renamed from veth6a92a16
[   70.244588] IPv6: ADDRCONF(NETDEV_CHANGE): veth5a0bf82: link becomes ready
[   70.245833] docker0: port 1(veth5a0bf82) entered blocking state
[   70.246457] docker0: port 1(veth5a0bf82) entered forwarding state
[   70.247088] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

[   70.283602] docker0: port 1(veth5a0bf82) entered disabled state
[   70.284235] veth6a92a16: renamed from eth0
[   70.322064] docker0: port 1(veth5a0bf82) entered disabled state
[   70.322938] device veth5a0bf82 left promiscuous mode
[   70.323482] docker0: port 1(veth5a0bf82) entered disabled state

Notes for reviewers/those interested

Kai's fix seems to work! Solved

Ship docker+containerd sysext images

• Our objective is to create two distinct sysext images, one for containerd and another for Docker. To achieve this, we can consider adding a flag to the emerge command that allows skipping the installation of a specific dependency. This way, when emerging Docker for the overlay, we can skip containerd since it will be built into a separate sysext image. One possible approach is to introduce a flag like "--exclude PKG" or "--nodeps," as suggested by Kai. The "nodeps" option seems suitable for Docker, and if necessary, a dependency can be explicitly passed as a second package for the script. Another option is to generate a list of files included in the containerd sysext image and remove those files using the mangle script. [Working on locally, will push]
• We should incorporate a mangle script. The purpose of this script would be to convert the unit dependencies from "wants" to "upholds." However, for now, we can set this aspect aside since we have a workaround in place for now. The "ensure-sysext.service" restarts the targets, which helps us achieve our goals.

Remove torcx

• Remove torcx from CoreOS ebuild and delete app-arch/torcx from coreos-overlay
• Remove torcx completely [To do during internship ]
• Removing Torcx will also require rewriting the mantle kola tests. [To do during internship]

[krishjainx]

To create the two seperate sysexts we could remove containerd stuff from the docker sysext image: something like comm -12 <(find "$rootfs" -type f | sort) <(find $(dirname "$rootfs")/containerd-flatcar -type f | sort) | xargs rm. Experimenting with this

[t-lo]

To create the two seperate sysexts we could remove containerd stuff from the docker sysext image: something like comm -12 <(find "$rootfs" -type f | sort) <(find $(dirname "$rootfs")/containerd-flatcar -type f | sort) | xargs rm. Experimenting with this

We could use a temporary package.provided to make emerge believe the dependencies are already installed.

[github-actions]

Build action triggered: https://github.com/flatcar/scripts/actions/runs/6415680058

[krishjainx]

@t-lo @pothos I have already added creating two separate sysext images—one for containerd and the other for Docker—and used the mangle script. Also, I have removed torcx from the coreos base ebuild and removed app-arch/torcx from coreos-overlay.

The remaining task for the end of my internship is to rewrite the mantle kola tests to account for the removal of torcx.

[jepio]

@krishjainx can you look into adding license/version information about the packages contained in the built sysexts to our generated license and packages files.

I think that needs to be done before merging this.

[krishjainx]

@krishjainx can you look into adding license/version information about the packages contained in the built sysexts to our generated license and packages files.

I think that needs to be done before merging this.

Regarding a4b3c24 as Kai suggested to me the license call uses the package list, so it should work as long as the packages get part of the list

[t-lo]

Superseded by #1216.