setup-root: added support for air-gapped updates

[sgaraev]

It is currently impossible to use your own nebraska server in air-gapped environments to enable extensions without updating the OS version or to download missing extensions

How to use

Enable necessary kernel parameters:

kernel_arguments:
  should_exist:
    - flatcar.release_file_server_url=https://update.release.flatcar-linux.net
    - flatcar.dev_file_server_url=https://bincache.flatcar-linux.net/images

Testing done

A image was created on the flatcar-4426 branch and extensions were added to the local nginx file server.
With kernel_arguments, without Internet access:

# butane.yaml:
variant: flatcar
version: 1.0.0
storage:
  files:
    - path: /etc/flatcar/enabled-sysext.conf
      mode: 0644
      contents:
        inline: |
          python
          zfs
kernel_arguments:
  should_exist:
    # - flatcar.release_file_server_url=http://172.18.0.1:8080/flatcar
    - flatcar.dev_file_server_url=http://172.18.0.1:8080/flatcar


# logs:
Oct 09 14:47:09 localhost ignition[1142]: INFO     : Ignition finished successfully
Oct 09 14:47:09 localhost systemd[1]: Finished ignition-files.service - Ignition (files).
Oct 09 14:47:09 localhost systemd[1]: Starting ignition-quench.service - Ignition (record completion)...
Oct 09 14:47:09 localhost systemd[1]: Starting initrd-setup-root-after-ignition.service - Root filesystem completion...
Oct 09 14:47:09 localhost systemd[1]: ignition-quench.service: Deactivated successfully.
Oct 09 14:47:09 localhost systemd[1]: Finished ignition-quench.service - Ignition (record completion).
Oct 09 14:47:09 localhost initrd-setup-root-after-ignition[1188]: grep: /sysroot/usr/share/flatcar/enabled-sysext.conf: No such file or directory
Oct 09 14:47:09 localhost initrd-setup-root-after-ignition[1173]: Did not find /etc/flatcar/sysext/flatcar-python-4426.1.0+nightly-20251007-2100.raw
Oct 09 14:47:09 localhost systemd[1]: afterburn-network-kargs.service - Afterburn Initrd Setup Network Kernel Arguments was skipped because no trigger condition checks were met.
Oct 09 14:47:09 localhost initrd-setup-root-after-ignition[1203]: Args { output_dir: "/sysroot/ue-rs/", target_filename: None, input_xml: None, payload_url: Some("http://172.18.0.1:8080/flatcar/amd64/4426.1.0+nightly-20251007-2100/flatcar_test_update-flatcar-python.gz"), pubkey_file: "/sysroot/usr/share/update_engine/update-payload-key.pub.pem", image_match: [], take_first_match: false }
Oct 09 14:47:09 localhost initrd-setup-root-after-ignition[1203]: writing to /sysroot/ue-rs/.unverified/flatcar_test_update-flatcar-python.gz
Oct 09 14:47:10 localhost initrd-setup-root-after-ignition[1203]: Parsed and verified signature data from file "/sysroot/ue-rs/.unverified/flatcar_test_update-flatcar-python.gz"
Oct 09 14:47:10 localhost initrd-setup-root-after-ignition[1173]: Did not find /etc/flatcar/sysext/flatcar-zfs-4426.1.0+nightly-20251007-2100.raw
Oct 09 14:47:10 localhost systemd[1]: afterburn-network-kargs.service - Afterburn Initrd Setup Network Kernel Arguments was skipped because no trigger condition checks were met.
Oct 09 14:47:10 localhost initrd-setup-root-after-ignition[1220]: Args { output_dir: "/sysroot/ue-rs/", target_filename: None, input_xml: None, payload_url: Some("http://172.18.0.1:8080/flatcar/amd64/4426.1.0+nightly-20251007-2100/flatcar_test_update-flatcar-zfs.gz"), pubkey_file: "/sysroot/usr/share/update_engine/update-payload-key.pub.pem", image_match: [], take_first_match: false }
Oct 09 14:47:10 localhost initrd-setup-root-after-ignition[1220]: writing to /sysroot/ue-rs/.unverified/flatcar_test_update-flatcar-zfs.gz
Oct 09 14:47:10 localhost initrd-setup-root-after-ignition[1220]: Parsed and verified signature data from file "/sysroot/ue-rs/.unverified/flatcar_test_update-flatcar-zfs.gz"
Oct 09 14:47:10 localhost systemd[1]: Finished initrd-setup-root-after-ignition.service - Root filesystem completion.
Oct 09 14:47:10 localhost systemd[1]: Reached target ignition-complete.target - Ignition Complete.


# Checks:
test-vm ~ # cat /proc/cmdline       
rootflags=rw mount.usrflags=ro BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=7130c94a-213a-4e5a-8e26-6cce9662f132 rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 flatcar.first_boot=detected flatcar.oem.id=qemu flatcar.dev_file_server_url=http://172.18.0.1:8080/flatcar verity.usrhash=e96b276db8e4d370fb6f1915cdfd8cfb4100bcefa977451d59b34e84f4e65d74

test-vm ~ # ls -lah /etc/flatcar/sysext/        
total 25M
drwxr-xr-x. 2 root root 4.0K Oct  9 14:47 .
drwxr-xr-x. 1 root root 4.0K Oct  9 14:47 ..
-rw-r--r--. 1 root root  21M Oct  9 14:47 flatcar-python-4426.1.0+nightly-20251007-2100.raw
-rw-r--r--. 1 root root 4.0M Oct  9 14:47 flatcar-zfs-4426.1.0+nightly-20251007-2100.raw

Without kernel_arguments

# butane.yaml:
variant: flatcar
version: 1.0.0
storage:
  files:
    - path: /etc/flatcar/enabled-sysext.conf
      mode: 0644
      contents:
        inline: |
          python
          # zfs

core@localhost ~ $ cat /etc/flatcar/enabled-sysext.conf 
python
# zfs
core@localhost ~ $ ls -lah /etc/flatcar/sysext/
total 21M
drwxr-xr-x. 2 root root 4.0K Oct  9 15:44 .
drwxr-xr-x. 1 root root 4.0K Oct  9 15:44 ..
-rw-r--r--. 1 root root  21M Oct  9 15:44 flatcar-python-4426.1.0+nightly-20251007-2100.raw
core@localhost ~ $ cat /proc/cmdline 
rootflags=rw mount.usrflags=ro BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=7130c94a-213a-4e5a-8e26-6cce9662f132 rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 flatcar.first_boot=detected flatcar.oem.id=qemu flatcar.autologin verity.usrhash=e96b276db8e4d370fb6f1915cdfd8cfb4100bcefa977451d59b34e84f4e65d74


Oct 09 15:44:12 localhost ignition[1121]: INFO     : Ignition finished successfully
Oct 09 15:44:12 localhost systemd[1]: Finished ignition-files.service - Ignition (files).
Oct 09 15:44:12 localhost systemd[1]: Starting ignition-quench.service - Ignition (record completion)...
Oct 09 15:44:12 localhost systemd[1]: Starting initrd-setup-root-after-ignition.service - Root filesystem completion...
Oct 09 15:44:12 localhost systemd[1]: ignition-quench.service: Deactivated successfully.
Oct 09 15:44:12 localhost systemd[1]: Finished ignition-quench.service - Ignition (record completion).
Oct 09 15:44:12 localhost initrd-setup-root-after-ignition[1151]: grep: /sysroot/usr/share/flatcar/enabled-sysext.conf: No such file or directory
Oct 09 15:44:12 localhost initrd-setup-root-after-ignition[1136]: Did not find /etc/flatcar/sysext/flatcar-python-4426.1.0+nightly-20251007-2100.raw
Oct 09 15:44:12 localhost systemd[1]: afterburn-network-kargs.service - Afterburn Initrd Setup Network Kernel Arguments was skipped because no trigger condition checks were met.
Oct 09 15:44:12 localhost systemd-networkd[911]: eth0: Gained IPv6LL
Oct 09 15:44:13 localhost initrd-setup-root-after-ignition[1166]: Args { output_dir: "/sysroot/ue-rs/", target_filename: None, input_xml: None, payload_url: Some("https://bincache.flatcar-linux.net/images/amd64/4426.1.0+nightly-20251007-2100/flatcar_test_update-flatcar-python.gz"), pubkey_file: "/sysroot/usr/share/update_engine/update-payload-key.pub.pem", image_match: [], take_first_match: false }
Oct 09 15:44:13 localhost initrd-setup-root-after-ignition[1166]: writing to /sysroot/ue-rs/.unverified/flatcar_test_update-flatcar-python.gz
Oct 09 15:44:26 localhost initrd-setup-root-after-ignition[1166]: Parsed and verified signature data from file "/sysroot/ue-rs/.unverified/flatcar_test_update-flatcar-python.gz"
Oct 09 15:44:26 localhost systemd[1]: Finished initrd-setup-root-after-ignition.service - Root filesystem completion.
Oct 09 15:44:26 localhost systemd[1]: Reached target ignition-complete.target - Ignition Complete.

• Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
• Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

[sgaraev]

flatcar/Flatcar#1520

[pothos]

Thanks, we for sure need better docs for this.

I thought that the Ignition approach is good enough and for manual usage one would copy the file while adding the extension entry. Having a way of specifying an own URL makes it a bit nicer because there is no version anymore to keep up to date in the Ignition file, so thanks for the PR!

I don't think we need to specify dev_public_key_md5 - this is really just for the default key and as soon as another one is used, this would be one's own release and one lands in the other case.

[sgaraev]

@pothos Thanks, I updated the commit.