Discussion about the files in the /boot and initrd

[ader1990]

This PR was opened to have a place where to discuss line by line on the files that we can or cannot remove from the /boot partition and initrd. Or maybe move them into the /usr.

This is a follow up on @pothos suggestion to cut things we do not need: flatcar/Flatcar#1381 (comment)

How the data was compiled:

# Content of the /boot partition for AMD64 built from
# flatcar/scripts commit a842366ed9a2c2b1204a87f8563ef09e2950e5e3
tree -sifF  /boot | sort -k2 -rn

# Main data
$: df | grep '/boot'
/dev/vda1          129039      64135      64905  50% /boot

$: file flatcar_production_image.vmlinuz
flatcar_production_image.vmlinuz: Linux kernel x86 boot executable bzImage, version 6.6.17-flatcar (build@pony-truck.infra.kinvolk.io) #1 SMP PREEMPT_DYNAMIC Mon Mar  4 18:50:2A

$: du flatcar_production_image.vmlinuz
56124   flatcar_production_image.vmlinuz

Added another commit with the initrd contents.

How the data was compiled:

bash extract-initramfs-from-vmlinuz.sh flatcar_production_image.vmlinuz vmlinuz
cd vmlinuz/
tree -sifF . | sort -k2 -rn

[pothos]

Thanks, we can also discuss the kernel config: https://alpha.release.flatcar-linux.net/amd64-usr/current/flatcar_production_image_kernel_config.txt

[github-actions]

Build action triggered: https://github.com/flatcar/scripts/actions/runs/16299456269

[jepio]

these are all the same file and i want to say that we only need EFI/boot/grubx64.efi, but this needs testing.

[pothos]

Unless we would also remove them in the update postinst hook these would still occupy space to prevent an update - but even then for some old clients the postinst hook might be too late because the kernel can't even be extracted to /boot.

[pothos]

In theory we could the network firmware and kernel modules from /sysusr/usr but it needs some wrangling to have them available at the right time…

[jepio]

Same file, and i'm not convinced pvboot elf works. Hasn't been tested in years/ever.

[pothos]

The microcode updates can be delayed by defining a service that does echo 1 > /sys/devices/system/cpu/microcode/reload (see https://www.kernel.org/doc/Documentation/x86/microcode.txt) once we would symlink/copy them to /lib/firmware/{intel-ucode,amd-ucode} in the initrd from /sysusr.

[pothos]

That binary could be symlinked to /sysusr/, same for xfs* and mkfs.* stuff and even ip and mdadm

[pothos]

Small win, but that file could be symlinked.

[pothos]

My impression is that we should start with the big stuff: find a way to load network firmware and kernel modules, and CPU microcode update from /sysusr/usr/.

[jepio]

can you also check how well individual files compress? this might make a big difference

[ader1990]

this means if the files we d like to move/remove have a great compression rate, it does not make too much sense to remove them?

[jepio]

the initrd is compressed with some algorithm (zstd i think), and so we would want to focus on files that are big when compressed so that removing them actually has the desired impact. Otherwise we might remove a file that looks big but doesn't give much benefit because it compresses to something really small.

[jepio]

i dont think we have anything that would call xfs_repair from the initrd

[ader1990]

Thanks, we can also discuss the kernel config: https://alpha.release.flatcar-linux.net/amd64-usr/current/flatcar_production_image_kernel_config.txt

Added this file for discussion.

[ader1990]

I will keep in this updated comment a log of the AMD64 /boot partition size history, to have a clear picture on the size increase over time:

# 2024-03-12, Flatcar HEAD https://github.com/flatcar/scripts/commit/89cca15171f0b2424a4c132bb0c2138159b7eb6e
/dev/sda1         129039       59962      69078  47% /boot
# 2024-03-18, Flatcar ALPHA 3913.0.0 https://alpha.release.flatcar-linux.net/amd64-usr/3913.0.0/
/dev/sda1         129039       61753     67287  48% /boot
# 2024-03-25, Flatcar HEAD https://github.com/flatcar/scripts/commit/6ebffa7a4fde3a1b2c8d362b065b35b4ce3a43e4
/dev/sda1         129039       62503     66536  49% /boot

[ader1990]

Hello, time to resurrect this thread @flatcar/flatcar-maintainers, with the current lists, to investigate if there are some more drivers or files that are not used or less likely to be used.

[chewi]

In the past year, I have already made savings wherever possible with GRUB, changed how we compress the kernel image, and trimmed parts of the initrd in the face of unavoidable increases elsewhere. It's very tight now, and trying to squeeze out more savings is just prolonging the inevitable. I've had a plan since the start of the year (which I have discussed openly) to move the kernel image out of /boot into /usr. Updating Dracut was just the start of that, and now I'm moving onto the main effort. Just bear with me.

[chewi]

Small update. Jeremi has requested that I write up the details of what I'm proposing. That's half done now, and I'll finish it soon, but I was also on the verge of a major milestone, so I was keen to go that little bit further. Flatcar now boots with the kernel under /usr, the kernel parameters and verity root hash GPG signed for verification by GRUB, and the verity root hash also S/MIME signed for verification by the kernel. 🙂

[chewi]

As promised, here is the write up.

[chewi]

Small update. I'm currently digesting all the feedback and trying to rework the plan to address the main concerns. Most notably, I have come up with a working thin wrapper around systemd-veritysetup-generator that can translate Flatcar's old verity kernel command line arguments into the new style. This was initially done so that the old GRUB could boot new releases, but given that the microcode update will immediately push everyone over the edge, a new GRUB config will be needed regardless. However, it does allow the bootengine changes to be merged ahead of the rest.

[jepio]

This was initially done so that the old GRUB could boot new releases, but given that the microcode update will immediately push everyone over the edge, a new GRUB config will be needed regardless.

What's the link between the microcode update and needing a new GRUB config? I don't think we have a shared understanding on the team in this area.

Has somebody looked into why an incremental microcode update would lead to such a jump in size? We really should be more conservative with what we allow into the kernel/initrd with regards to code size - we can't just keep growing without any control. And yes - this does require solid engineering effort.

[chewi]

Being able to boot new releases with the old GRUB config assumes that at least some users will have enough space left in /boot to install the new kernels there for a while before they really do become too large. This microcode update inflates the image by about 3MB, which is roughly how much space we have left now in the best case. All users would therefore have no choice but to at least take a new GRUB config, if not a new GRUB image.

Krzesimir observed this size increase in CI. As yet, I have only looked at the size of the original tarball. This is also about 3MB larger than the previous one, so that tallies. Krzesimir said that the increase is due to a new Intel CPU family. We can probably postpone that for a short time, but if anything, it's the older CPU families we should be pruning. I imagine the updates for the really old ones might be quite small though.

Having worked closely with Dracut, there are only small gains to be had now. I already had to work hard to avoid going over the limit with the recent update, e.g. preventing the inclusion of libgcrypt. I would prefer to have more space to enable the features users would like to have than to break our backs keeping the size down when it will inevitably not be enough.

[chewi]

In case you missed it on Matrix, I'm now working on an entirely new idea. That's not to say the old idea was bad, this one is just better, and most importantly, much simpler. 😁

While comparing what we do with Bottlerocket, I was intrigued by the fact that they do not use an initrd at all. That got me thinking about why Flatcar does use one, and the only reason seems to be that our image is for /usr, not /. Isn't it also because we need to be able to wipe the ROOT before mounting USR? No, because we already mount USR at /sysusr to get Ignition before that happens. So, why don't we just mount it at /usr in the first place? How much of an initrd do you need to do that? Hardly anything, it turns out.

After copying some bits from our existing Dracut-based initrd to USR like the Ignition systemd units, I have more or less managed to boot Flatcar normally using an initrd that has just:

• A few empty directories and symlinks.
• A sysroot.mount unit for ROOT.
• A sysroot-usr.mount unit to bind mount from /usr.
• A static busybox.
• This shell script:

#!/busybox ash

set -e
/busybox mount -t devtmpfs devtmpfs /dev
/busybox mount -t proc proc /proc
/busybox mount -o ro /dev/vda3 /usr
exec /lib/systemd/systemd "${@}"

Okay, I've hardcoded /dev/vda3 and there's no verity here, but this is just a prototype. I know from Bottlerocket that adding verity wouldn't be hard. Don't like busybox? That could be changed, but I think it's fine.

Any drivers needed to access the disk will admittedly need to change from being modules to built-ins, but I don't believe that will hurt much. With this approach, it would be worth me going through the kernel config see whether anything currently built-in could become a module.

The upshot? Excluding microcode, instead of an 86MB uncompressed initrd, we now have a 2.4MB one. No kernel move or GRUB config changes required. It also allows users to use tools in early boot that are currently not available until after switching root.

Suffice to say I'm, going to divert onto this plan. Suggestions welcome.

[jepio]

Isn't it also because we need to be able to wipe the ROOT before mounting USR? No, because we already mount USR at /sysusr to get Ignition before that happens. So, why don't we just mount it at /usr in the first place? How much of an initrd do you need to do that? Hardly anything, it turns out.

Interesting concept - but there is one thing in the description that isn't quite right. We do need to be able to wipe/reformat ROOT (as in partition 9) before switching to it. Wiping ROOT doesn't mean wiping USR.

It also allows users to use tools in early boot that are currently not available until after switching root

early boot meaning initrd? I don't think we have any extensibility points for users to control the initrd execution.

Any drivers needed to access the disk will admittedly need to change from being modules to built-ins, but I don't believe that will hurt much. With this approach, it would be worth me going through the kernel config see whether anything currently built-in could become a module.

Let us know what you find, there's quite many storage configurations we support.

This is somehow similar to an idea i've had (never tried it) of moving networking modules/firmware out of the initrd and loading them from /sysusr/.

[chewi]

Interesting concept - but there is one thing in the description that isn't quite right. We do need to be able to wipe/reformat ROOT (as in partition 9) before switching to it. Wiping ROOT doesn't mean wiping USR.

Yeah, I did mean that, sorry if I wasn't clear. What I was trying to say was we already mount USR before ROOT even though we can wipe ROOT.

early boot meaning initrd? I don't think we have any extensibility points for users to control the initrd execution.

I was thinking about things like Clevis now being able to use PKCS#11, but we don't currently include those bits because they're too big.

I've since had a concern that there may be more exotic setups we need to support where USR isn't simply on a local disk. PXE springs to mind, and I can probably handle that, but there are others?