Enable virtio-fs kernel module for shared directory support in QEMU VMs

[aaronk6]

Add CONFIG_VIRTIO_FS=m to commonconfig-6.6

How to use

Create a Flatcar QEMU/KVM machine with a virtiofs share called my_mount.

In the VM, run the following commands:

modprobe virtiofs
mount -t virtiofs my_mount /mnt/my_mount

As a result, the host directory should be mounted at /mnt/my_mount.

Testing done

As discussed in flatcar/Flatcar#1657, no testing has been done yet. Testing will be done once a build has been published by the CI.

[felinira]

While we're at it, we can consider enabling CONFIG_FUSE_DAX=y for performance improvements. See https://virtio-fs.gitlab.io/howto-qemu.html for details. The option allows deduplicating the kernel page cache between host and virtual machine entirely.

See for example https://github.com/rhvgoyal/virtiofs-tests/blob/master/performance-results/dec-10-2020/results-different-caching-modes.txt for some benchmark numbers, it seems to be quite effective. (triple the performance in many benchmarks)

[aaronk6]

That’s very interesting. Thanks for sharing those benchmarks! Quite significant. I’ve added CONFIG_FUSE_DAX=y to the PR.

According to home-assistant/operating-system@11c2eab, there are more kernel modules that might be related to virtio-fs.

Module	Note
CONFIG_FUSE_FS	✅ Already included in Flatcar
CONFIG_VIRTIO_FS	➡️ Added by this PR
CONFIG_FUSE_DAX	➡️ Added by this PR
CONFIG_DAX	⤴️ Dependency of CONFIG_FUSE_DAX
CONFIG_FS_DAX	⤴️ Dependency of CONFIG_FUSE_DAX
CONFIG_ZONE_DEVICE	❓ Added by this PR, but is it needed?

My understanding is that dependent modules don’t need to be specified explicitly, so I’m not adding CONFIG_DAX and CONFIG_FS_DAX (just CONFIG_FUSE_DAX as suggested by @felinira).

I’m unsure about CONFIG_ZONE_DEVICE though. I’ve added it, but do you know if it’s a requirement for DAX?

I’m also unsure if it’s better to bake them into the kernel, or have them as modules. For now, I’m adding them as modules to avoid increasing the kernel size. I see that the existing CONFIG_FUSE_FS option is also added as a module. Let’s see if this works.

By the way, the Home Assistant commit also adds these:

CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
CONFIG_VIRTIO_IOMMU=m
CONFIG_CRYPTO_ENGINE=m
CONFIG_CRYPTO_DEV_VIRTIO=m

From what I can tell, these are unrelated to virtio-fs (even though they are in the same commit), so I’m not going to include them.

Let me know your thoughts.

[felinira]

I don't think you can set CONFIG_FUSE_DAX to m, it's a boolean option that doesn't support modularisation

https://github.com/torvalds/linux/blob/master/fs/fuse/Kconfig#L43

It is configured as bool, not tristate. So this won't compile.

I’m unsure about CONFIG_ZONE_DEVICE though. I’ve added it, but do you know if it’s a requirement for DAX?

https://github.com/torvalds/linux/blob/master/mm/Kconfig#L1104

I don't know either, but the direct advice is to enable it when FS_DAX is enabled. Grepping through the kernel it does not seem to add much code size at all, just select different code paths with perhaps slightly different performance characteristics.

[aaronk6]

@felinira I’ve updated the PR to set both CONFIG_FUSE_DAX and CONFIG_ZONE_DEVICE to y. Thanks for pointing out that they’re booleans. Based on the comment in https://github.com/torvalds/linux/blob/master/mm/Kconfig#L1104, I think it does makes sense to enable CONFIG_ZONE_DEVICE.

[jepio]

Had to enable FS_DAX to fix a build failure.

Amd64 images are here for testing: https://bincache.flatcar-linux.net/images/amd64/9999.0.0+virtiofs.1/.

Arm64 is failing with:

12:20:41  make[1]: Leaving directory '/build/arm64-usr/var/tmp/portage/sys-kernel/coreos-modules-6.6.84/work/coreos-modules-6.6.84/build'
12:20:41  make: Leaving directory '/build/arm64-usr/var/tmp/portage/sys-kernel/coreos-modules-6.6.84/work/coreos-modules-6.6.84/source'
12:20:41   �[31;01m*�[0m ERROR: sys-kernel/coreos-modules-6.6.84::coreos-overlay failed (configure phase):
12:20:41   �[31;01m*�[0m   Requested options not enabled in build:
12:20:41   �[31;01m*�[0m     CONFIG_FS_DAX
12:20:41   �[31;01m*�[0m     CONFIG_FUSE_DAX
12:20:41   �[31;01m*�[0m     CONFIG_ZONE_DEVICE
12:20:41   �[31;01m*�[0m 
12:20:41   �[31;01m*�[0m Call stack:
12:20:41   �[31;01m*�[0m     ebuild.sh, line  136:  Called src_configure
12:20:41   �[31;01m*�[0m   environment, line 2183:  Called coreos-kernel_src_configure
12:20:41   �[31;01m*�[0m   environment, line  763:  Called die
12:20:41   �[31;01m*�[0m The specific snippet of code:
12:20:41   �[31;01m*�[0m           die "Requested options not enabled in build:
12:20:41   �[31;01m*�[0m ${missing}";
12:20:41   �[31;01m*�[0m 

[github-actions]

Build action triggered: https://github.com/flatcar/scripts/actions/runs/14905926619

[jepio]

Images from the latest commit here: https://bincache.flatcar-linux.net/images/amd64/9999.0.0+virtiofs.2/
Let me know how these work for this usecase.

[aaronk6]

@jepio Thanks for the additional fixes to make this work! I’ll check out the build and let you know.

[aaronk6]

Works fine at first glance, using the Butane snippet below: I can browse the mounted directory and was able to create a file. 🎉

storage:
  files:
    - path: /etc/systemd/system/var-mnt-storage.mount
      mode: 0644
      contents:
        inline: |
          [Unit]
          Description=Mount shared storage (virtiofs)
          
          [Mount]
          What=storage
          Where=/var/mnt/storage
          Type=virtiofs

          [Install]
          WantedBy=multi-user.target

It seems as if the virtiofs kernel module gets loaded automatically. I didn’t need to add any extra module loading config, which is great.

I will do further testing next week, for example, benchmarking performance against my existing FCOS machine and playing with DAX.

@felinira @skydiablo Do you also have some time to test?

[skydiablo]

I'm on vacation, after eastern i will give it a try.

Thx for Invest!

[felinira]

This works fine for me. However we can probably disable dax again, as it is not actually supported on recent virtiofsd daemons. Sorry for the noise, I expected this to still work, but alas it does not. See https://gitlab.com/virtio-fs/virtiofsd/-/issues/41#note_1869838311 for details.

I made some benchmarks.

baseline - direct scsi disk access from VM host:

# podman run -v /var/bench:/data docker.io/xridge/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test.file --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=65

Run status group 0 (all jobs):
   READ: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=665MiB (698MB), run=5249-5249msec
  WRITE: bw=68.3MiB/s (71.7MB/s), 68.3MiB/s-68.3MiB/s (71.7MB/s-71.7MB/s), io=359MiB (376MB), run=5249-5249msec

virtiofsd mounted with default options:

# podman run -v /mnt/bench:/data docker.io/xridge/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test.file --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=65

Run status group 0 (all jobs):
   READ: bw=11.6MiB/s (12.1MB/s), 11.6MiB/s-11.6MiB/s (12.1MB/s-12.1MB/s), io=665MiB (698MB), run=57567-57567msec
  WRITE: bw=6381KiB/s (6534kB/s), 6381KiB/s-6381KiB/s (6534kB/s-6534kB/s), io=359MiB (376MB), run=57567-57567msec

[skydiablo]

I wanted to let you know that the integration of virtiofs is working smoothly for me and meets all expectations. There are no issues or difficulties, and everything is running as expected.

Thanks for the great work!

volker.

[jepio]

I've rebased, will run another CI build to see that everything is fine and then i'll get this merged. Thanks!

[jepio]

CI passed here: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/5856/
(Openstack hit quota but Qemu/Vmware is fine).

[jepio]

I've added a changelog.

Follow-up for some day: I noticed that fuse modules are part of the initrd, they don't seem to be needed there.

[aaronk6]

@jepio Thank you for making this happen!

[skydiablo]

oh yeah! great! can you estimate when will this be applied to a stable version?

[skydiablo]

@jepio sry for re-asking, but current public releases aren't support virtiofs? in my case this one: "https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img"

or i'm wrong?

[ader1990]

@jepio sry for re-asking, but current public releases aren't support virtiofs? in my case this one: "https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img"

or i'm wrong?

only the recently released Alpha has the support. For the other channels, this feature needs to be backported, which has not been proposed yet.

[skydiablo]

so i tried this image: https://alpha.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img

but not success with my mount:

systemd:
  units:
    - name: mnt-docker.mount
      enabled: true
      contents: |
        [Unit]
        Description=Mount shared storage (virtiofs)

        [Mount]
        What=docker
        Where=/mnt/docker
        Type=virtiofs

        [Install]
        WantedBy=multi-user.target

here log output:

$ journalctl -u mnt-docker.mount -n 100
May 08 12:14:47 docker-runner-230 mount[1045]: mount: /mnt/docker: unknown filesystem type 'virtiofs'.
May 08 12:14:47 docker-runner-230 mount[1045]:        dmesg(1) may have more information after failed mount system call.
May 08 12:14:49 docker-runner-230 systemd[1]: Mounting mnt-docker.mount - Mount shared storage (virtiofs)...
May 08 12:14:49 docker-runner-230 mount[1243]: mount: /mnt/docker: unknown filesystem type 'virtiofs'.
May 08 12:14:49 docker-runner-230 mount[1243]:        dmesg(1) may have more information after failed mount system call.
May 08 12:14:49 docker-runner-230 systemd[1]: mnt-docker.mount: Mount process exited, code=exited, status=32/n/a
May 08 12:14:49 docker-runner-230 systemd[1]: mnt-docker.mount: Failed with result 'exit-code'.
May 08 12:14:49 docker-runner-230 systemd[1]: Failed to mount mnt-docker.mount - Mount shared storage (virtiofs).

so i checked the kernel module:

$ modprobe virtiofs
modprobe: FATAL: Module virtiofs not found in directory /lib/modules/6.6.83-flatcar

hmmm, seems to be missing, so maybe the current alpha is also not re-created since this merge?

[ader1990]

@skydiablo yes, sorry for that - the alpha release has not been updated yet for this cycle. There is work in progress on the alpha release, will update here once it has been done.

[ader1990]

There are nightly builds of the latest branch, you can check the latest one here: https://bincache.flatcar-linux.net/images/amd64/4327.0.0+nightly-20250506-2100/ or https://bincache.flatcar-linux.net/images/arm64/4327.0.0+nightly-20250506-2100/.