gpt-auto-generator fails with BTRFS root and without initramfs

[Karlson2k]

systemd version the issue has been seen with

244, 245

Used distribution

Gentoo

Expected behaviour you didn't see

No gpt-auto-generator errors in log

Unexpected behaviour you saw

gpt-auto-generator in log

[    0.000000] kernel: microcode: microcode updated early to revision 0xd6, date = 2020-04-23
[    0.000000] kernel: Linux version 5.7.19-gentoo (root@host) (gcc version 9.3.0 (Gentoo 9.3.0-r1 p3), GNU ld (Gentoo 2.33.1 p2) 2.33.1) #1 SMP Sun Aug 30 19:36:07 MSK 2020
....
[    1.675394] kernel: BTRFS: device label rootfs devid 1 transid 29806 /dev/root scanned by swapper/0 (1)
[    1.682853] kernel: BTRFS info (device nvme0n1p2): disk space caching is enabled
[    1.682854] kernel: BTRFS info (device nvme0n1p2): has skinny extents
[    1.689825] kernel: BTRFS info (device nvme0n1p2): enabling ssd optimizations
[    1.690632] kernel: VFS: Mounted root (btrfs filesystem) readonly on device 0:18.
[    1.690904] kernel: devtmpfs: mounted
....
[    1.727637] systemd[1]: systemd 245 running in system mode. (+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL -XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
....
[    1.744141] systemd-gpt-auto-generator[288]: Disabling root partition auto-detection, root= is defined.
[    1.744161] systemd-gpt-auto-generator[288]: Failed to determine block device of root file system: No such file or directory
....
[    1.754426] systemd[280]: /lib/systemd/system-generators/systemd-gpt-auto-generator failed with exit status 1
....
[    1.807716] BTRFS info (device nvme0n1p2): disk space caching is enabled
....
[    2.102263] systemd-remount-fs[303]: Remounting /...
....
[    2.134333] systemd[1]: Started udev Kernel Device Manager.
....
[    2.160822] systemd[1]: dev-nvme0n1.device: Changed dead -> plugged
[    2.160859] systemd[1]: sys-devices-pci0000:00-0000:00:1d.0-0000:02:00.0-nvme-nvme0-nvme0n1.device: Changed dead -> plugged
[    2.169856] kernel: BTRFS info (device nvme0n1p2): device fsid 43c77ec4-9cb8-48df-96a3-9024b881b015 devid 1 moved old:/dev/root new:/dev/nvme0n1p2
....
[    2.168393] systemd[1]: dev-disk-by\x2dpartuuid-5c4e98cb\x2ded7f\x2d435f\x2d83a9\x2d0b78b567b4f0.device: Changed dead -> plugged
[    2.168420] systemd[1]: dev-disk-by\x2dpartuuid-5c4e98cb\x2ded7f\x2d435f\x2d83a9\x2d0b78b567b4f0.device: Job 27 dev-disk-by\x2dpartuuid-5c4e98cb\x2ded7f\x2d435f\x2d83a9\x2d0b78b567b4f0.device/start finished, result=done
[    2.168451] systemd[1]: Found device Viper M.2 VPR100 boot.
[    2.168484] systemd[1]: dev-nvme0n1p1.device: Changed dead -> plugged
....
[    2.172881] systemd[1]: dev-disk-by\x2dpartuuid-97f8aae5\x2d86b6\x2d40b5\x2d9ffb\x2d496d80551b37.device: Changed dead -> plugged
[    2.172905] systemd[1]: dev-disk-by\x2dpartuuid-97f8aae5\x2d86b6\x2d40b5\x2d9ffb\x2d496d80551b37.device: Job 22 dev-disk-by\x2dpartuuid-97f8aae5\x2d86b6\x2d40b5\x2d9ffb\x2d496d80551b37.device/start finished, result=done
[    2.172926] systemd[1]: Found device Viper M.2 VPR100 rootfs.
[    2.172951] systemd[1]: dev-disk-by\x2did-nvme\x2deui.6479a72782353833\x2dpart2.device: Changed dead -> plugged
[    2.172970] systemd[1]: dev-disk-by\x2dpath-pci\x2d0000:02:00.0\x2dnvme\x2d1\x2dpart2.device: Changed dead -> plugged
[    2.172993] systemd[1]: dev-disk-by\x2duuid-43c77ec4\x2d9cb8\x2d48df\x2d96a3\x2d9024b881b015.device: Changed dead -> plugged
[    2.173018] systemd[1]: dev-disk-by\x2dpartlabel-rootfs.device: Changed dead -> plugged
[    2.173053] systemd[1]: sys-devices-pci0000:00-0000:00:1d.0-0000:02:00.0-nvme-nvme0-nvme0n1-nvme0n1p2.device: Changed dead -> plugged
[    2.173089] systemd[1]: dev-nvme0n1p2.device: Changed tentative -> plugged

Steps to reproduce the problem
0. Ensure that you have machine with UEFI and your drive is GPT-partitioned with BTRFS root partition.

1. Build (obtain) kernel with drivers for your HDD/SSD compiled-in (modules will not work as you need to mount your HDD/SSD before modules are loaded).
2. Create boot entry (preferably in GRUB2 loader) without initramfs and kernel command line with "root=PARTUUID=xxx-xx-xx" (substitute your PARTUUID of root partition).
3. Check errors in systemd log.

I saw similar reports with "root=/dev/...", but I checked only with PARTUUID identification.

Additional information
When kernel mounts root fs without iniramfs helper, it's mounted using fake temporal mount point alias /dev/root (as real /dev is not available before root fs). After drive is re-scanned by udev mount point is properly updated (see kernel: BTRFS info (device nvme0n1p2): device fsid xxx-x-x-xxx devid 1 moved old:/dev/root new:/dev/nvme0n1p2 in log), but gpt-auto-generator already finished with error.

Old bug report: https://bugs.freedesktop.org/show_bug.cgi?id=84689
Related Linux Kernel bug report: https://bugzilla.kernel.org/show_bug.cgi?id=89721

Workaround
As a workaround, I specified all mounts (/, swap) in /etc/fstab and disabled gpt-auto-generator completely by systemd.gpt_auto=no kernel command line parameter.
It even speeds up boot process by ~100-200 msec.

[poettering]

There's little we can do about this, btrfs currently doesn't have any suitable API that would tells us anything better than /dev/root. This needs to be fixed in the kernel, and until that happens there's very little we can do.

[Karlson2k]

There's little we can do about this, btrfs currently doesn't have any suitable API that would tells us anything better than /dev/root. This needs to be fixed in the kernel, and until that happens there's very little we can do.

Actually it could be solved just fine within gpt-auto-generator. If /dev/root is returned as device mount point, it is necessary to wait until device is rescanned by udev when proper mount point as assigned.

[poettering]

Not a fix, but an improvement: #16973

[poettering]

Actually it could be solved just fine within gpt-auto-generator. If /dev/root is returned as device mount point, it is necessary to wait until device is rescanned by udev when proper mount point as assigned.

This doesn't work, the code that uses this is a generator, runs during earliest boot to generate mount units for additional file systems. It runs way before udev runs, and it needs to complete before we invoke units, i.e. before udev can even be started.

I am tempted to close this here, it's not actionable to us beyond making the issue more discoverable, which #16973 does. Right now there's simply no way to determine the correct btrfs backing devices if no initrd is used and a btrfs scan/udev repobe started. This is something the kernel people have to fix, and if they do thinks magically start working here, too.

[Karlson2k]

Actually it could be solved just fine within gpt-auto-generator. If /dev/root is returned as device mount point, it is necessary to wait until device is rescanned by udev when proper mount point as assigned.

This doesn't work, the code that uses this is a generator, runs during earliest boot to generate mount units for additional file systems. It runs way before udev runs, and it needs to complete before we invoke units, i.e. before udev can even be started.

I see that while EXT4 devices are not available in /dev at the same moment, gpt-auto doesn't produce any error:

[ 1.079542] systemd-gpt-auto-generator[215]: Disabling root partition auto-detection, root= is defined.
[ 1.079572] systemd-gpt-auto-generator[215]: Failed to chase block device '/', ignoring: No such file or directory
....
[ 1.080414] systemd-gpt-auto-generator[215]: sda3: Root device /dev/sda.
....
[ 1.089351] systemd-gpt-auto-generator[215]: Waiting for device (parent + 3 partitions) to appear...