v245-stable: ordering cycle reported, prevents non-root login or failed automount

[andyrtr]

Using Arch Linux we run into a new bug in stable release 245.6 that introduces a "Ordering cycle reported" leading to random services not properly loaded and login or automount failing.

Bisected to this commit and reverting it seems to fix it:
e1c091b

Downstream report: https://bugs.archlinux.org/task/66908

[cdown]

Seems I don't have rights in systemd-stable repo to transfer issues, @poettering, @keszybz, any chance you can hook me up? :-)

---

Since this is a straight backport of systemd/systemd@b3d7aef, this is something we should ideally centralise discussion of on the main repo.

It's not immediately obvious to me why that change would cause this problem, but probably it's exposing some other overall dependency issue. cc @fbuihuu, who has more context.

[fbuihuu]

From a quick glance, it seems that all problematic cases make the automount unit use x-systemd.requires=network-online.target.

This doesn't look correct since automount units are supposed to be ordered before local-fs.target.

@cdown do you know why this requirement is needed ?

[fbuihuu]

Something strange is that the ordering cycle should have already existed if the automount unit was generated by fstab-generator, which seems to be the case from one report.

[cdown]

From a quick glance, it seems that all problematic cases make the automount unit use x-systemd.requires=network-online.target.

Right, but shouldn't this then also have happened with fstab-generator before?

This doesn't look correct since automount units are supposed to be ordered before local-fs.target.

@cdown do you know why this requirement is needed ?

I assume it's a well-intentioned attempt to make network shares contingent on the network. Certainly it feels like the way we expose this issue to users (hey, you can't log in!) is probably the worst manifestation I could conjure, though. It would be nice if we could surface this somewhere earlier.

[fbuihuu]

Right, but shouldn't this then also have happened with fstab-generator before?

That's what is puzzling me currently (see my previous comment).

I did give it a try on my archlinux VM and I still could see the ordering cycle with systemd-243.162-2-x86_64.pkg.tar.xz (that was the latest version available locally which doesn't have the suspected commits):

Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found ordering cycle on systemd-resolved.service>
Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found dependency on systemd-tmpfiles-setup.servi>
Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found dependency on local-fs.target/start
Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found dependency on mnt.automount/start
Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found dependency on network-online.target/start
Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found dependency on network.target/start

But for some reasons no jobs was deleted in order to break this cycle here, which was the case when I tested systemd-245.6-4-x86_64.pkg.tar.zst...

[fbuihuu]

I assume it's a well-intentioned attempt to make network shares contingent on the network.

But that seems wrong - automount units are supposed to be ordered before local-fs.target so also ordering them after network-online.target will just create ordering cycles and a (random) job to be killed during the boot process.

I think only mount units should depend on the network but ordering against network-online.target is probably the wrong target to choose. And fstab-generator is supposed to add the network deps automatically anyway (you can also force it to do so with _netdev option).

[cdown]

Sure, I don't think it actually does what they expect, but it's well-intentioned. I can see why people would think they should do that.

[fbuihuu]

Hmm... wait.

If an entry in fstab uses the following options: x-systemd.automount,x-systemd.requires=network-online.target.

Is the dependency on network-online.target supposed to be applied to the mount unit or to the automount one ?

Currently it applies to the automount unit but I'm not sure if this is really intended...

@poettering @keszybz ?