kola: Extend test to cover recreation of files in /etc

[pothos]

For files like /etc/resolv.conf we want it to be recreated because the tmpfiles rule would do this. For a systemd bug workaround we filtered out some rules but even if we add them back, we still want to ensure that not only the file will always be recreated but that it also is not frozen as upcopy because we prefer the overlay lowerdir as source for updates.
Extend the /etc overlayfs test to ensure recreation of the file and to check against upcopies.

How to use

Will pass for fixed releases.

Testing done

sudo ./kola run -d -k --keys --key ~/.ssh/id_rsa.pub --platform=qemu  --qemu-image /var/tmp/flatcar_production_image.bin.whiteouts cl.overlay.cleanup

passed (and fails for current releases)

• Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
• Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

[tormath1]

While merging the Mantle PRs, I noticed that it fails on Stable:

not ok - cl.overlay.cleanup
  ---
  Error: "cluster.go:145: \"if test ! -e /etc/resolv.conf ; then echo \\\"Files with tmpfile rule not recreated: $_\\\" ; exit 1; fi && if ! sudo unshare -m bash -c 'umount /etc && test ! -e /etc/resolv.conf'; then echo \\\"File with tmpfile rule exists as upcopy: $_\\\"; exit 1; fi\" failed: output Files with tmpfile rule not recreated: /etc/resolv.conf, status Process exited with status 1"
  ...

EDIT: Checked the instance, it has the r16 of bootengine.

[pothos]

Thanks, fixed in flatcar/bootengine@010fa85