test: Fix factory reset test failure on CS10

[henrywang]

Sometimes systemd daemons are still running old binaries and response "Access denied" when send reboot request.

This PR forces a full sync before reboot and allow more delay for bootc to settle.

[gemini-code-assist]

Code Review

This pull request attempts to fix a failing factory reset test by introducing a sleep and a forceful reboot. While this might make the test pass, it introduces potential flakiness with the fixed sleep duration and uses a dangerous reboot command that can cause data loss and mask underlying system issues. My review provides a suggestion to use sync for better reliability and raises concerns about the forceful reboot, recommending an investigation into the root cause of the shutdown problem.

[gemini-code-assist]

Using a fixed sleep is not a reliable way to ensure file operations are complete and can lead to flaky tests. It's better to use sync to flush file system buffers to disk. This is more deterministic and avoids unnecessary delays.

Additionally, using systemctl --force --force reboot is very aggressive and can mask underlying issues that prevent a graceful reboot, potentially leading to data loss. While this might be a temporary measure for debugging, it would be best to investigate the root cause of the reboot failure for a more robust solution.

I suggest replacing sleep 10 with sync.

    sync
    tmt-reboot -c "systemctl --force --force reboot"

[henrywang]

Before reboot, system looks good:

07:38:09                 out: apiVersion: org.containers.bootc/v1
07:38:09                 out: kind: BootcHost
07:38:09                 out: metadata:
07:38:09                 out:   name: host
07:38:09                 out: spec:
07:38:09                 out:   image:
07:38:09                 out:     image: localhost/bootc-integration:latest
07:38:09                 out:     transport: registry
07:38:09                 out:   bootOrder: default
07:38:09                 out: status:
07:38:09                 out:   staged: null
07:38:09                 out:   booted:
07:38:09                 out:     image:
07:38:09                 out:       image:
07:38:09                 out:         image: localhost/bootc-integration:latest
07:38:09                 out:         transport: registry
07:38:09                 out:       version: '10'
07:38:09                 out:       timestamp: 2025-11-13T07:35:04.529519474Z
07:38:09                 out:       imageDigest: sha256:51ddfe8084b40413037f5ca8f6bb80796ab52a688c346d151a85080e115ec23a
07:38:09                 out:       architecture: arm64
07:38:09                 out:     cachedUpdate: null
07:38:09                 out:     incompatible: false
07:38:09                 out:     pinned: false
07:38:09                 out:     softRebootCapable: true
07:38:09                 out:     store: ostreeContainer
07:38:09                 out:     ostree:
07:38:09                 out:       stateroot: default
07:38:09                 out:       checksum: e2a46408f929c157d15440380cf5792de9dfcf5a86ca55000223241c7ef2f57a
07:38:09                 out:       deploySerial: 0
07:38:09                 out:     composefs: null
07:38:09                 out:   rollback: null
07:38:09                 out:   otherDeployments:
07:38:09                 out:   - image:
07:38:09                 out:       image:
07:38:09                 out:         image: localhost/bootc-integration:latest
07:38:09                 out:         transport: registry
07:38:09                 out:       version: '10'
07:38:09                 out:       timestamp: 2025-11-13T07:35:04.529519474Z
07:38:09                 out:       imageDigest: sha256:51ddfe8084b40413037f5ca8f6bb80796ab52a688c346d151a85080e115ec23a
07:38:09                 out:       architecture: arm64
07:38:09                 out:     cachedUpdate: null
07:38:09                 out:     incompatible: false
07:38:09                 out:     pinned: false
07:38:09                 out:     softRebootCapable: false
07:38:09                 out:     store: ostreeContainer
07:38:09                 out:     ostree:
07:38:09                 out:       stateroot: state-2025-0
07:38:09                 out:       checksum: e2a46408f929c157d15440380cf5792de9dfcf5a86ca55000223241c7ef2f57a
07:38:09                 out:       deploySerial: 0
07:38:09                 out:     composefs: null
07:38:09                 out:   rollbackQueued: false
07:38:09                 out:   type: bootcHost

After reboot

07:39:02                 out: apiVersion: org.containers.bootc/v1
07:39:02                 out: kind: BootcHost
07:39:02                 out: metadata:
07:39:02                 out:   name: host
07:39:02                 out: spec:
07:39:02                 out:   image:
07:39:02                 out:     image: localhost/bootc-integration:latest
07:39:02                 out:     transport: registry
07:39:02                 out:   bootOrder: default
07:39:02                 out: status:
07:39:02                 out:   staged: null
07:39:02                 out:   booted:
07:39:02                 out:     image:
07:39:02                 out:       image:
07:39:02                 out:         image: localhost/bootc-integration:latest
07:39:02                 out:         transport: registry
07:39:02                 out:       version: '10'
07:39:02                 out:       timestamp: 2025-11-13T07:35:04.529519474Z
07:39:02                 out:       imageDigest: sha256:51ddfe8084b40413037f5ca8f6bb80796ab52a688c346d151a85080e115ec23a
07:39:02                 out:       architecture: arm64
07:39:02                 out:     cachedUpdate: null
07:39:02                 out:     incompatible: false
07:39:02                 out:     pinned: false
07:39:02                 out:     softRebootCapable: true
07:39:02                 out:     store: ostreeContainer
07:39:02                 out:     ostree:
07:39:02                 out:       stateroot: default
07:39:02                 out:       checksum: e2a46408f929c157d15440380cf5792de9dfcf5a86ca55000223241c7ef2f57a
07:39:02                 out:       deploySerial: 0
07:39:02                 out:     composefs: null
07:39:02                 out:   rollback: null
07:39:02                 out:   rollbackQueued: false
07:39:02                 out:   type: bootcHost
07:39:02                 out: Error:   x Assertion failed.
07:39:02                 out:     ,-[/var/ARTIFACTS/work-test-28-factory-resetaj3rbki2/tmt/plans/integration/test-28-factory-reset/discover/default-0/tests/tmt/tests/booted/test-factory-reset.nu:54:22]
07:39:02                 out:  53 |     let status = bootc status --json | from json
07:39:02                 out:  54 |     assert not equal $status.status.booted.ostree.stateroot "default"
07:39:02                 out:     :                      ^^^^^^^^^^^^^^^^^^^^^^^^|^^^^^^^^^^^^^^^^^^^^^^^
07:39:02                 out:     :                                              `-- These are both '"default"'.
07:39:02                 out:  55 | 
07:39:02                 out:     `----

[henrywang]

The testing-farm:centos-stream-10-x86_64 failure in recent PRs can be fixed by this PR. I'll make a rebase on those PRs after this PR landed.

[cgwalters]

But this looks like purely a systemd bug that we should report right?

I don't understand why systemd would have old binaries...we shouldn't be touching the running OS?

[cgwalters]

Thanks for chasing this!