test: add integration tests for 'bootc switch --apply'

[p5]

This uses the existing test-20-local-upgrade.fmf with a lot of the unneeded tests stripped out.
It builds a bootc OCI image, switches into it with an --apply flag and validates both a reboot occurred, and that kargs from the new image are applied.

[gemini-code-assist]

Code Review

This pull request adds a new integration test for bootc switch --apply. The test creates a derived container image, switches to it, and verifies the system reboots into the new image with the correct kernel arguments.

The changes are logical and well-structured. I've found a critical issue with a filename typo in one of the tmf files that would prevent the test from running. I've also pointed out a few minor improvements in the nushell test script, such as removing unused variables and an unnecessary bash -c wrapper to improve code clarity and maintainability.

Once these points are addressed, the new test will be a valuable addition to the test suite.

[p5]

Hmm. Reading the docs, it doesn't seem like tmt supports rebooting using anything other than the tmt-reboot command. Or if it does, it won't increment the TMT_REBOOT_COUNT variable.

Will rebase to the fix, just in case it happens to work as I expect. Though I doubt it.

[cgwalters]

Hmm. Reading the docs, it doesn't seem like tmt supports rebooting using anything other than the tmt-reboot command. Or if it does, it won't increment the TMT_REBOOT_COUNT variable.

Yes see #1419

[henrywang]

Hi @p5,
I reopen this PR to add test for --apply. I added my research in #1419.

I can't push to your rsturla/bootc.git repository due to permission error. Could you please help to update the test-image-upgrade-reboot.nu. This should work. Thanks!

diff --git i/tmt/tests/booted/test-image-upgrade-reboot.nu w/tmt/tests/booted/test-image-upgrade-reboot.nu
index e4e8498..6c5ff5a 100644
--- i/tmt/tests/booted/test-image-upgrade-reboot.nu
+++ w/tmt/tests/booted/test-image-upgrade-reboot.nu
@@ -39,7 +39,7 @@ COPY usr/ /usr/
     podman build -t localhost/bootc-derived .
 
     # Now, switch into the new image
-    bootc switch --apply --transport containers-storage localhost/bootc-derived
+    tmt-reboot -c "bootc switch --apply --transport containers-storage localhost/bootc-derived"
 
     # We cannot perform any other checks here since the system will be automatically rebooted
 }

[p5]

Thanks for the pointer @henrywang !

Your patch has been applied and squashed into the existing commit. The integration tests seemingly are passing, so think this is ready for some reviews!

This test should:

1. boot into a VM
2. create a new image with kargs changes
3. switch into the new image with the --apply flag
4. verify kargs changes persisted

---

Edit: I'm not actually sure that this is working as we intended.
This branch is prior to the fixes in #1417, so I would expect the builds to be failing until we rebased. Looking into it.

[henrywang]

Hi @p5, can we land this PR? Thanks.

[p5]

Hi @p5, can we land this PR? Thanks.

I'm honestly not sure that we would want to...

My view is this current PR gives a false sense of security that the --apply flag is working since the current branch should be failing the tests. This branch is prior to the fix for #1416 , so unless the tests are for some reason automatically rebasing the changes onto the latest main branch, the test is mostly useless.

The purpose is to ensure --apply works, but the tests pass when --apply should be failing.

[henrywang]

I think the --apply can reboot system if we run it in command line, like sudo bootc switch --transport containers-storage --apply localhost/bootc-apply:update. It only has issue on bootc-fetch-apply-updates.service service.

[p5]

It only has issue on bootc-fetch-apply-updates.service service.

Ah, right. I didn't realise that was the case. Will mark this ready to review.
If it needs rebasing, I will be able to do so in about an hour, when I get back to a non-work computer.

[cgwalters]

I think the --apply can reboot system if we run it in command line, like sudo bootc switch --transport containers-storage --apply localhost/bootc-apply:update. It only has issue on bootc-fetch-apply-updates.service service.

Wait I don't think that's true. How could that be? It's the same code being invoked...unless it somehow theoretically related to being invoked inside or outside of a systemd unit but I don't think that's the case.

[cgwalters]

My view is this current PR gives a false sense of security that the --apply flag is working since the current branch should be failing the tests.

One possibility is that the packit testing flow is defaulting to rebasing the PR on main?

I mean, we can find out for sure if we're having a "false test pass" by pushing a commit which explicitly breaks the functionality into this PR.

[cgwalters]

For this test, I think using kernel arguments is a bit of a distraction. We already have coverage of kernel arguments in the original test this was copied from. It seems simpler to just create some new content e.g. touch /usr/share/testing-bootc-upgrade-apply or something and verify that file exists.

(This would also mean this test would automatically get optimized a bit with soft reboots in the future)

[p5]

Once the tests complete, will add the following:

commit 913b199ab92c78c5d8f27dec36f41a6eb6bc8d06 (HEAD -> systemd-run-reboot-test)
Author: Robert Sturla <robertsturla@outlook.com>
Date:   Wed Jul 30 20:41:03 2025 +0100

    chore: verify file is present in newer deployment
    
    Signed-off-by: Robert Sturla <robertsturla@outlook.com>

diff --git a/tmt/tests/booted/test-image-upgrade-reboot.nu b/tmt/tests/booted/test-image-upgrade-reboot.nu
index 5d7b9bc3..31dda7a5 100644
--- a/tmt/tests/booted/test-image-upgrade-reboot.nu
+++ b/tmt/tests/booted/test-image-upgrade-reboot.nu
@@ -7,8 +7,6 @@
 use std assert
 use tap.nu
 
-const kargsv0 = ["testarg=foo", "othertestkarg", "thirdkarg=bar"]
-
 # This code runs on *each* boot.
 # Here we just capture information.
 bootc status
@@ -28,11 +26,9 @@ def initial_build [] {
 
     bootc image copy-to-storage
 
-    mkdir usr/lib/bootc/kargs.d
-    { kargs: $kargsv0 } | to toml | save usr/lib/bootc/kargs.d/05-testkargs.toml
-    # A simple derived container that adds a file, but also injects some kargs
+    # A simple derived container that adds a file
     "FROM localhost/bootc
-COPY usr/ /usr/
+RUN touch /usr/share/testing-bootc-upgrade-apply
 " | save Dockerfile
     # Build it
     podman build -t localhost/bootc-derived .
@@ -49,13 +45,8 @@ def second_boot [] {
     assert equal $booted.image.transport containers-storage
     assert equal $booted.image.image localhost/bootc-derived
 
-    # Verify we have updated kargs
-    let cmdline = parse_cmdline
-    print $"cmdline=($cmdline)"
-    for x in $kargsv0 {
-        print $"Verifying karg ($x)"
-        assert ($x in $cmdline)
-    }
+    # Verify the new file exists
+    "/usr/share/testing-bootc-upgrade-apply" | path exists
 
     tap ok

Uses https://www.nushell.sh/commands/docs/path_exists.html to check if the file exists

[cgwalters]

Yep LGTM

[cgwalters]

There's another possibility, which is that tmt-reboot is eventually rebooting on its own (or something like that).

I tried to trace through the code, it's "complicated". Clicking on tmt-reboot-core shows that internally it writes the command to be executed as JSON into a file found by an environment variable TMT_TEST_PIDFILE...searching for that string I ended up at https://github.com/teemtee/tmt/blob/c7a856295c81ecd016dd580d2dce094862f1e4b0/tmt/steps/execute/internal.py#L57 and...I dunno...

[cgwalters]

(Those folks who know me know I'm really a strong "dynamic languages are only suitable for toy projects" person, and yeah, jinja2 to template shell script from python is like three whole levels of dynamism...)

[cgwalters]

One possibility is that the packit testing flow is defaulting to rebasing the PR on main?

Yep I was right, check out the RPM build logs:

2025-07-30 19:34:29.973 remote.py         DEBUG  b'head info: ["689340333d392f3476b0503b48689a338178037f\\t\\tbranch \'main\' of https://github.com/bootc-dev/bootc\\n"]'
2025-07-30 19:34:29.974 cmd.py            DEBUG  Popen(['git', 'checkout', 'main'], cwd=/tmp/tmpi2rd5f56, stdin=None, shell=False, universal_newlines=False)
2025-07-30 19:34:29.989 cmd.py            INFO   git checkout main -> 0; stdout: 'Your branch is up to date with 'origin/main'.'; stderr: 'Switched to branch 'main''
2025-07-30 19:34:29.989 local_project.py  INFO   Merging (main) with commit:
(68934033)	Merge pull request #1457 from cgwalters/doc-non-root
2025-07-30 19:34:29.989 cmd.py            DEBUG  Popen(['git', 'merge', 'pr-changes/1420'], cwd=/tmp/tmpi2rd5f56, stdin=None, shell=False, universal_newlines=False)
2025-07-30 19:34:30.005 cmd.py            INFO   git merge pr-changes/1420 -> 1; stdout: 'Auto-merging crates/lib/src/reboot.rs
CONFLICT (content): Merge conflict in crates/lib/src/reboot.rs
Automatic merge failed; fix conflicts and then commit the result.'
2025-07-30 19:34:30.005 local_project.py  WARNING Merge failed with: Cmd('git') failed due to: exit code(1)
  cmdline: git merge pr-changes/1420
  stdout: 'Auto-merging crates/lib/src/reboot.rs
CONFLICT (content): Merge conflict in crates/lib/src/reboot.rs
Automatic merge failed; fix conflicts and then commit the result.'
Usage: packit prepare-sources [OPTIONS] [PATH_OR_URL]
Try 'packit prepare-sources -h' for help.

Error: Invalid value for '[PATH_OR_URL]': Cmd('git') failed due to: exit code(1)
  cmdline: git merge pr-changes/1420
  stdout: 'Auto-merging crates/lib/src/reboot.rs
CONFLICT (content): Merge conflict in crates/lib/src/reboot.rs

it now directly fails because of a conflict.

[p5]

One possibility is that the packit testing flow is defaulting to rebasing the PR on main?

Yep I was right, check out the RPM build logs

Ah, that answers it then! That's why the tests never failed where I expected them to! Though I can't say I'm too keen on the hidden behaviour 🤷

I've squashed the commits, rebased on latest upstream/main, removed the debug commit and implemented the changes suggested above. The tests should pass and should be ready for a review!

Thanks for the help both!

[cgwalters]

Thank you so much for doing this and especially your careful attention to consideration as to the test unexpectedly passing!

[p5]

Is it normal for these tests to take 2 hours? Or has my change stalled them?

[henrywang]

Is it normal for these tests to take 2 hours? Or has my change stalled them?

Testing Farm is outage according to https://status.testing-farm.io/. :(

[henrywang]

Hi @cgwalters, I tried on bootc 1.4.0 and run sudo bootc switch --transport containers-storage --apply localhost/bootc-apply:update. System can be updated and rebooted. So --apply can reboot system at least, right? Thanks.