Add tests for the NVIDIA GPU tests

[sayanchowdhury]

Add tests for the NVIDIA GPU tests

Testing done

• Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
• Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

[pothos]

With

Suggested change

	m := c.Machines()[0]
	if kola.AzureOptions.Size != "Standard_NC6s_v3" {
	c.Skip("skipping due to wrong instance size")
	}
	m := c.Machines()[0]

I think we could do this in scripts:

diff --git a/ci-automation/vendor-testing/azure.sh b/ci-automation/vendor-testing/azure.sh
index 17081b3598..c57210d651 100755
--- a/ci-automation/vendor-testing/azure.sh
+++ b/ci-automation/vendor-testing/azure.sh
@@ -74,7 +74,7 @@ query_kola_tests() {
     kola list --platform=azure --filter "${@}"
 }
 
-other_instance_types=()
+other_instance_types=('Standard_NC6s_v3')
 if [[ "${CIA_ARCH}" = 'amd64' ]]; then
     other_instance_types+=('V1')
 fi
@@ -85,6 +85,6 @@ run_kola_tests_on_instances \
     "${CIA_FIRST_RUN}" \
     "${other_instance_types[@]}" \
     '--' \
-    'cl.internet' \
+    'cl.internet' 'cl.misc.nvidia' \
     '--' \
     "${@}"

[tormath1]

If we want to keep this check, we could even move it to the SkipFunc (as kola.AzureOptions.Size should be accessible)

[tormath1]

Any chance to track the avc message? (for issue tracking and/or documentation purpose?)

[sayanchowdhury]

Good idea! I will need to search the logs, if it's not there then I can trigger a local/Jenkins build to track the avc messages for tracking on a issue.

[tormath1]

You should see it in the journal logs (before disabling the SELinux flags)

[sayanchowdhury]

Yes, I meant searching the journal logs if they are not deleted on bincache.

[tormath1]

Just FYI, you could do this way:

Suggested change

	out, err := c.SSH(m, "systemctl is-active nvidia.service")
	if !bytes.Contains(out, []byte("inactive")) {
	return fmt.Errorf("nvidia.service: %q: %v", out, err)
	}
	_, err := c.SSH(m, "systemctl --quiet is-active nvidia.service")
	return err

we just want to assert if the unit is active or not, if the unit is active with --quiet it will simply return 0.

[pothos]

There is also _ = c.MustSSH(…) which fails the test.

[tormath1]

That is correct but in this case we don't want to fail the test as we are in a retry loop no?