CI observability enhancements.#1193

jglogan · 2026-02-11T04:23:23Z

Type of Change

Bug fix
New feature
Breaking change
Documentation update

Motivation and Context

Adds a a --log-root option to swift system start, propagating the value as CONTAINER_LOG_ROOT to services for logging to files instead of the OS log facility. This is not a "production" capability as it neither merges nor rotates logs.
Currently we don't collect logs on CI builds, and we don't have permission to run the log command there. The PR adds --log-root to the CI test phase, archives the results, and uploads the archive as an artifact.
Use FilePath from swift-system for the log root. Foundation URL is a bit of a footgun for filesystem paths, so unless we identify a showstopper, we should incrementally transition to this type everywhere except where we really need network URLs.
Output the hostname of the CI runner at the start of the test phase so we can identify runner-specific issues where they exist.
Fix formatting for log messages with multiple metadata items, and fix unstructured messages on instances that weren't found using grep -r 'log\.' Sources.
Adds command reference documentation for --log-root.

Testing

Tested locally
Added/updated tests
Added/updated docs

Sources/Helpers/APIServer/APIServer+Start.swift

Ronitsabhaya75 · 2026-02-12T03:34:52Z

@jglogan can you have look at #1167 I have added retry logic for api server so that we can atleast 5 tries during install kernel.

I saw CI has same failure. so i added retry logic instead of 1 time trying install kernel added 5 times.

lemme know if this fix can help

jglogan · 2026-02-12T03:59:05Z

@Ronitsabhaya75 Retrying around undiagnosed problems is not the answer, and in this case you can retry a million times and it still won't work.

Ronitsabhaya75 · 2026-02-12T04:01:30Z

@Ronitsabhaya75 Retrying around real problems is not the answer, and in this case you can retry a million times and it still won't work.

let me see what can potential issue for installing-kernel can be. thats where CI has been failing all time.

jglogan · 2026-02-12T04:02:20Z

I am committing a likely fix. The problem has nothing to do with installing the kernel.

Ronitsabhaya75 · 2026-02-12T04:04:06Z

Verifying apiserver is running...
make: *** [install-kernel] Error 1
tests failed with status: 2
Removing data directory /Users/runner/actions-runner/_work/_temp/tmp.Moka3ycaZe
Error: Process completed with exit code 2.

yeah time out issue (wording mistake)

Ronitsabhaya75 · 2026-02-12T04:49:04Z

@jglogan do you think this is good idea adding endpoint that captures the system state like running containers,
network status, service health at any point.

then calling it automatically before operation when CI is failing ( like install kernel stage).

this can give us snapshot of before and after for comparision.

Ronitsabhaya75 · 2026-02-12T16:42:29Z

Hey @jglogan, I was thinking about this last night after our conversation. You're right that retries won't fix this but I think I might have an idea about what's actually going wrong.

I'm wondering if the install-kernel timeout is actually a race condition. Like, we're trying to install the kernel before the API server is fully ready to accept XPC connections. The service might be "started" but not fully initialized yet.

Instead of retrying or just increasing the timeout, what if we add a proper readiness check? And actually, your new JSON status output makes this really easy to implement:

install-kernel: start-system
	@echo "Waiting for system to be ready..."
	@for i in 1 2 3 4 5; do \
		if $(CONTAINER) system status --output json | jq -e '.apiServerRunning == true' > /dev/null 2>&1; then \
			echo "API server ready"; \
			break; \
		fi; \
		echo "Waiting for API server (attempt $$i/5)..."; \
		sleep 2; \
	done
	$(CONTAINER) kernel install

sample output

$ make install-kernel
Waiting for system to be ready...
Waiting for API server (attempt 1/5)...
Waiting for API server (attempt 2/5)...
Waiting for API server (attempt 3/5)...
Waiting for API server (attempt 4/5)...
Waiting for API server (attempt 5/5)...
Warning: API server did not become ready, attempting install anyway...
Installing kernel extension...
Error: Failed to connect to API server (timeout)

jglogan · 2026-02-12T23:42:32Z

I'm wondering if the install-kernel timeout is actually a race condition. Like, we're trying to install the kernel before the API server is fully ready to accept XPC connections. The service might be "started" but not fully initialized yet.

Perhaps. But the API server accepted an XPC connection at ClientHealthCheck.ping of SystemStart before trying to make the XPC connection through the installDefaultKernel call further down?

JaewonHur · 2026-02-13T02:01:32Z

Sources/Helpers/RuntimeLinux/RuntimeLinuxHelper+Start.swift

-            let log = RuntimeLinuxHelper.setupLogger(debug: debug, metadata: ["uuid": "\(uuid)"])
+            let commandName = RuntimeLinuxHelper._commandName
+            let logPath = logRoot.map { $0.appending("\(commandName)-\(uuid).log") }
+            let log = ServiceLogger.bootstrap(category: "NetworkVmnetHelper", metadata: ["uuid": "\(uuid)"], debug: debug, logPath: logPath)


Should be RuntimeLinuxHelper.

ajemory · 2026-02-16T22:17:38Z

Sources/ContainerLog/ServiceLogger.swift

+    ) -> Logger {
+        LoggingSystem.bootstrap { label in
+            if let logPath {
+                if let handler = try? FileLogHandler(label: label, path: logPath) {


Will the user be aware of a fallback to OS logs here in case of failure?

Logging isn't set up and this is in a service context, so I don't see a good way to communicate it.

We could try to make note of the condition here and then log a warning above L46, but that would still go into the OS log; it wouldn't be immediately visible to the user.

Makes sense. I'm not seeing a good way to go about it either. Not blocking for this PR, but maybe something to think about

ajemory

Looks good. Always happy to see well-thought-out logging enhancements 😄

jglogan force-pushed the ci-observability branch from 3f7a39b to 37447a2 Compare February 11, 2026 04:44

jglogan changed the base branch from ci-observability to main February 11, 2026 04:47

jglogan force-pushed the ci-observability branch from 37447a2 to c98a999 Compare February 11, 2026 04:47

jglogan commented Feb 11, 2026

View reviewed changes

Sources/Helpers/APIServer/APIServer+Start.swift Outdated Show resolved Hide resolved

ajemory approved these changes Feb 11, 2026

View reviewed changes

jglogan force-pushed the ci-observability branch 3 times, most recently from 16eb03e to 7534596 Compare February 12, 2026 04:32

jglogan force-pushed the ci-observability branch 3 times, most recently from cf477f4 to e661893 Compare February 12, 2026 22:26

jglogan force-pushed the ci-observability branch 3 times, most recently from c7d5c6b to c75e290 Compare February 13, 2026 00:50

JaewonHur reviewed Feb 13, 2026

View reviewed changes

JaewonHur self-requested a review February 13, 2026 19:38

JaewonHur approved these changes Feb 13, 2026

View reviewed changes

jglogan force-pushed the ci-observability branch 3 times, most recently from f538082 to 4d3481e Compare February 16, 2026 18:42

ajemory reviewed Feb 16, 2026

View reviewed changes

ajemory approved these changes Feb 16, 2026

View reviewed changes

jglogan force-pushed the ci-observability branch 4 times, most recently from d7e3a2b to b86cda2 Compare February 18, 2026 02:36

Collect logs.

874c89a

jglogan force-pushed the ci-observability branch from eb40b71 to 7efd570 Compare February 19, 2026 21:11

Reformat metadata, add category to handler

daf43dc

jglogan force-pushed the ci-observability branch from 7efd570 to daf43dc Compare February 20, 2026 00:55

ajemory approved these changes Feb 20, 2026

View reviewed changes

jglogan merged commit d299471 into apple:main Feb 20, 2026
3 checks passed

jglogan deleted the ci-observability branch February 20, 2026 01:47

Conversation

jglogan commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of Change

Motivation and Context

Testing

Uh oh!

Uh oh!

Ronitsabhaya75 commented Feb 12, 2026

Uh oh!

jglogan commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ronitsabhaya75 commented Feb 12, 2026

Uh oh!

jglogan commented Feb 12, 2026

Uh oh!

Ronitsabhaya75 commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ronitsabhaya75 commented Feb 12, 2026

Uh oh!

Ronitsabhaya75 commented Feb 12, 2026

Uh oh!

jglogan commented Feb 12, 2026

Uh oh!

JaewonHur Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

ajemory Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

jglogan Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

ajemory Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

ajemory left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jglogan commented Feb 11, 2026 •

edited

Loading

jglogan commented Feb 12, 2026 •

edited

Loading

Ronitsabhaya75 commented Feb 12, 2026 •

edited

Loading