fix: wait for logger to finish before reading logs of stopped containers

[haytok]

TestLogsFollowNoExtraneousLineFeed sometimes fails on CI.

• https://github.com/containerd/nerdctl/actions/runs/24197307638/job/71240723501?pr=4835
• https://github.com/containerd/nerdctl/actions/runs/24197307638/job/71215370984?pr=4835

        +------------------------------------------------------------------------------------------------------------+
        | ➡️      | ⚙️ /usr/local/bin/nerdctl.gomodjail run --name testlogsfollownoextraneouslinefeed-f5d64e9b ghcr. |
        |         | io/stargz-containers/alpine:3.13-org sh -c printf 'Hello without newline'                        |
        +------------------------------------------------------------------------------------------------------------+
        |         | 🟢 Hello without newline
...
    container_logs_test.go:562: 	🔗

        <<<<<<<<<<<<<<<<<<<<
        	🖊️ Inspecting output (equals)
        	👀 testing:		``
        	❌ FAILED!		= `Hello without newline`
        >>>>>>>>>>>>>>>>>>>>

My Investigation revealed that the flakiness of
TestLogsFollowNoExtraneousLineFeed is caused by nerdctl logs -f not waiting for the logger to finish writing when the container has already stopped.

Therefore, this commit fixes nerdctl logs -f to wait for the logger to finish writing when the container has already stopped.

This commit should fix the flakiness of TestLogsFollowNoExtraneousLineFeed.

After applying this fix, running the test 1000 times showed no flakiness.

$ sudo go test -count=1000 -run '^TestLogsFollowNoExtraneousLineFeed$'
test target: "nerdctl"
PASS
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/container	419.653s

[haytok]

My Investigation

Timing-dependent bugs often pass in a single test run but can be detected by increasing the number of iterations.

sudo go test -count=100 -run '^TestLogsFollowNoExtraneousLineFeed$'

Running this command confirmed that TestLogsFollowNoExtraneousLineFeed fails intermittently for the same reason CI failed.

So I then added the following debug code and reproduced the failure:

	testCase.Expected = func(data test.Data, helpers test.Helpers) *test.Expected {
		return &test.Expected{
			ExitCode: 0,
			Output: func(stdout string, info tig.T) {
				info.Helper()
				if stdout != "Hello without newline" {
					logPath := helpers.Capture("inspect", "--format", "{{.LogPath}}", data.Identifier())
					content, err := os.ReadFile(strings.TrimSpace(logPath))
					if err != nil {
						info.Log(fmt.Sprintf("=== failed to read JSON log: %v ===", err))
					} else {
						info.Log(fmt.Sprintf("=== JSON log content (stdout mismatch) ===\n%s", content))
					}
				}
				expect.Equals("Hello without newline")(stdout, info)
			},
		}
	}

The result showed that the JSON log file contained the expected data, but the output of nerdctl logs -f did not reflect it.

To summarize the analysis:

• The JSON log file already contains the expected output at the time the debug message is logged in the comparator.
• When helpers.Command("logs", "-f", data.Identifier()) is executed, helpers.Ensure("run", "--name", data.Identifier(), "sh", "-c", "printf 'Hello without newline'") has already completed writing to stdout, but the logger process has not finished writing to the JSON log file.

Based on this, analyzing the logging code confirms that when a container has already stopped, nerdctl logs -f does not wait for the logger to finish processing the remaining logs:

nerdctl/pkg/cmd/container/logs.go

Lines 86 to 104 in 4e5cc62

	if status.Status != containerd.Running {
	follow = false
	} else {
	waitCh, err := task.Wait(ctx)
	if err != nil {
	return fmt.Errorf("failed to get wait channel for task %#v: %w", task, err)
	}
	// Setup goroutine to send stop event if container task finishes:
	go func() {
	<-waitCh
	// Wait for logger to process remaining logs after container exit
	if err = logging.WaitForLogger(dataStore, l[labels.Namespace], found.Container.ID()); err != nil {
	log.G(ctx).WithError(err).Error("failed to wait for logger shutdown")
	}
	log.G(ctx).Debugf("container task has finished, sending kill signal to log viewer")
	stopChannel <- os.Interrupt
	}()
	}

Therefore, this commit should fix the flakiness of TestLogsFollowNoExtraneousLineFeed.

[AkihiroSuda]

Thanks