api: move `pkg/streamformatter`, `pkg/progress` to `daemon/internal` #51153

corhere · 2025-10-09T22:32:23Z

For api: remaining bits for jsonstream, jsonmessage, progress cleanup #50575

- What I did
- How I did it
The aforementioned packages are out of scope for the api module. They provide facilities to write streams of event messages to a sink, but those sinks are either a network socket or a terminal. There is no Engine API endpoint which consumes streams of events from a client, and rendering events to a terminal is out of scope of the api. Move the packages under daemon/internal and remove dead code.

Move the definition of the outer JSON event message to api/types and export it, dropping the deprecated ErrorMessage field. Arrange for the daemon to continue to emit that field for compatibility with docker-py v7.1.0.

Some of the code in these packages is used by github.com/docker/cli. The removals from the api module may be resolved simply by copying the code into that repository, where it belongs.

What to do with the "github.com/moby/moby/client/pkg/jsonmessage".JSONMessage struct remains to be sorted out. Its Progress field is a custom type that knows how to render a progress bar to a terminal; it will not be trivial to refactor it to reference the api jsonstream.Message type.

Proposal: client/pkg/jsonmessage: refactor in terms of api/types/jsonstream #51156

- How to verify it

- Human readable description for the release notes

- Go-SDK: `api/pkg/progress` and `api/pkg/streamformatter` have been removed.

- A picture of a cute animal (not mandatory but encouraged)

thaJeztah

Looks like we may have some work to do before we can move this internal

thaJeztah · 2025-10-10T12:53:55Z

daemon/internal/streamformatter/streamformatter.go

+// NewJSONProgressOutput returns a progress.Output that formats output
+// using JSON objects
+func NewJSONProgressOutput(out io.Writer, newLines bool) progress.Output {
+	return &progressOutput{sf: &jsonProgressFormatter{}, out: out, newLines: newLines}
+}


We need to look at this part; the CLI (and I think compose) currently depends on these for the classic builder and for various swarm functions 🫠

https://github.com/docker/cli/blob/6ddff81bee8a46745ce31a01fa7dd4ebaad4e4bd/cli/command/image/build.go#L301
https://github.com/docker/cli/blob/6ddff81bee8a46745ce31a01fa7dd4ebaad4e4bd/cli/command/image/build/context.go#L225-L230

https://github.com/docker/cli/blob/6ddff81bee8a46745ce31a01fa7dd4ebaad4e4bd/cli/command/swarm/progress/root_rotation.go#L31
https://github.com/docker/cli/blob/6ddff81bee8a46745ce31a01fa7dd4ebaad4e4bd/cli/command/service/progress/progress.go#L75
https://github.com/docker/cli/blob/6ddff81bee8a46745ce31a01fa7dd4ebaad4e4bd/cli/command/container/run_test.go#L245

Yes, I had noticed. I don't see what the holdup is in this repo, though. Just make an internal copy of the progress code in those repos when you bump the api dependency, and optionally refactor later to get rid of the silly in-process JSON round trips. You've got to do that for the raw progress formatter regardless.

austinvazquez · 2025-10-20T22:56:14Z

I opened docker/cli#6570 for Docker CLI to make internal copies of streamformatter and progress packages.

austinvazquez · 2025-10-21T13:35:59Z

I opened docker/compose#13304 for Docker compose to have copies there as well.

ndeloof · 2025-10-21T14:55:24Z

👎 Having 3 copies of this piece of code (internal + docker/cli + docker/compose) doesn't make the code cleaner

austinvazquez · 2025-10-23T19:25:37Z

Okay after discussion with the moby maintainers and feedback from common client PRs (docker/cli#6570 and docker/compose#13304).

This change now moves the streamformatter and progress packages up into the client alongside with the internal daemon copy. This is a compromise to have a common shared location outside the API module but the long term location may change.

Just for some background for folks, the first release of the client module will be v0.1.0. So this is not a long term commitment for the shape and size of these packages. Ideally long term probably something in CLI or a shared Go SDK.

Move the streamformatter package up into the client for a temporary shared location between common clients like CLI and compose. The streamformatter package is used by the daemon to write streams of status and progress messages to API clients. It is completely out of scope of the api module and not used outside the daemon. Remove the unused rawSteamFormatter, whose purpose is to render the progress as a TUI. Co-authored-by: Cory Snider <[email protected]> Signed-off-by: Austin Vazquez <[email protected]>

Move the progress package up into the client as a temporary shared location for common clients like CLI and compose. The progress package is used by the daemon to write progress updates to some sink, typically a streamformatter. This package is of little use to API clients as this package does not provide any facilities to consume the progress updates. Co-authored-by: Cory Snider <[email protected]> Signed-off-by: Austin Vazquez <[email protected]>

The schema of a JSON-stream message is very pertinent to the api module. Provide a canonical definition in the api module and refactor the daemon code to use it. Drop the long-deprecated ErrorMessage field from the API definition, but have the daemon continue to emit it for compatibility with docker-py v7.1.0. Co-authored-by: Cory Snider <[email protected]> Signed-off-by: Austin Vazquez <[email protected]>

austinvazquez · 2025-10-24T13:08:48Z

Rebased. PTAL.

austinvazquez · 2025-10-28T02:47:24Z

@thaJeztah , if you could give this another review; I have addressed the concern and no longer have the packages as internal based on feedback from consumers like compose. Probably we can continue this discussion further to determine the shape and location of the packages long term.

thaJeztah

LGTM

corhere added this to the 29.0.0 milestone Oct 9, 2025

corhere requested review from cpuguy83, dmcgowan, thaJeztah and tianon October 9, 2025 22:32

corhere added area/api API impact/changelog kind/refactor PR's that refactor, or clean-up code impact/go-sdk Noteworthy (compatibility changes) in the Go SDK labels Oct 9, 2025

github-actions bot added the module/api label Oct 9, 2025

corhere force-pushed the excise-json-streams-from-api branch 2 times, most recently from c268cfc to eab4869 Compare October 9, 2025 23:27

corhere mentioned this pull request Oct 9, 2025

client/pkg/jsonmessage: refactor in terms of api/types/jsonstream #51156

Merged

corhere force-pushed the excise-json-streams-from-api branch from eab4869 to 4f46ba6 Compare October 10, 2025 00:07

thaJeztah requested changes Oct 10, 2025

View reviewed changes

thompson-shaun added the release-blocker PRs we want to block a release on label Oct 16, 2025

thompson-shaun assigned corhere Oct 16, 2025

thompson-shaun added this to 🔦 Maintainer spotlight Oct 16, 2025

github-project-automation bot moved this to New in 🔦 Maintainer spotlight Oct 16, 2025

thompson-shaun assigned austinvazquez Oct 20, 2025

This comment was marked as off-topic.

Sign in to view

austinvazquez mentioned this pull request Oct 21, 2025

Refactor to remove dependencies on github.com/docker/docker/pkg/streamformatter and docker/compose#13304

Closed

austinvazquez force-pushed the excise-json-streams-from-api branch from 4f46ba6 to 04f0c73 Compare October 23, 2025 18:55

github-actions bot added area/images Image Distribution area/daemon Core Engine area/builder/classic-builder Build area/builder/buildkit Build labels Oct 23, 2025

github-actions bot added containerd-integration Issues and PRs related to containerd integration area/dependencies module/client labels Oct 23, 2025

corhere added 3 commits October 24, 2025 07:56

austinvazquez force-pushed the excise-json-streams-from-api branch from 04f0c73 to 687c3d7 Compare October 24, 2025 12:58

vvoland approved these changes Oct 24, 2025

View reviewed changes

thaJeztah approved these changes Oct 28, 2025

View reviewed changes

thaJeztah merged commit 9e7e01e into moby:master Oct 28, 2025
254 of 255 checks passed

vvoland added impact/api impact/go-sdk Noteworthy (compatibility changes) in the Go SDK and removed impact/go-sdk Noteworthy (compatibility changes) in the Go SDK impact/api labels Oct 29, 2025

vvoland moved this from New to Complete in 🔦 Maintainer spotlight Oct 30, 2025

This was referenced Oct 31, 2025

Copy streamformatter and progress packages from github.com/moby/moby docker/cli#6570

Closed

client: pkg/streamformatter: un-export unused utilities #51429

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

api: move `pkg/streamformatter`, `pkg/progress` to `daemon/internal` #51153

api: move `pkg/streamformatter`, `pkg/progress` to `daemon/internal` #51153

Uh oh!

corhere commented Oct 9, 2025 •

edited by austinvazquez

Loading

Uh oh!

thaJeztah left a comment

Uh oh!

thaJeztah Oct 10, 2025

Uh oh!

corhere Oct 10, 2025

Uh oh!

austinvazquez commented Oct 20, 2025

Uh oh!

This comment was marked as off-topic.

austinvazquez commented Oct 21, 2025

Uh oh!

ndeloof commented Oct 21, 2025 •

edited

Loading

Uh oh!

austinvazquez commented Oct 23, 2025

Uh oh!

austinvazquez commented Oct 24, 2025

Uh oh!

austinvazquez commented Oct 28, 2025

Uh oh!

thaJeztah left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

api: move pkg/streamformatter, pkg/progress to daemon/internal #51153

api: move pkg/streamformatter, pkg/progress to daemon/internal #51153

Uh oh!

Conversation

corhere commented Oct 9, 2025 • edited by austinvazquez Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

corhere Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

austinvazquez commented Oct 20, 2025

Uh oh!

This comment was marked as off-topic.

austinvazquez commented Oct 21, 2025

Uh oh!

ndeloof commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

austinvazquez commented Oct 23, 2025

Uh oh!

austinvazquez commented Oct 24, 2025

Uh oh!

austinvazquez commented Oct 28, 2025

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

api: move `pkg/streamformatter`, `pkg/progress` to `daemon/internal` #51153

api: move `pkg/streamformatter`, `pkg/progress` to `daemon/internal` #51153

corhere commented Oct 9, 2025 •

edited by austinvazquez

Loading

ndeloof commented Oct 21, 2025 •

edited

Loading