[Bug]: container build fails to resolve local images in COPY --from

[ivankra]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

cat >Dockerfile.base <<'EOF'
FROM alpine:latest
RUN echo hello >/hello.txt
EOF

cat >Dockerfile.test <<'EOF'
FROM alpine:latest
COPY --from=bug-base /hello.txt /copied.txt
RUN cat /copied.txt
EOF

container build -t bug-base -f Dockerfile.base .
container build -t bug-test -f Dockerfile.test .

Current behavior

It tries to pull a non-existent image from docker.io rather then using a local image.

Expected behavior

COPY --from=<tag> should use a local image with that tag first if available, just like FROM <tag> does.

Environment

- OS: macOS 26.2
- Xcode: i don't use it
- Container: container CLI version 0.9.0 (build: release, commit: 3e49dce)

Relevant log output

+ container build -t bug-base -f Dockerfile.base .
[+] Building 4.0s (5/5) FINISHED
 => [resolver] fetching image...docker.io/library/alpine:latest                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                          0.0s
 => oci-layout://docker.io/library/alpine:latest@sha256:25109184c71bdad752c8312a8623239686a9a2071e8825f20acb8f2198c3f659                                 0.0s
 => => resolve docker.io/library/alpine:latest@sha256:25109184c71bdad752c8312a8623239686a9a2071e8825f20acb8f2198c3f659                                   0.0s
 => CACHED [linux/arm64 1/2] RUN echo hello >/hello.txt                                                                                                  0.0s
 => exporting to oci image format                                                                                                                        0.0s
 => => exporting layers                                                                                                                                  0.0s
 => => exporting manifest sha256:fad670ed74832a71f87d6cdf26a2083bce90651dddee35c328790f43532cb8da                                                        0.0s
 => => exporting config sha256:b8376d1efdb7f2b30e0bdc3f227b751fab323fcf0c81628505262d580d24a6f0                                                          0.0s
 => => exporting manifest list sha256:9891ae92d2f8a70aedcf316a926307c075ce319f4411c6f5eaedf5a885703271                                                   0.0s
 => => sending tarball                                                                                                                                   0.0s
Successfully built bug-base:latest
+ container build -t bug-test -f Dockerfile.test .
[+] Building 0.0s (0/1)
 => [resolver] fetching image...docker.io/library/alpine:latest                                                                                          0.0s
Error: internalError: "HTTP request to https://registry-1.docker.io/v2/library/bug-base/manifests/latest failed with response: 401 Unauthorized. Reason: Unknown, no credentials found for host registry-1.docker.io"

Code of Conduct

• I agree to follow this project's Code of Conduct

[saehejkang]

Commenting to follow this issue and mention I was able to reproduce it on my machine. I don’t have much context on how Dockerfiles are parsed and built, but I’m working on learning more about this domain.

I took a look at the container-builder-shim repo as it seems to basically be a wrapper around buildkit. There is doc note here that using COPY --from is a valid instruction. Furthermore, you are correct in assuming that the local image should be pulled first and it looks like the logic might need to be updated somewhere around here?

Someone that has more domain knowledge on this should help out here soon 😁.

[ivankra]

Probably just use agentic LLM (codex, claude code, opencode etc) to analyze and patch the code. If it helps, here's what I get from GPT and Claude - both seem to nail it in one shot. Yes, both complain about frontend.go's resolveStates(). I unfortunately don't have mac dev environment set up and can't easily test their patches (I develop inside those linux containers normally). If you have some time to look into it and polish a patch for PR I'd really appreciate it!

GPT 5.3

> investigate the codebase here for the following bug: [paste of bug report]

...

Root cause is a two-step mismatch between container-builder-shim and container image lookup.

1. COPY --from= is treated as an implicit stage in BuildKit, not a FROM stage.

• BuildKit creates an unregistered stage for unknown COPY --from sources at container-builder-shim/vendor/github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb/convert.go:829.
• That stage name is normalized to docker.io/library/...:latest before resolution at container-builder-shim/vendor/github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb/convert.go:474 and container-builder-shim/vendor/github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb/convert.go:482.
• If no named context exists, it calls resolver metadata directly at container-builder-shim/vendor/github.com/moby/
  buildkit/frontend/dockerfile/dockerfile2llb/convert.go:543.

2. Apple’s pre-seeding logic only pre-registers FROM base images, not external COPY --from sources.

• resolveStates() iterates parsed stages (FROMs) at container-builder-shim/pkg/build/frontend.go:132.
• It intentionally resolves with the original unqualified name for local preference at container-builder-shim/pkg/build/frontend.go:201.
• It writes named contexts keyed by familiar refs (bug-base::platform) at container-builder-shim/pkg/build/frontend.go:225.
• Because external COPY --from=bug-base is not pre-seeded, BuildKit falls back to resolver with normalized docker.io/library/bug-base:latest.

3. Resolver then misses local image due to exact-reference matching and pulls remote.

• Resolver handler uses ClientImage.fetch(reference: ref, ...) at container/Sources/ContainerBuild/BuildImageResolver.swift:81.
• fetch() first does local get(reference:); if not found, it pulls at container/Sources/Services/ContainerAPIService/Client/ClientImage.swift:245 and container/Sources/Services/ContainerAPIService/Client/ClientImage.swift:261.
• Local lookup compares exact stored refs (bug-base or bug-base:latest) vs normalized ref (docker.io/library/bug-base:latest) and misses at container/Sources/Services/ContainerAPIService/Client/ClientImage.swift:247.

Why FROM bug-base works:

• Shim’s resolveStates() explicitly avoids passing normalized refs to resolver (DO NOT USE ref.String()) at container-builder-shim/pkg/build/frontend.go:197, so local tag lookup succeeds there.

Test gap:

• Existing tests only cover COPY --from= (e.g., builder), not COPY --from= in container-builder-shim/pkg/build/frontend_test.go:449.

Most direct fix options:

1. In shim: extend resolveStates() to also pre-seed external COPY --from= sources as named contexts (preferred behavior-aligned fix).
2. In container client: make ClientImage._search match normalized and denormalized equivalents (defensive fix that helps other code paths too).

Here’s a minimal patch sketch for container-builder-shim/pkg/build/frontend.go, focused only on pre-seeding external COPY --from= refs.

diff --git a/pkg/build/frontend.go b/pkg/build/frontend.go
index ... .. ...
--- a/pkg/build/frontend.go
+++ b/pkg/build/frontend.go
@@ -21,6 +21,7 @@ import (
      "encoding/json"
      "fmt"
      "strings"
+	"strconv"
      "sync"

      "github.com/containerd/containerd/reference"
@@ -129,6 +130,84 @@ func resolveStates(ctx context.Context, bopts *BOpts, platform ocispecs.Platform, clog
func(string, ...interface{})) (map[string]stateMeta, error) {
      states := map[string]stateMeta{}
      stateLock := sync.Mutex{}

+	resolveSource := func(sourceName string, sourcePlatform ocispecs.Platform) error {
+		if strings.EqualFold(sourceName, "scratch") || strings.EqualFold(sourceName, "context") {
+			return nil
+		}
+
+		ref, err := dref.ParseAnyReference(sourceName)
+		if err != nil {
+			if err == reference.ErrObjectRequired {
+				return nil
+			}
+			return fmt.Errorf("invalid ref: %s", sourceName)
+		}
+
+		clog("[resolver] fetching image...%s", ref.String())
+
+		resolverOpts := sourceresolver.Opt{}
+		resolverOpts.OCILayoutOpt = &sourceresolver.ResolveOCILayoutOpt{
+			Store: sourceresolver.ResolveImageConfigOptStore{
+				StoreID:   "container",
+				SessionID: "",
+			},
+		}
+		resolverOpts.ImageOpt = &sourceresolver.ResolveImageOpt{
+			Platform:    &sourcePlatform,
+			ResolveMode: llb.ResolveModePreferLocal.String(),
+		}
+
+		_, digest, img, err := bopts.Resolver.ResolveImageConfig(ctx, sourceName, resolverOpts)
+		if err != nil {
+			if err == reference.ErrObjectRequired {
+				return nil
+			}
+			return err
+		}
+
+		fqdn := ref.String()
+		if _, ok := ref.(dref.Digested); !ok {
+			fqdn += "@" + digest.String()
+		}
+		st := llb.OCILayout(fqdn, llb.OCIStore("", "container"), llb.Platform(sourcePlatform))
+
+		named, err := dref.ParseNormalizedNamed(ref.String())
+		if err != nil {
+			return fmt.Errorf("invalid context name %s %v", ref.String(), err)
+		}
+		name := strings.TrimSuffix(dref.FamiliarString(named), ":latest")
+		pname := name + "::" + platforms.FormatAll(platforms.Normalize(sourcePlatform))
+
+		imgMetaMap := map[string][]byte{
+			exptypes.ExporterImageConfigKey: img,
+		}
+		imgMeta, err := json.Marshal(imgMetaMap)
+		if err != nil {
+			return err
+		}
+
+		stateLock.Lock()
+		states[pname] = stateMeta{
+			state:   st.Platform(sourcePlatform),
+			imgMeta: imgMeta,
+		}
+		stateLock.Unlock()
+		return nil
+	}
+
@@ -145,11 +224,6 @@ func resolveStates(ctx context.Context, bopts *BOpts, platform ocispecs.Platform, clog
func(string, ...interface{})) (map[string]stateMeta, error) {
                      if err != nil {
                              errCh <- fmt.Errorf("invalid arg for stage[%s]: %v", stage.BaseName, err)
                              return
                      }
-
-			if strings.EqualFold(resolvedBaseStageName.Result, "scratch") || strings.EqualFold(resolvedBaseStageName.Result, "context") {
-				return
-			}
-
                      // if platform is specified for the stage, parse and use as the target platform
                      stagePlatform := platform
                      if stage.Platform != "" {
@@ -171,71 +245,39 @@ func resolveStates(ctx context.Context, bopts *BOpts, platform ocispecs.Platform, clog
func(string, ...interface{})) (map[string]stateMeta, error) {
                              if hasNamedStage && namedIndex < i {
                                      return
                              }
-
-				ref, err := dref.ParseAnyReference(resolvedBaseStageName.Result)
-				...
-				states[pname] = stateMeta{ ... }
+				if err := resolveSource(resolvedBaseStageName.Result, stagePlatform); err != nil {
+					logrus.Errorf("error resolving image: %v", err)
+					errCh <- err
+					return
+				}
                      }(stage)
              }
+
+	// Pre-seed external COPY --from=<image> refs as named contexts so BuildKit
+	// does not fall back to remote resolver for normalized docker.io refs.
+	for i, stage := range stages {
+		for _, cmd := range stage.Commands {
+			c, ok := cmd.(*instructions.CopyCommand)
+			if !ok || c.From == "" {
+				continue
+			}
+			if _, err := strconv.Atoi(c.From); err == nil {
+				continue // COPY --from=<stage index>
+			}
+			if namedIndex, hasNamedStage := instructions.HasStage(stages, c.From); hasNamedStage && namedIndex < i {
+				continue // COPY --from=<previous stage name>
+			}
+			// BuildKit does not support arg expansion in COPY --from, so use literal c.From.
+			if err := resolveSource(c.From, platform); err != nil {
+				logrus.Errorf("error resolving COPY --from image: %v", err)
+				return nil, err
+			}
+		}
+	}

Opus 4.6

> investigate the codebase here for the following bug: [paste of bug report]

...

Root Cause

The bug is in container-builder-shim/pkg/build/frontend.go — the resolveStates() function only pre-resolves images
referenced in FROM instructions, but not images referenced in COPY --from= instructions.

The Two Paths

FROM bug-base (works):

1. resolveStates() in frontend.go:114-252 parses the Dockerfile and processes each FROM base image
2. Line 201: It calls the resolver with the raw, un-normalized name "bug-base" (there's even a comment at lines 195-200 warning NOT to use the normalized ref.String())
3. The Swift BuildImageResolver receives "bug-base", calls ClientImage.fetch("bug-base"), which calls _search("bug-base") — the reference "bug-base" matches the stored Image.Description.reference of "bug-base"
4. Line 215: Creates an llb.OCILayout() state pointing to the local OCI store
5. In solvePlatform() (lines 310-318), the resolved state is registered as a named context via frontendOpt["context:bug-base::linux/arm64"]
6. When Dockerfile2LLB later processes the FROM, it finds the named context and uses the pre-resolved local image content

COPY --from=bug-base (broken):

1. resolveStates() does NOT process COPY --from — it only loops over stages (i.e., FROM instructions)
2. Dockerfile2LLB in vendor code convert.go:827-856 creates an unregistered dispatchState with BaseName: "bug-base"
3. In resolveReachableStages (convert.go:474): reference.ParseNormalizedNamed("bug-base") normalizes it to "docker.io/library/bug-base:latest"
4. Line 482: d.stage.BaseName becomes "docker.io/library/bug-base:latest"
5. Line 513: namedContext("docker.io/library/bug-base:latest", ...) checks for a named context — not found (only "alpine" was pre-resolved)
6. Line 543: Falls through to metaResolver.ResolveImageConfig(ctx, "docker.io/library/bug-base:latest", ...) — sends the normalized name to the Swift resolver
7. Swift receives "docker.io/library/bug-base:latest", calls _search("docker.io/library/bug-base:latest") in ClientImage.swift:224:
   - Annotation check: nameFromAnnotation() doesn't match "docker.io/library/bug-base:latest"
   - Reference check: image.reference == "docker.io/library/bug-base:latest" → "bug-base" !=
   "docker.io/library/bug-base:latest" — no match
8. fetch() falls through to pull(), which tries registry-1.docker.io/v2/library/bug-base/manifests/latest → 401 Unauthorized
9. Even if the resolver somehow succeeded, line 593 would create llb.Image(d.stage.BaseName) — a docker-image://source that pulls from the registry, not llb.OCILayout() that uses the local content store

Two Contributing Factors

1. Name normalization mismatch: The vendored BuildKit code (convert.go:474,482) normalizes "bug-base" → "docker.io/library/bug-base:latest" before calling the resolver, but the image was stored locally with reference "bug-base". The _search() function can't match these.
2. Wrong LLB source type: Even if resolution succeeded, convert.go:593 uses llb.Image() (pulls from registry) instead of llb.OCILayout() (reads from local store). Only resolveStates() in frontend.go knows to use llb.OCILayout with the "container" OCI store.

Fix Location

The fix should be in frontend.go:resolveStates() — it needs to also parse COPY --from directives and pre-resolve any external image references (that aren't stage names) the same way it handles FROM images. This way:

• The image name is sent to the resolver un-normalized → found locally
• An llb.OCILayout state is created → content served from local store
• A named context is registered → Dockerfile2LLB finds it and skips the broken path

The key files involved:

• container-builder-shim/pkg/build/frontend.go:114-252 — needs the fix (also resolve COPY --from images)
• container-builder-shim/vendor/.../dockerfile2llb/convert.go:474-598 — the vendored BuildKit code that normalizes the name (can't easily change)
• container/Sources/ContainerBuild/BuildImageResolver.swift — Swift resolver
• container/Sources/Services/ContainerAPIService/Client/ClientImage.swift:224-249 — _search() function that fails to match normalized vs raw names

✻ Brewed for 12m 15s

❯ show me a minimal fix here (for container-builder-shim/pkg/build/frontend.go:114-252). just print diff in chat

● Here's the minimal diff. The fix adds a second pass in resolveStates() that collects COPY --from= references
to external images and pre-resolves them as named contexts — using the raw (un-normalized) name and llb.OCILayout —
exactly like the existing FROM resolution does:

--- a/container-builder-shim/pkg/build/frontend.go                                                                  
+++ b/container-builder-shim/pkg/build/frontend.go
@@ -19,6 +19,7 @@ import (
      "context"
      "encoding/json"
      "fmt"
+     "strconv"
      "strings"
      "sync"

@@ -123,6 +124,38 @@ func resolveStates(ctx context.Context, bopts *BOpts, platform ocispecs.Platform
              return nil, err
      }

+     // Collect external image references from COPY --from that are not
+     // stage names, numeric stage indices, or FROM base images.
+     // These must be pre-resolved as named contexts so that Dockerfile2LLB
+     // can use them without normalizing the reference (which adds the
+     // default docker.io/library/ domain and breaks local image lookup).
+     stageNames := make(map[string]struct{})
+     stageBaseNames := make(map[string]struct{})
+     for _, stage := range stages {
+             if stage.Name != "" {
+                     stageNames[strings.ToLower(stage.Name)] = struct{}{}
+             }
+             stageBaseNames[stage.BaseName] = struct{}{}
+     }
+     var copyFromRefs []string
+     seen := make(map[string]struct{})
+     for _, stage := range stages {
+             for _, cmd := range stage.Commands {
+                     c, ok := cmd.(*instructions.CopyCommand)
+                     if !ok || c.From == "" {
+                             continue
+                     }
+                     from := c.From
+                     if _, err := strconv.Atoi(from); err == nil {
+                             continue // numeric stage index
+                     }
+                     if _, ok := stageNames[strings.ToLower(from)]; ok {
+                             continue // references a named stage
+                     }
+                     if _, ok := stageBaseNames[from]; ok {
+                             continue // already resolved as a FROM base
+                     }
+                     if strings.EqualFold(from, "scratch") || strings.EqualFold(from, "context") {
+                             continue
+                     }
+                     if _, ok := seen[from]; !ok {
+                             seen[from] = struct{}{}
+                             copyFromRefs = append(copyFromRefs, from)
+                     }
+             }
+     }
+
      wg := sync.WaitGroup{}
      doneCh := make(chan struct{})
      errCh := make(chan error)
@@ -241,6 +274,58 @@ func resolveStates(ctx context.Context, bopts *BOpts, platform ocispecs.Platform
              }(stage)
      }

+     // Resolve external images referenced by COPY --from so they become
+     // named contexts for Dockerfile2LLB. This mirrors the FROM resolution
+     // above: we pass the raw image name to the resolver (not the normalized
+     // form) and create an llb.OCILayout source backed by the local store.
+     for _, copyFromRef := range copyFromRefs {
+             wg.Add(1)
+             go func(imageName string) {
+                     defer wg.Done()
+
+                     ref, err := dref.ParseAnyReference(imageName)
+                     if err != nil {
+                             if err == reference.ErrObjectRequired {
+                                     return
+                             }
+                             errCh <- fmt.Errorf("invalid ref in COPY --from: %s", imageName)
+                             return
+                     }
+
+                     clog("[resolver] fetching image...%s", ref.String())
+
+                     resolverOpts := sourceresolver.Opt{}
+                     resolverOpts.OCILayoutOpt = &sourceresolver.ResolveOCILayoutOpt{
+                             Store: sourceresolver.ResolveImageConfigOptStore{
+                                     StoreID:   "container",
+                                     SessionID: "",
+                             },
+                     }
+                     resolverOpts.ImageOpt = &sourceresolver.ResolveImageOpt{
+                             Platform:    &platform,
+                             ResolveMode: llb.ResolveModePreferLocal.String(),
+                     }
+
+                     // NOTE: Use the raw imageName, NOT ref.String(), to avoid
+                     // adding the default domain which breaks local image lookup.
+                     _, digest, img, err := bopts.Resolver.ResolveImageConfig(ctx, imageName, resolverOpts)
+                     if err != nil {
+                             if err == reference.ErrObjectRequired {
+                                     return
+                             }
+                             logrus.Errorf("error resolving COPY --from image: %v", err)
+                             errCh <- err
+                             return
+                     }
+
+                     fqdn := ref.String()
+                     if _, ok := ref.(dref.Digested); !ok {
+                             fqdn = fqdn + "@" + digest.String()
+                     }
+                     st := llb.OCILayout(fqdn, llb.OCIStore("", "container"), llb.Platform(platform))
+
+                     named, err := dref.ParseNormalizedNamed(ref.String())
+                     if err != nil {
+                             errCh <- fmt.Errorf("invalid context name %s %v", ref.String(), err)
+                             return
+                     }
+                     name := strings.TrimSuffix(dref.FamiliarString(named), ":latest")
+                     pname := name + "::" + platforms.FormatAll(platforms.Normalize(platform))
+                     imgMetaMap := map[string][]byte{
+                             exptypes.ExporterImageConfigKey: img,
+                     }
+                     imgMeta, err := json.Marshal(imgMetaMap)
+                     if err != nil {
+                             errCh <- err
+                             return
+                     }
+
+                     stateLock.Lock()
+                     defer stateLock.Unlock()
+                     states[pname] = stateMeta{
+                             state:   st.Platform(platform),
+                             imgMeta: imgMeta,
+                     }
+             }(copyFromRef)
+     }
+
      go func() { wg.Wait(); doneCh <- struct{}{} }()
      select {
      case err := <-errCh:

How it works

The new code collects all COPY --from= references that point to external images (filtering out numeric stage
indices, named stages, scratch/context, and images already handled as FROM bases). For each one, it resolves the
image the same way the existing FROM loop does:

1. Sends the raw, un-normalized name (e.g. "bug-base") to the Swift resolver via ResolveImageConfig — this allows
   the Swift _search() to match it against the locally-stored Image.Description.reference
2. Creates an llb.OCILayout state backed by the local "container" OCI store — so content is served from the local
   content store, not pulled from a registry
3. Registers it in the states map — which solvePlatform() then passes as a named context via
   frontendOpt["context:"]

When Dockerfile2LLB later encounters the COPY --from=bug-base, its NamedContext() lookup normalizes
"docker.io/library/bug-base:latest" back to the familiar "bug-base", finds the pre-registered named context, and
uses the local image directly.

Note

RUN --mount=type=bind,from= has the same bug (see detectRunMount in vendored convert_runmount.go). Extending
the collection loop to also scan instructions.GetMounts(c) on *instructions.RunCommand would fix that too, but I
kept this focused on the reported COPY --from case.

There is doc note here that using COPY --from is a valid instruction

Seems a bit underdocumented there, but anyway the behavior of COPY --from=<image> that I described (local image tag checked first) - it works like that with docker and podman. I ran into this bug when porting my Dockerfiles that are working perfectly fine with podman on Linux to Apple's container.

[saehejkang]

If you have some time to look into it and polish a patch for PR I'd really appreciate it!

I’ll take a look at patching this up for you. I’ll also review the agent code you shared and see what works and what doesn’t. Since you’ve already done some digging into the issue (and likely would have fixed it yourself if you had a mac dev env), I’ll add you as a co-author when I merge the patch.

[saehejkang]

@JaewonHur pinging you because you recently made changes to the builder. Are there different build steps to setup container-builder-shim locally to test changes? I don't see anything on the docs and can draft something out for others to use in the future.

[JaewonHur]

Yes. container uses buildkit image downloaded from internet by default (i.e., run container system property get image.builder to see).

Following commands make it use local built container-builder-shim

git clone https://github.com/apple/container-builder-shim
cd container-builder-shim
# After updating code
container build -t builder .

container system property set image.builder builder:latest
container rm -f buildkit # Be sure to remove already existing buildkit container to use the new one

container build ... # Now it works with newer builder (i.e., builder:latest)

Also, be sure to update the builder image (if modified one is broken) before building container-builder-shime again :)

[saehejkang]

@ivankra This does not seem like a trivial patch. Although the agents came up with a solution, none of it makes much sense to me (yet), and blindly copying pasting it won't be effective here. I could even be looking in the wrong place and this could be an update that needs to be made elsewhere.

Does the file below work for you? Just something temporary for you to use, while I continue taking a look.

FROM alpine:latest AS bug-base
RUN echo hello >/hello.txt
FROM alpine:latest
COPY --from=bug-base /hello.txt /copied.txt
RUN cat /copied.txt

If you would like a patch sooner, I can ping the proper maintainers and get someone else to take a look 😃.

[ivankra]

Yes, multistage builds defined within the same Dockerfile work. I can work around broken COPY --from=<tag>, but it's still a real bug in container (compared to docker/podman's behaviour)

[saehejkang]

@ivankra I was able to come back to this after gaining more context on the builder and see how the AI code you pasted was usefeul! What is your email, so I can credit you?