docker-compose config not always considered

When running an embedding model via Docker Model Runner, the runtime configuration (--ubatch-size, context_size, etc.) behaves inconsistently depending on how the request is sent.

Using Docker Desktop GUI:
The model starts with the custom configuration (e.g., ubatch-size=2048).

Using curl from inside a container:
The model falls back to default values (ubatch-size=512, n_ctx=4096).

This makes it impossible to reliably control the physical batch size.

Steps to Reproduce

docker-compose.yaml

models:
  embedding:
    model: ai/embeddinggemma:300M-Q8_0
    context_size: 2048
    runtime_flags:
      - "--ubatch-size"
      - "2048"

services:
  curl-tester:
    image: curlimages/curl:8.11.1
    command: ["sh", "-lc", "sleep 1000000"]
    models:
      embedding:
        endpoint_var: EMBEDDING_ENDPOINT
        model_var: EMBEDDING_MODEL

Trigger a request via Docker Desktop GUI
Observe logs:

[2025-09-23T17:19:35.420131000Z] llama_context: constructing llama_context
[2025-09-23T17:19:35.420164000Z] llama_context: n_seq_max     = 1
[2025-09-23T17:19:35.420178000Z] llama_context: n_ctx         = 2048
[2025-09-23T17:19:35.420190000Z] llama_context: n_ctx_per_seq = 2048
[2025-09-23T17:19:35.420200000Z] llama_context: n_batch       = 2048
[2025-09-23T17:19:35.420210000Z] llama_context: n_ubatch      = 2048

Trigger a request via curl inside the container

docker exec -it tailscale-curl-tester-1 sh -lc '
echo "EMBED:" $EMBEDDING_ENDPOINT $EMBEDDING_MODEL

curl -sS "$EMBEDDING_ENDPOINT/embeddings" \
  -H "Authorization: Bearer dummy" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$EMBEDDING_MODEL\",\"input\":\"hello world\"}" \
  | head -c 300; echo
'

Observe logs:

[2025-09-23T17:21:57.905751000Z] llama_context: constructing llama_context
[2025-09-23T17:21:57.905780000Z] llama_context: n_ctx         = 4096
[2025-09-23T17:21:57.905798000Z] llama_context: n_ctx_per_seq = 4096
[2025-09-23T17:21:57.905807000Z] llama_context: n_batch       = 2048
[2025-09-23T17:21:57.905815000Z] llama_context: n_ubatch      = 512

@updiversity

Hey @updiversity, thanks for raising the issue. I have the following theory, could you confirm the timeline?

You did a docker-compose up
The model was loaded, and you interacted with it through the GUI.
You either interacted with other models, leading to an evection of the loaded model by the scheduler, or there was a longer period of time, in which you haven't interacted with the model, hence it was also evicted by the scheduler
You then called the model from within the container, and it was loaded again implicitly, but this case without the values provided by Compose

Edit:
No sorry, I can reproduce, even without running other model in-between or having the timeout being in effect. Thanks for sharing.

Thanks !

I wanted to simplify the issue posting. But my original docker-compose file is this one

models:
  llm:
    model: ai/qwen3:30B-A3B-Q4_K_M
    context_size: 16384
    runtime_flags:
      - "--temp"
      - "0"
  embedding:
    model: ai/embeddinggemma:300M-Q8_0
    context_size: 2048
    runtime_flags:
      - "--ubatch-size" 
      - "2048"
services:
  curl-tester:
    image: curlimages/curl:8.11.1
    command: ["sh","-lc","sleep 1000000"]
    models:
      embedding:
        endpoint_var: EMBEDDING_ENDPOINT
        model_var: EMBEDDING_MODEL 
      llm:
        endpoint_var: LLM_ENDPOINT
        model_var: LLM_MODEL

but interestingly, the llm model is loaded properly each time, either when using the docker desktop gui, or via curl inside a container.

Very curious to know what is the root cause.

@doringeman

Thanks again, we actually found the root cause and it is related to a bug in correctly configuring embedding models for the embeddings endpoint. @doringeman might share more details, but we'll look into fixing It ASAP.

@updiversity

Hey @updiversity,

The root cause is that currently the configuration for embedding models is broken. It works from the Docker Desktop GUI because that one uses the completions endpoint (/v1/chat/completions instead of /v1/embeddings).
Thank you for the report! We'll look into fixing this ASAP and keep you posted on the progress.

Reopening as the fix is not part of Docker Desktop, yet.

Desktop: release notes 4.48: Docker Model Runner (#23535)

@updiversity

The fix is now available in Docker Desktop 4.48.0 (Release notes)!

In order for the embedding model to be configured you need to add the --embeddings runtime flag (docs).
E.g.,

models:
  llm:
    model: ai/embeddinggemma:300M-Q8_0
    context_size: 2048
    runtime_flags:
      - "--embeddings"

Thank you again @updiversity for your valuable feedback!
Please don't hesitate to provide more feedback!
We're working on being able to ship updates and such fixes for Docker Model Runner faster, independently from Docker Desktop releases.

Navigation Menu

Repository navigation

docker-compose config not always considered #166

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

docker-compose config not always considered #166

Description

Activity

kiview commented on Sep 24, 2025

updiversity commented on Sep 24, 2025

kiview commented on Sep 24, 2025

doringeman commented on Sep 24, 2025

doringeman commented on Oct 1, 2025

doringeman commented on Oct 10, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions