docker-compose config not always considered

[updiversity]

When running an embedding model via Docker Model Runner, the runtime configuration (--ubatch-size, context_size, etc.) behaves inconsistently depending on how the request is sent.

Using Docker Desktop GUI:
The model starts with the custom configuration (e.g., ubatch-size=2048).

Using curl from inside a container:
The model falls back to default values (ubatch-size=512, n_ctx=4096).

This makes it impossible to reliably control the physical batch size.

Steps to Reproduce

1. docker-compose.yaml

models:
  embedding:
    model: ai/embeddinggemma:300M-Q8_0
    context_size: 2048
    runtime_flags:
      - "--ubatch-size"
      - "2048"

services:
  curl-tester:
    image: curlimages/curl:8.11.1
    command: ["sh", "-lc", "sleep 1000000"]
    models:
      embedding:
        endpoint_var: EMBEDDING_ENDPOINT
        model_var: EMBEDDING_MODEL

3. Trigger a request via Docker Desktop GUI
   Observe logs:

[2025-09-23T17:19:35.420131000Z] llama_context: constructing llama_context
[2025-09-23T17:19:35.420164000Z] llama_context: n_seq_max     = 1
[2025-09-23T17:19:35.420178000Z] llama_context: n_ctx         = 2048
[2025-09-23T17:19:35.420190000Z] llama_context: n_ctx_per_seq = 2048
[2025-09-23T17:19:35.420200000Z] llama_context: n_batch       = 2048
[2025-09-23T17:19:35.420210000Z] llama_context: n_ubatch      = 2048

4. Trigger a request via curl inside the container

docker exec -it tailscale-curl-tester-1 sh -lc '
echo "EMBED:" $EMBEDDING_ENDPOINT $EMBEDDING_MODEL

curl -sS "$EMBEDDING_ENDPOINT/embeddings" \
  -H "Authorization: Bearer dummy" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$EMBEDDING_MODEL\",\"input\":\"hello world\"}" \
  | head -c 300; echo
'

Observe logs:

[2025-09-23T17:21:57.905751000Z] llama_context: constructing llama_context
[2025-09-23T17:21:57.905780000Z] llama_context: n_ctx         = 4096
[2025-09-23T17:21:57.905798000Z] llama_context: n_ctx_per_seq = 4096
[2025-09-23T17:21:57.905807000Z] llama_context: n_batch       = 2048
[2025-09-23T17:21:57.905815000Z] llama_context: n_ubatch      = 512

[kiview]

Hey @updiversity, thanks for raising the issue. I have the following theory, could you confirm the timeline?

• You did a docker-compose up
• The model was loaded, and you interacted with it through the GUI.
• You either interacted with other models, leading to an evection of the loaded model by the scheduler, or there was a longer period of time, in which you haven't interacted with the model, hence it was also evicted by the scheduler
• You then called the model from within the container, and it was loaded again implicitly, but this case without the values provided by Compose

Edit:
No sorry, I can reproduce, even without running other model in-between or having the timeout being in effect. Thanks for sharing.

[updiversity]

Thanks !

I wanted to simplify the issue posting. But my original docker-compose file is this one

models:
  llm:
    model: ai/qwen3:30B-A3B-Q4_K_M
    context_size: 16384
    runtime_flags:
      - "--temp"
      - "0"
  embedding:
    model: ai/embeddinggemma:300M-Q8_0
    context_size: 2048
    runtime_flags:
      - "--ubatch-size" 
      - "2048"
services:
  curl-tester:
    image: curlimages/curl:8.11.1
    command: ["sh","-lc","sleep 1000000"]
    models:
      embedding:
        endpoint_var: EMBEDDING_ENDPOINT
        model_var: EMBEDDING_MODEL 
      llm:
        endpoint_var: LLM_ENDPOINT
        model_var: LLM_MODEL

but interestingly, the llm model is loaded properly each time, either when using the docker desktop gui, or via curl inside a container.

Very curious to know what is the root cause.

[kiview]

Thanks again, we actually found the root cause and it is related to a bug in correctly configuring embedding models for the embeddings endpoint. @doringeman might share more details, but we'll look into fixing It ASAP.

[doringeman]

Hey @updiversity,

The root cause is that currently the configuration for embedding models is broken. It works from the Docker Desktop GUI because that one uses the completions endpoint (/v1/chat/completions instead of /v1/embeddings).
Thank you for the report! We'll look into fixing this ASAP and keep you posted on the progress.

[doringeman]

Reopening as the fix is not part of Docker Desktop, yet.

[doringeman]

The fix is now available in Docker Desktop 4.48.0 (Release notes)!

In order for the embedding model to be configured you need to add the --embeddings runtime flag (docs).
E.g.,

models:
  llm:
    model: ai/embeddinggemma:300M-Q8_0
    context_size: 2048
    runtime_flags:
      - "--embeddings"

Thank you again @updiversity for your valuable feedback!
Please don't hesitate to provide more feedback!
We're working on being able to ship updates and such fixes for Docker Model Runner faster, independently from Docker Desktop releases.