fix(configure): handle mode embedding

[doringeman]

Part of a set of changes required to fix #166.
Next, I'll need to patch docker/model-cli, docker/compose-spec and docker/compose.

In docker/model-cli I need to add a new flag --mode <completion|embedding> (default: completion).
In docker/compose-spec I plan to add mode part of ModelConfig. Now, the latter, together with the docker/model-cli change will be used in docker/compose to configure the model on docker compose up.

(I pushed to a branch in docker/model-runner so I can use this reference in docker/model-cli.)

E.g.,

services:
  model1:
    provider:
      type: model
      options:
        model: ai/embeddinggemma:300M-Q8_0
        context-size: 2048
        runtime-flags: "--no-prefill-assistant --embeddings"

Note: The flag is now added twice, but it doesn't break llama.cpp.

llamaCppArgs: [--jinja -ngl 999 --metrics --threads 5 --model /Users/doringeman/.docker/models/bundles/sha256/b6635ddcd4cb1990b332980e34202fca9eeaef18ec0806afbf89bcb9989b7bbe/model.gguf --host inference-runner-0.sock --embeddings --ctx-size 2048 --no-prefill-assistant --embeddings]

E.g.,

$ MODEL_RUNNER_PORT=8080 make run LLAMA_SERVER_VERSION=v0.0.16

$ MODEL_RUNNER_HOST=http://localhost:8080 docker model configure --mode embedding ai/embeddinggemma:300M-Q8_0 --context-size 2048 -- --embeddings

$ curl http://localhost:8080/engines/llama.cpp/v1/embeddings -X POST -d '{"model": "ai/embeddinggemma:300M-Q8_0", "input": "hello"}'

Summary by Sourcery

Introduce a 'mode' parameter in the ConfigureRequest API and use it to determine the backend mode when configuring the model runner instead of always defaulting to completion mode.

Bug Fixes:

• Parse and pass the requested mode to setRunnerConfig instead of hardcoding completion mode

Enhancements:

• Add a 'Mode' field to the ConfigureRequest struct in the HTTP scheduling API

[sourcery-ai]

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

This PR adds support for specifying the backend inference mode (completion or embedding) by extending the ConfigureRequest API and propagating the requested mode through the scheduler’s configuration logic instead of using the hard-coded completion mode.

Sequence diagram for model configuration with mode support

sequenceDiagram
participant "compose (caller)"
participant "Scheduler"
participant "ModelManager"
participant "Loader"

"compose (caller)"->>Scheduler: POST /configure {Model, Mode, ...}
Scheduler->>ModelManager: GetModel(Model)
ModelManager-->>Scheduler: Model
Scheduler->>ModelManager: ResolveModelID(Model)
ModelManager-->>Scheduler: modelID
Scheduler->>Loader: setRunnerConfig(ctx, backend, modelID, Mode, runnerConfig)
Loader-->>Scheduler: (success or error)
Scheduler-->>"compose (caller)": Response

Loading

Class diagram for updated ConfigureRequest and Scheduler configuration

classDiagram
class ConfigureRequest {
  +string Model
  +string Mode
  +int64 ContextSize
  +[]string RuntimeFlags
  +string RawRuntimeFlags
}
class Scheduler {
  +Configure(w http.ResponseWriter, r *http.Request)
}
ConfigureRequest --> Scheduler : used in

Loading

File-Level Changes

Change	Details	Files
Extend the ConfigureRequest API to include an optional mode parameter	Add a Mode field with JSON tag to ConfigureRequest	pkg/inference/scheduling/api.go
Parse and apply the requested backend mode in the scheduler	Extract mode from the ConfigureRequest Call parseBackendMode on the provided mode string Pass the parsed mode to loader.setRunnerConfig instead of the fixed completion mode	pkg/inference/scheduling/scheduler.go

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[Copilot]

Pull Request Overview

This PR adds support for handling embedding mode in the model configuration by introducing a Mode field to the ConfigureRequest struct and using a mode parser to determine the appropriate backend mode instead of hardcoding completion mode.

• Added Mode field to ConfigureRequest struct to accept mode configuration
• Implemented mode parsing logic to convert string mode to BackendMode enum
• Updated scheduler configuration to use parsed mode instead of hardcoded completion mode

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
pkg/inference/scheduling/api.go	Added Mode field to ConfigureRequest struct
pkg/inference/scheduling/scheduler.go	Updated Configure method to parse and use the mode from request

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[gemini-code-assist]

Summary of Changes

Hello @doringeman, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a crucial update to the model configuration API by introducing a mode parameter. This enhancement allows for flexible specification of a model's operational mode, such as 'completion' or 'embedding', during configuration. It is a preparatory step for a broader set of changes aimed at resolving a known issue related to model runner functionality and will facilitate future integrations across Docker's model-related components.

Highlights

• Model Configuration API Enhancement: Introduced a new Mode field to the ConfigureRequest struct in pkg/inference/scheduling/api.go, allowing clients to specify the operational mode (e.g., completion, embedding) for a model during configuration.
• Dynamic Mode Handling in Scheduler: Updated the Configure handler in pkg/inference/scheduling/scheduler.go to parse the new Mode from the incoming request and pass it to the setRunnerConfig function, enabling dynamic configuration based on the specified mode rather than a hardcoded default.
• Prerequisite for Embedding Support: This change is a foundational step towards fully supporting 'embedding' mode for models, forming part of a larger effort to address docker/model-runner issue docker-compose config not always considered #166 and facilitate integration with docker/model-cli, docker/compose-spec, and docker/compose.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[sourcery-ai]

Hey there - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[gemini-code-assist]

Code Review

This pull request introduces a Mode field to the ConfigureRequest to allow specifying whether a model should be configured for completion or embedding. The change correctly propagates this new mode to the runner configuration, replacing the previously hardcoded completion mode. My main feedback is to consider making the mode parsing stricter. Currently, an invalid mode string will silently default to completion, which could hide configuration errors from the user. It would be more robust to return an error for invalid modes.