rollout

rollout is a reinforcement learning platform for language-model agents. It trains them on long, multi-turn, multi-agent episodes, such as four bots sharing a Minecraft world, a gridworld or a math problem. Episodes keep playing while the policy trains, and each new checkpoint is served to the agents during the run. It runs on one machine or on a Kubernetes cluster.

• One gateway for every agent loop. Agents sample through a gateway that speaks OpenAI Chat Completions, OpenAI Responses and Anthropic Messages, and records every token and logprob. Claude Code, Codex or your own loop can produce training data unchanged.
• Asynchronous, off-policy training. Episodes keep playing while the trainer steps. Each turn records the checkpoint that sampled it, max_lag bounds how many checkpoints behind the newest a turn may begin (you can change it mid-run), and importance weights correct for the rest, including the steps a long episode's early turns wait through before they train. Whole multi-turn segments train together.
• Training and inference backends. Train with Tinker, or with a LoRA or full-weight trainer on your own GPUs, and serve on vLLM. Each checkpoint is converted for each inference backend.
• Shared sandboxes. Sandboxes such as a Minecraft server or a computer are leased by claim, so four agents on four machines play in the same world.
• Recovery from crashes. Every process coordinates through a fenced, append-only ledger in Postgres. If a runner, server or trainer is killed mid-episode, no work is lost or counted twice. Runs pause and resume.
• Evals. Versioned suites of environments run on a schedule or against any checkpoint, with a score for each environment.

On one machine, the minecraft-one-gpu preset puts the 4-bit Qwen3.5-9B on vLLM and a rank-32 LoRA trainer on one 16 GB GPU; the engines sleep while the trainer steps. On Kubernetes, the included Helm chart deploys the platform with KubeRay.

 environments ──► episode runners ──► gateway ──► inference (vLLM, Tinker, pods)
                        │                │               ▲
                        ▼                ▼               │ follows serving records
                   ┌──────────── ledger (Postgres) ──────┴───┐   blob store (S3)
                   └── training loop ──► training provider ──► checkpoints ──► bridges

1. Runners play episodes, and the gateway records each turn they sample.
2. The training loop scores groups of episodes and has the training provider take a step.
3. Bridges convert the checkpoint for each inference backend, and the loop records what should be served.
4. Inference servers follow that record.

No process holds state that another depends on; they coordinate only through the ledger and the blob store. Read more in the architecture.

from rollout.contracts import Message
from rollout.harness import End, Observation, RunContext, Task


class Arithmetic(Task):
    def __init__(self, parameters: dict[str, str]) -> None:
        self.question, self.answer = parameters["question"], parameters["answer"]

    async def start(self, run: RunContext) -> Observation:
        return Observation(self.question)

    async def respond(self, run: RunContext, reply: Message) -> Observation:
        return End(reward=1.0 if reply.text.strip() == self.answer else 0.0)

This task asks a question and rewards an exact answer. To check an environment, train on it and watch:

uv run rollout env check minecraft_team.environment:environment
uv run rollout preset load deploy/chart/rollout/files/presets --cluster
uv run rollout train minecraft_team.environment:environment --preset minecraft-one-gpu
uv run rollout monitor --cluster   # runs, every transcript, the checkpoint tree, evals, machines

rollout is a uv workspace for Python 3.13.

uv sync                  # everything that needs no GPU
uv sync --all-extras     # plus vLLM, the trainers, Tinker and the renderers (Linux, NVIDIA GPU)
uv run pytest            # the tests, including every example in the docs

Once the extras are installed, keep syncing with --all-extras: a plain uv sync uninstalls them.

• Start here: the ideas, in five minutes.
• Run a first episode: a task played against a scripted model.
• Deploy: on one machine, or on Kubernetes (K3s with a GPU).
• All documentation and the API reference.

To check your changes, run uv run ruff check && uv run ruff format --check && uv run pyright. After changing public names or docstrings, regenerate the API reference with uv run python scripts/generate_reference.py.