VLAPilot: A Scheduling Agent for Vision-Language-Action Models

Project page | License

VLAPilot turns short-horizon VLA skills into long-horizon robot missions. It pairs a VLM planner with a separate VLM verifier, keeps every step explicit and traceable, and drives the robot through an MCP backend that owns hardware control.

This repository uses a controlled plan-mode pipeline:

Nanobot / CLI
  -> vlapilot.interface.mcp_server     # planner MCP server
  -> vla_pi05_piper.server             # Pi0.5/Piper backend MCP server
  -> VLARuntime / RTCExecutor          # arm execution plus verification

The planner observes the scene, emits a strict JSON plan, validates each step against the trained task vocabulary, and then executes the approved plan one step at a time. The verifier compares baseline and current camera frames after each step starts, so the system can continue, stop, or fail with a concrete reason.

Desk-cleaner demos cover direct interaction, object hand-over, object placement, and multi-step plan -> act -> verify missions. Preview assets live under examples/demo/previews/; raw clips can be placed under examples/demo/ as described in examples/demo/README.md. Multi-step rows below use in-repo full-length GIFs so GitHub resolves them reliably; click opens MP4 on the project page for best quality.

Clean up the desk multi-step | autonomous | 1x

Wrap up the desk multi-step | autonomous | 1x

Hand me the bottle single-step | 1x

Hand me the screwdriver single-step | 1x

Hand me a pen single-step | 1x

Wave hello single-step | 1x

More demos add MP4 clips or generated previews under examples/demo/ Demo asset guide

An upstream caller, such as Nanobot or the CLI, sends a high-level goal to VLAPilot. The planner MCP server captures camera frames from the backend, builds a planner prompt from the active mission, and asks the planner VLM for a strict JSON plan:

{
  "steps": [
    {
      "step": 1,
      "instruction": "Put the bottle into the white box.",
      "completion_mode": "single",
      "rationale": "The bottle is large and blocks the workspace."
    }
  ]
}

Every instruction must exactly match the intersection of:

• the mission vocabulary in examples/missions/<name>/tasks.yaml
• the robot capability vocabulary configured by arm.capability_tasks_file

After approval, vla_execute starts the plan in the background. For each step, VLAPilot calls the backend vla_start(..., verify_mode="manual"), polls vla_verify_once, saves observations and events, and then moves to the next step only after the verifier returns completed. A failure verdict, timeout, or backend error stops the whole plan and records the reason.

Python 3.10 or higher is required.

git clone https://github.com/JinghangLi/vlapilot.git
cd vlapilot
python3.10 -m venv .venv
source .venv/bin/activate
pip install -e .[dev]

Install the example backend dependencies when running the real Pi0.5/Piper robot stack:

pip install -e .[examples]

mkdir -p ~/.vlapilot
cp config.example.json ~/.vlapilot/config.json

Edit ~/.vlapilot/config.json:

• agent.planner.api_key: LLM key for the planner
• agent.mission_dir: path to a mission directory
• arm.verify.providers.*: VLM provider config for verification
• arm.capability_tasks_file: task vocabulary supported by the backend
• arm.mcp_command and arm.mcp_args: backend MCP server command

Config lookup order:

1. Explicit --config
2. ~/.vlapilot/config.json
3. ~/.vla/config.json

Run one mission directly from the CLI:

python3 scripts/run_agent.py \
  --config ~/.vlapilot/config.json \
  --instruction "Clean up everything on the desk."

Expose the planner MCP server to Nanobot or another upstream agent:

vlapilot --config ~/.vlapilot/config.json

Start the robot backend directly when debugging lower-level runtime behavior:

vla-pi05-piper --config ~/.vlapilot/config.json

Use the local REPL to exercise either MCP layer:

python3 scripts/mcp_repl.py --server agent --config ~/.vlapilot/config.json
python3 scripts/mcp_repl.py --server arm --config ~/.vlapilot/config.json

The outer planner MCP server exposes:

The backend robot MCP server exposes:

Wrap a robot in a stdio MCP server that exposes the backend tools listed above. VLAPilot talks to that server through ArmClient, so the planner and robot do not need direct Python coupling. The bundled reference backend lives in examples/vla_pi05_piper/.

A mission is a directory of runtime inputs:

my_mission/
|-- mission.md       # user-facing mission notes and compatibility guidance
|-- mission_mcp.md   # planner prompt for plan-mode execution
|-- tasks.yaml       # executable task vocabulary
`-- verify.md        # verifier prompt and success criteria

Point agent.mission_dir at the mission directory and keep arm.capability_tasks_file aligned with the same task vocabulary.

vlapilot
|-- src/vlapilot/
|   |-- agent.py                     # one-shot CLI plan pipeline
|   |-- interface/mcp_server.py      # planner MCP server used by Nanobot
|   |-- backend/mcp_client.py        # backend MCP client
|   `-- core/
|       |-- plan.py                  # planner output parsing and validation
|       |-- execution.py             # shared step execution and verify loop
|       |-- mission.py               # mission, task, and prompt loading
|       `-- session.py               # JSONL sessions and media files
|
|-- examples/vla_pi05_piper/         # Pi0.5/Piper backend MCP server
|-- examples/missions/desk_cleanup/  # reference mission
|-- examples/demo/                   # demo previews and clip guide
|-- scripts/                         # CLI, MCP REPL, prompt testing helpers
`-- tests/                           # pytest regression tests

Each run writes a session under agent.session_dir:

~/.vlapilot/sessions/
|-- m_<timestamp>.jsonl
`-- m_<timestamp>/
    |-- events.jsonl
    `-- media/
        |-- plan-001-frame.jpg
        `-- exec-step-1-002-frame.jpg

When arm.verify.debug=true, the backend verifier also writes request, response, baseline, and current-frame artifacts under debug/verify/.

Run the main regression suite:

pytest -q tests

Run only the core MCP/runtime path:

pytest tests/test_agent_mcp.py tests/test_runtime.py tests/test_server_resources.py

After changing examples/missions/*/verify.md, run the verify prompt regression test:

python3 scripts/test_verify_prompt.py examples/missions/desk_cleanup/verify.md

• Jinghang Li
• Qing Lian
• Yuhan Xi
• Qing Jiang

MIT