Goal
Produce reproducible evidence of whether Maka is more effective and token-efficient than OpenCode when both use GLM-5.2 Max under the same Terminal-Bench 2.1 conditions.
Scope
- Reuse the existing
packages/headless Harbor path and Maka/OpenCode adapters.
- Run paired Pass@1 trials over a deterministic ordering of all 89 tasks.
- Treat the first 40 tasks as a resumable prefix of the full run.
- Freeze both harness configurations before the first formal trial.
- Store an immutable manifest and append-only results so interrupted runs execute only missing cells.
- Retry only infrastructure failures; agent failures remain benchmark outcomes.
- Report paired pass rates, token usage, and API-equivalent cost in JSON, CSV, and Markdown.
- Never store credentials or publish results automatically.
Done when
- Tests cover manifest identity, deterministic ordering, resume safety, failure classification, and metric calculation.
- A dataset-external smoke test validates both adapters without spending benchmark quota.
- The 40-task run can extend to all 89 tasks without rerunning valid results.
- The report supports an honest two-axis effectiveness/economy comparison.
Non-goals
- Pi, Kimi, Ollama, Pass@5, prompt tuning, LiteLLM, or broader model-catalog work.
Goal
Produce reproducible evidence of whether Maka is more effective and token-efficient than OpenCode when both use GLM-5.2 Max under the same Terminal-Bench 2.1 conditions.
Scope
packages/headlessHarbor path and Maka/OpenCode adapters.Done when
Non-goals