Skip to content

feat(headless): add a resumable Maka vs OpenCode GLM-5.2 benchmark #861

Description

@Astro-Han

Goal

Produce reproducible evidence of whether Maka is more effective and token-efficient than OpenCode when both use GLM-5.2 Max under the same Terminal-Bench 2.1 conditions.

Scope

  • Reuse the existing packages/headless Harbor path and Maka/OpenCode adapters.
  • Run paired Pass@1 trials over a deterministic ordering of all 89 tasks.
  • Treat the first 40 tasks as a resumable prefix of the full run.
  • Freeze both harness configurations before the first formal trial.
  • Store an immutable manifest and append-only results so interrupted runs execute only missing cells.
  • Retry only infrastructure failures; agent failures remain benchmark outcomes.
  • Report paired pass rates, token usage, and API-equivalent cost in JSON, CSV, and Markdown.
  • Never store credentials or publish results automatically.

Done when

  • Tests cover manifest identity, deterministic ordering, resume safety, failure classification, and metric calculation.
  • A dataset-external smoke test validates both adapters without spending benchmark quota.
  • The 40-task run can extend to all 89 tasks without rerunning valid results.
  • The report supports an honest two-axis effectiveness/economy comparison.

Non-goals

  • Pi, Kimi, Ollama, Pass@5, prompt tuning, LiteLLM, or broader model-catalog work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions