Skip to content

feat(headless): add resumable GLM harness benchmark - #862

Merged
Astro-Han merged 28 commits into
mainfrom
feat/headless-glm-harness-ab
Jul 13, 2026
Merged

feat(headless): add resumable GLM harness benchmark#862
Astro-Han merged 28 commits into
mainfrom
feat/headless-glm-harness-ab

Conversation

@Astro-Han

Copy link
Copy Markdown
Contributor

Summary

  • Add a fixed, resumable Terminal-Bench 2.1 comparison between Maka and OpenCode on GLM-5.2 Max.
  • Freeze task order, harness identities, reasoning effort, pricing, and execution constraints in a fingerprinted manifest.
  • Export separate Pass@1 effectiveness and cache-aware API-equivalent economy reports in JSON, CSV, and Markdown.

Why

We need reproducible evidence for comparing harness effectiveness and economy without mixing model, prompt, task-order, or billing changes into the result. This extends the existing headless, fixed-prompt, and Harbor seams so a 40-task pilot can resume into the full 89-task suite without rerunning completed cells.

Closes #861.

Scope

This PR adds benchmark infrastructure only. It does not run the benchmark, commit result artifacts, add Kimi or Pi arms, introduce LiteLLM, or change product defaults.

Verification

  • npm test --workspace @maka/headless — 822 tests passed after rebasing onto the latest origin/main.
  • npm run check:stale — passed.
  • git diff --check origin/main...HEAD — passed.
  • Secret-shaped token scan over the branch diff — no matches.
  • No model or benchmark calls were made.
  • No visual evidence is applicable because this change has no user-visible UI.

Impact

Headless benchmark operators gain a single fixed entrypoint with append-only resume state, execution attestation, and normalized reports. Existing product behavior and public defaults are unchanged; no migration is required.

Reviewer notes

Please focus on fairness invariants, resume correctness, secret isolation, OpenCode token accounting, and whether the implementation reuses the closest existing headless/Harbor seams without parallel state.

Ready for review

  • Verification is complete and the results above are current
  • Impact, compatibility, migration, documentation, and release-note needs are addressed
  • User-visible UI/UX changes include visual evidence, or Verification explains why it is not applicable

Astro-Han added 28 commits July 13, 2026 16:47
@Astro-Han
Astro-Han merged commit 62c503e into main Jul 13, 2026
3 checks passed
@Astro-Han
Astro-Han deleted the feat/headless-glm-harness-ab branch July 13, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(headless): add a resumable Maka vs OpenCode GLM-5.2 benchmark

1 participant