docs: publish DeepSeek Flash Terminal-Bench comparison - #1719
Merged
Conversation
jackwener
pushed a commit
that referenced
this pull request
Jul 31, 2026
…log (#1723) The plan-reminder edit/duplicate flow defers opening the Astryx form dialog by two requestAnimationFrames after the row menu closes (the menu stashes an intent, runs it on close, then remounts a closed form session before Astryx observes a false->true transition). Astryx Dialog restores focus on Escape to whatever was `document.activeElement` when it opened, and the menu's own close-focus-return sits in that same window. On a loaded CI runner the captured element can land on <body> instead of the menu trigger, so Escape strands focus away from the row and `expect(menu).toBeFocused()` times out. This made the `plan-reminders` E2E flaky on docs-only PR #1719 (same code tree passed on #1714 and failed on #1719; a rerun went green), and the suite runs with retries: 0 so the flake blocks unrelated PRs. Keep a ref to each row's menu trigger and re-focus it inside the deferred intent, right before the dialog opens, so the dialog captures and later restores focus to the row's trigger deterministically — independent of the menu-close vs deferred-open micro-timing. Verified: packages/ui typecheck + 296 contract tests, biome lint, and the plan-reminders E2E (12/12 with --repeat-each 6).
Astro-Han
added a commit
that referenced
this pull request
Jul 31, 2026
#1720 (opencode-free as a zero-credential default provider) added `ensureBootstrapConnection`, which unconditionally seeds an `opencode-free` connection on first launch when no connections exist. The E2E gate that skips it, `if (!e2eFixture)`, only triggers when `MAKA_E2E_FIXTURE` is set, but the `emptyWindow` fixture (used by all four `first-run.spec.ts` tests) passes `seed: false` and no `e2eFixtureScenario`, so `MAKA_E2E_FIXTURE` is unset and the bootstrap runs — seeding `opencode-free` into the "empty" workspace. The onboarding state becomes `ready_empty` instead of `needs_connection`, so `OnboardingHero` no longer renders `.maka-firstrun-row`, and the four tests fail with `toHaveCount 0` / click timeouts. This is a deterministic regression on current main (not a flake): #1720's own e2e failed exactly these four; every post-#1720 run fails them identically; pre-#1720 runs passed. #1720 merged with the failing e2e, so every PR's e2e is now red on these four. The `needs_connection` first-run hero is now unreachable for a fresh install (opencode-free is always seeded, and re-seeded whenever the connection list is empty), so these tests assert a contract that no longer holds. Drop them now to unblock e2e; follow-up will add coverage for the post-#1720 fresh-install boot contract (opencode-free seeded -> ready_empty). Also remove the now-unused `emptyWindow` fixture (it was "Used by first-run only"). Unblocks the plan-reminder focus-restore fix (#1723) and the docs PR #1719, whose e2e runs merge with main and so also hit this cluster. Verified: `npx playwright test --list` parses the remaining 93 tests across 32 files with no import/syntax errors; `emptyWindow` has no remaining references. The 93 remaining tests passed in the CI runs that failed only on these four.
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1680.
Docs-only: English write-up of Terminal-Bench 2.1 harness A/B run
deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v7(Maka vs OpenCode, both on DeepSeek V4 Flash at max reasoning effort), plus the committed per-task outcome and failure-class CSV.Full report:
docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.mdResults
Exact two-sided McNemar over the 20 discordant pairs: p = 0.0118.
Why this comparison is clean
Unlike the earlier Kimi Code and GLM-5.2 comparisons, both arms run the same model at the same reasoning effort in a paired design, so the +13.48 pp gap is a statistically significant harness difference, not a descriptive observation — at effectively identical cost per pass (0.02% apart). Scope: one frozen run over one fixed 89-task suite; this is evidence about harness systems on this suite, not a universal model or single-component claim. Cost per pass is an observed point estimate, not a tested equivalence.
On the 61-pair non-budget subset (pairs where neither arm exhausted its budget), Maka passed 52/61 (85.25%) vs OpenCode 46/61 (75.41%) — diagnostic decomposition, not an alternate headline.
Recovery disclosure
The fully scored dataset required, and discloses: 183 Agent admissions (178 initial + 5 bounded whole-cell infrastructure retries), 25 taxonomy-only corrections that did not change pass/fail, and one verifier-only replay (
polyglot-rust-c, byte-for-byte candidate reconstruction, no additional model sampling). All are append-only and reconstructible from the frozen WALs, adjudication ledger, and verifier-replay evidence manifest; the CSV contains only final outcomes.Verification
git diff --checkRepository code tests were not run because this is a documentation-only change and the isolated worktree has no installed
node_modules.