chore: prune source-regex tests and parallelize the e2e suite - #1727
Merged
Conversation
jackwener
force-pushed
the
chore/prune-static-contract-tests
branch
from
July 31, 2026 18:40
aaaad73 to
ae2e826
Compare
Second pass of the test cleanup, now repo-wide rather than desktop-only. A scan of all 811 test files sorted them by whether they execute the code they assert about; these do not: - 6 scripts/cu-*-contract tests regexed OTHER SCRIPTS' source for string literals (`/claude-sonnet-4-6/`, `/parallel_tool_calls: false/`). They ran nothing and broke on any legitimate edit of the file they watched. - 3 runtime contracts pinned import structure and retired identifier names by scanning the monorepo source tree — lint rules wearing a test costume. - 3 prN-localized-copy-contract files scanned a hardcoded list of ~40 renderer files for inline CJK. They were scaffolding for an i18n migration that finished; their shared helper went with them. The bilingual invariant they nominally guarded is still covered by the copy-catalog tests that call the catalogs (shell-copy, settings-preferences-copy, and friends), which assert no silent Chinese fallback in English. - 1 type-only test whose body was three `assertType<...>()` calls: zero runtime assertions, so it could not fail at runtime. tsc owns that. Also in this pass, e2e: - first-run.spec.ts asserted an onboarding registry screen that #1720 removed from the default flow (zero-credential `opencode-free` resolves a fresh install to `ready_empty`). Its 4 tests failed on main. Reduced to the boot smoke its own docblock called the valuable part; the provider catalog it browsed is covered by providers.spec.ts. - The suite ran `workers: 1` on the theory that parallel Electron windows fight over screen/IPC. They don't: the fixture mkdtemps a userData dir per test. Measured on the full suite: 1 worker ≈ 7min, 4 ≈ 2.4min. CI goes to 2 for headroom on a shared runner, local to 4. Kept, deliberately: static guards that protect security or packaging boundaries (preload/IPC surface, renderer token leak, noopener), the compiler-API dependency-boundary tests, and the rendered-geometry e2e specs — sidebar-geometry and scroll-geometry lock real regressions (the footer pushed off-screen at 60 sessions; the endless-scroll content-visibility bug) that no unit test can reach.
The first sweep matched `readFile(`, so nine files using `readFileSync` survived it. They are the same species — assertions about the text of the project's own source: - storybook-baseline-contract (21 tests) pinned story structure - onboarding-one-time-regression asserted that a source expression still reads `sessions.length === 0` - main-process-wiring-contract asserted main.ts imports each registrar - build-hygiene-contract asserted package.json still declares certain npm scripts - chip-converge, reactive-locale-foundation, form-consumer-ownership, startup-loading-shell, opencli (a version pin the lockfile already owns) @babel/parser was only there to parse source for one of them, so it leaves with them; knip is back to zero findings for apps/desktop. headless-storage-dependency stays: it uses the TypeScript compiler API to walk the real dependency graph rather than regexing text.
A third pass, this time reading the tests rather than pattern-matching
them. What turned up is the same species wearing a disguise: files that
import a real module (so the earlier scans cleared them) but whose
assertions are regexes over the project's own .tsx/.ts/.css text.
Deleted outright:
- icon-governance-contract (1029 lines) — banned inline <svg> and direct
lucide imports by scanning source. A lint rule.
- settings-roadmap-cleanup-contract — asserted that deleted code stayed
deleted ("must not keep the generic ComingSoon template"), including
extracting a function's source text to assert on its body.
- composer-mention-contract — matched the onTextareaKeyDown body out of
composer.tsx and compared indexOf positions to pin the ORDER of its if
branches. The behavior is covered end-to-end by
composer-skill-invocation.spec.ts.
- settings-app-info / settings-network / settings-theme /
daily-review-copy-feedback / voice-capture-smoke — all assert React
source text ("catch(...) { setInfoError(message) }") rather than
rendering anything.
- custom-relay-provider-contract, and three extraction contracts in core
and headless that pinned a refactor's file split in place ("the schema
lives in a leaf with no controller dependency"). One of them regexed a
harbor script knip already reports as unused.
Trimmed rather than deleted: session-project-view-contract kept its 11
rendering tests (project grouping, aliases, filters, lifecycle rows) and
lost the 5 that regexed sidebar.css and app-shell.tsx useMemo bodies.
Kept, again deliberately: every guard whose subject is a security or
credential boundary — the IPC surface pins, the renderer token-leak
gate, cleartext-key echo, the subscription experimental flag re-check in
main. Those assert source text too, but what they protect is not taste.
A 2-worker CI run failed exactly where parallel Electron windows share a resource that per-test userData dirs do not isolate: OS focus. composer-mode-indicator compares hover backgrounds and plan-reminders asserts toBeFocused(); Chromium blurs the document when another window steals activation, so both fail intermittently. CI goes back to 1 worker — green, and identical to today's behavior. Local keeps 4, which is where the dev loop feels the 7min → 2.4min difference. Hardening those two specs (bring the window to front before asserting hover/focus) is what unlocks parallel CI.
Two tests, both regexes over the Settings source asserting that the bot provider tables stay wired the same way. The tables themselves are typed and exported from @maka/core; a divergence is a type error, not a missing string match.
sync-model-metadata.test.mjs carried 25 provider tests with one shape: mkdtemp, build a catalog holding one provider, spawn a node process to run the sync script, assert the generated file contains that provider's id and one model's displayName. They exercise the same generic mapping path 25 times with different literals, at one process spawn each. Kept two as representatives — Cerebras for the plain provider shape, ZenMux for the creator/model id shape — plus every test that exercises a real branch: provider separation (Z.AI direct vs coding plan, GitHub Copilot vs GitHub Models), the multi-plan families (Tencent ×3, StepFun ×3, Xiaomi's three Token Plan regions), OpenCode protocol overrides, modality mapping, and the three rejection paths. 30 tests → 16, 1324 lines → 766, and 14 fewer process spawns per run.
Reading the small files turned up two species the scans had no signal for.
Barrel tautology — sandbox-export.test.ts imported the same seven symbols
through `../index.js` and `../sandbox/index.js` and asserted they were
`===`. The module system guarantees that, and a dropped re-export is a
compile error in this very file, so the test could never fail on its own.
Prompt-prose pins — swarm-mode.test.ts and graph-mode.test.ts each render
a prompt and assert it contains ten English phrases ("preferred default
execution strategy", "Do not manufacture parallelism"). They assert
nothing about behavior and break on any rewording, which is what prompts
get. Deleted both. Prompt tests that check interpolation stay, since
those verify the values a prompt carries rather than its prose —
session-environment-prompt asserting `Git branch: main` is a real test.
Source-regex tails, trimmed rather than deleted:
- project-context-badge kept the two tests that resolve real git
metadata in tmpdirs (normal checkout, worktree gitdir, upward root
walk) and lost the seven that regexed main.ts for exact expressions
like `fallbackRoots: () => [process.cwd(), app.getAppPath()]`.
273 lines → 54.
- attachment-frontend-contract had one test that renders a ChatView with
an image attachment; the other seven regexed source and CSS tokens.
Reduced to that one and renamed to attachment-chat-render, since it no
longer pins anything.
jackwener
force-pushed
the
chore/prune-static-contract-tests
branch
from
August 1, 2026 03:54
1577bdf to
1c32419
Compare
This was referenced Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second pass of the test cleanup, repo-wide this time. I classified all 811 test files by one question: does it execute the code it asserts about? Everything deleted here answers no.
Deleted (14 files, ~1,500 lines)
scripts/cu-*-contract.test.mjs/claude-sonnet-4-6/,/parallel_tool_calls: false/) — ran nothing, broke on any legitimate edit*-contract.test.tsprN-localized-copy-contract.test.ts+ helperai-sdk-compaction-contract.test.tsassertType<…>()calls — zero runtime assertions, cannot fail at runtime,tscalready owns itThe bilingual invariant the
prNfiles nominally guarded is still covered by the copy-catalog tests that actually call the catalogs (shell-copy,settings-preferences-copy, …), which assert no silent Chinese fallback in English.e2e
first-run.spec.tsasserted an onboarding registry screen that feat: add opencode-free as a zero-credential default provider #1720 removed from the default flow —opencode-freeis a zero-credential default, so a fresh install resolves toready_emptyand lands on the normal shell. Its 4 tests fail on current main. Reduced to the boot smoke its own docblock called the valuable part; the provider catalog it browsed is covered end-to-end byproviders.spec.ts.workers: 1on the theory that parallel Electron windows fight over screen/IPC. They don't — the fixturemkdtemps a userData dir per test. Measured on the full suite: 1 worker ≈ 7 min, 4 workers ≈ 2.4 min. CI goes to 2 for headroom on a shared runner (the suite keepsretries: 0), local to 4.Kept, deliberately — worth stating since the sweep was aggressive:
sidebar-geometry/scroll-geometrye2e — these lock real regressions (footer pushed off-screen at 60 sessions; the endless-scroll content-visibility bug) that no unit test can reachVerification
first-run.spec.ts+composer-toolbar-controls.spec.tspass locally after the rewrite (3 passed).e2ejob time versus the ~7 min baseline is the number to look at.Review focus
The
workers: 2choice for CI is a judgment call — if the runner has more cores to spare, raising it further is a one-line change with a proportional payoff.