test(cu): measure a rendering change against recorded trajectories, and name what a model got stuck on - #1884
Conversation
1ee9014 to
e4e32c0
Compare
|
Review findings addressed. Two of the three tools leave this branch, and the reason is the dependency ordering, so I will start there. On the dependency ordering cu-prune-eval.mjs and its test import packages/runtime/dist/computer-use-observation-text.js. I checked: that module exists on neither main, nor this branch, nor pr/cu-model-surface, nor any other open pull request. It is only on the unmerged feat/maka-cu-sole-executor. So the evaluator cannot be imported, which means the entire safety net the file's own header says it needs — the byte-for-byte re-render comparison, the round trip, the negative control — cannot execute. Wiring it into CI as the finding asks would only make CI red. Verified locally: node scripts/cu-prune-eval.test.mjs fails with ERR_MODULE_NOT_FOUND before a single test runs. Rather than ship 1,013 lines that cannot run behind a comment saying so, I have taken both files off this branch. They are preserved in this branch's parent commit e4e32c0 and will come back with the change that lands the renderer. That is the only honest option here: I cannot bring the renderer into a test-harness pull request without duplicating exactly the mistake this file is written around, which is an evaluator carrying its own copy of the policy it evaluates. cu-ax-oracle.swift moves to pr/cu-desktop-harness. It had no consumer on this branch; both of its callers are over there, and shipping a witness on one branch while the harness that depends on it sits on another is precisely why that harness was skipping every scenario and exiting 0. Stacking the two was the other option and it is worse, because it would block the desktop harness on this branch's unrelated renderer dependency. The two oracle findings — depth truncation and AXIsProcessTrusted — are fixed there. What is left here is cu-trace-analyse.mjs, which is self-contained, and it now has a test. Finding 13, confirmed and fixed blind was documented as a mutating action with no observation in front of it and implemented as a search of every preceding call for any observe. After the first observe in a run nothing could be flagged again. It is now no live observation of this target: a mutation retires the tree it acted on unless its own result carried a replacement, and an observation of one window says nothing about another. On a trajectory of one observe followed by four clicks the count goes from 0 to 3. abandoned was documented as within one call of a refusal and implemented as the last call was a refusal. Refusal detection hung on one regex over rendered prose. It now reads CuDebugRecord.error, the executor's own code written beside the result rather than inside it, so a rewording cannot zero it. The regex stays as a second source for older journals. A result that reads like a failure and yields no code from either is counted as unclassified and printed loudly rather than passed over as a success. Wired into test:scripts, so it runs on every change under scripts/ and in the full test run. 17 tests, each shape asserted twice — once on a trajectory that must trip it, once on a neighbour that must not. One thing added beyond the findings: a corpus in which no file carried a single call now exits 2. Nothing on main writes MAKA_CU_DEBUG_LOG, so that is the state this reports today, and it must not read as no problems found. Negative controls Each fix was reverted in place and the test rerun. blind reverted to any preceding observe: 2 failures — one observation does not excuse every click that follows it, an observation of one window says nothing about another. Restored: 17/17. abandoned reverted to the last call was a refusal: 1 failure — a turn that gave up one call after a refusal is abandoned. Restored: 17/17. Structural refusal field dropped so only the prose regex remains: 1 failure — a refusal is read from the executor field, not from how it was worded. Restored: 17/17. unclassified forced to false: 1 failure — a failure with no code anywhere is counted, not passed over. Restored: 17/17. An action's own fresh observation ignored, so the counter over-flags: 1 failure — an action that hands back a tree leaves the next one sighted. This is the control in the other direction, against a metric that only ever goes up. Restored: 17/17. Empty corpus: node scripts/cu-trace-analyse.mjs on an empty trace prints "no trajectory in any of these files carried a single Computer Use call" and exits 2. npm run test:scripts green. npx biome check clean on the touched files. node scripts/check-console.mjs passes. |
|
All five findings held up under measurement, and all five are fixed. The measurements are below, taken against the code as it stood. carriesObservation tested for observation_id= — a shell-style form with zero occurrences in this repository. The renderer emits JSON: observationText in packages/runtime/src/computer-use-tools.ts writes {"observation_id":"obs-1",...}, and persistedObservationText writes the same key. So the flag was false for every real observation. Measured: carriesObservation() === false, and a correct run of four clicks that each returned a fresh observation reported BLIND 3 where the test asserted 0. blindCalls reads that flag to decide whether an action handed back a replacement tree, so the counter was flagging the product working correctly. It now matches the JSON key, and the test asserts both branches of the real renderer — the fresh-observation form and the "Observation consumed; call observe before the next action" form. The MUTATING and OBSERVING regexes were a second copy of the wire enum and had drifted off it entirely, exactly as described. Matching neither list: left_click, right_click, left_mouse_down, left_mouse_up, left_click_drag, mouse_move, cursor_position, type, key, hold_key, wait, zoom. In the regexes and not on the wire: click, type_text, drag, scroll_element, element_sequence, window_action, launch_app, wait_for_text, wait_for_text_gone. Measured: twenty coordinate clicks with no observation at all reported BLIND 0, because left_click was neither observing nor mutating and the walk stepped over every one of them. Rather than correct the regexes, the vocabulary now comes from the product. @maka/core gains CU_TOOL_ACTION_TYPES — one list of every action the tool accepts — plus the observing/mutating partition and three predicates. The runtime's wire enum and its two lease predicates are built from that list instead of restating it, which is a straight de-duplication: the observing set is exactly the old requiresObservationLease chain and the mutating set is exactly the old requiresActionLease chain. The analyser imports the same list. An action added to the tool is now classified here the day it lands, and an action nobody has heard of is refused rather than counted. Runtime tests: 2703 pass, 0 fail. Core: 0 fail. classify defaulted a record it could not read to {} and '', and the only "nothing to analyse" guard was calls === 0. Measured: twenty records shaped {kind:'call', arguments:{…}, result:'failed: dispatch_refused'} reported {"calls":20,"refusals":0,"blind":0,"actions":["?"]} and exit 0 — a clean bill of health for a corpus it had not read one field of. Every field is now read strictly: a call line with no arguments object, no action name, an action not on the wire, or no result of any kind is returned with the reason attached, printed, and it makes the run exit 3. An unparseable line is counted rather than dropped, and a file that cannot be opened is distinguished from a file that is empty. signature() was key-order sensitive. Measured: three byte-identical calls emitted with the keys in three orders reported repeated: 0, thrash: ['click_element×3 shapes'] — verbatim repetition diagnosed as schema-guessing, which is the confusion the file's header claims to have fixed. It now sorts keys at every depth, and the test asserts both directions: three orderings of the same call are one repeat and no thrash, three genuinely different calls are still three shapes. On the fifth point, you are right and it is now said where it is read rather than only in the tail of main(). MAKA_CU_DEBUG_LOG and the CuDebugRecord shape have no producer anywhere — not on main, not on any open branch except these harness scripts, which only pass the variable through. The header says so, classify says so, and the exit-3 message says so, because the likeliest way this file will next be wrong is that the first writer to land disagrees with the shape read here. The negative control, since for a harness that is the deliverable. Reverting each fix one at a time in a scratch copy, and running the same test file against it: carriesObservation back to the observation_id= form vocabulary back to the two hand-written regexes classify back to defaulting a record it cannot read to {} and "" signature back to key-order-sensitive JSON.stringify parseJournal back to dropping unreadable lines silently Each fix has assertions that go red without it, and no fix is covered only by assertions that another fix would also break. Unmodified: 26 pass, 0 fail. End to end, on a corpus it can read: $ node scripts/cu-trace-analyse.mjs good.trace.jsonl and on one it cannot: $ node scripts/cu-trace-analyse.mjs alien.trace.jsonl $ node scripts/cu-trace-analyse.mjs absent.trace.jsonl Scope change, decided while this was in progress: scripts/cu-ax-oracle.swift moves here from the desktop-harness branch, so the analyser and the independent witness ship as one reviewable change. It walks the accessibility tree in its own process through the system framework, which is the only way to check an observation without asking the thing that produced it. It carries two fixes and one addition. A depth cut used to prune the subtree and leave "truncated": false, and every consumer of that field makes negative assertions off it. Real output, same app, same role filter, only the depth changed: $ swift scripts/cu-ax-oracle.swift com.apple.finder --depth 1 --role AXNothing $ swift scripts/cu-ax-oracle.swift com.apple.finder --depth 12 --role AXNothing Two empty element lists; one of them says it is empty because the walk stopped, and one says it is empty because the world is. Without Accessibility permission every attribute read fails silently, and the oracle would have emitted a well-formed {"window_count":0,"element_count":0,"truncated":false} — indistinguishable from an application with nothing in it. It now checks AXIsProcessTrusted() and says so. Negative control, with the guard forced to fire so that no TCC grant had to be touched: $ swift /tmp/oracle-untrusted.swift com.apple.finder $ swift scripts/cu-ax-oracle.swift com.apple.finder --role AXNothing The addition: every state the oracle cannot see through — screen_locked, not_permitted, not_running — now exits 1 instead of 0. It printed those as JSON and exited 0, so $ swift scripts/cu-ax-oracle.swift com.example.nope Gates: npx biome check clean on all five touched files, node scripts/check-console.mjs OK, npm run test:scripts 49 pass 0 fail, @maka/core 0 fail, @maka/runtime 2703 pass 0 fail. |
c5f7297 to
1ce25d9
Compare
…nd name what a model got stuck on Two tools and a witness, all of which exist because looking at a Computer Use run and forming an opinion about it does not work. `cu-prune-eval.mjs` replays recorded observations through the real renderer and reports what a pruning rule would have hidden. It imports `computer-use-observation-text.js` rather than carrying its own copy — twice before, an evaluator with its own copy of the rendering reached the opposite conclusion from the shipped one. Two negative controls ship with it: a rule that hides nothing and a rule that hides everything, so a result in between means the measurement ran. That is how the naive pruning rule was rejected. It read as an obvious win and it hid 1,023 elements a model could have acted on. `cu-trace-analyse.mjs` reads a run's tool calls and counts five shapes rather than success rates: | shape | what it means | |---|---| | repeated | the same call re-sent after reading the reply — the reply said nothing new | | thrash | one action tried in three argument shapes — guessing the schema, not reading the screen | | dead end | the same action re-sent after a refusal — the refusal did not say where to go | | blind | acting with no observation | | abandoned | the turn ends on a refusal | Each is measurable evidence that the tool surface is hard to use, rather than an opinion about a model. `cu-ax-oracle.swift` reads Accessibility directly, in its own process. A model sees the screen through the tool and reports through the same tool, so a click that did not land and a report saying it did are indistinguishable on that one path. The witness is the second path.
Review found three of the five shapes this file counts could not report a non-zero number, and one of them could go to zero without anyone noticing. `blind` was documented as "a mutating action with no observation in front of it" and implemented as a search of every preceding call for any observe at all. After the first observe in a run nothing could be flagged again: a trajectory that looked once and then fired twenty clicks reported BLIND 0. It is now "no live observation of this target" — a mutation retires the tree it acted on unless its own result carried a replacement, and an observation of one window says nothing about another. `abandoned` was documented as "the turn ended within one call of a refusal" and implemented as "the last call was a refusal", which misses the commonest shape of giving up: a refusal, one more attempt, and then nothing. Refusal detection hung on `/failed:\s*([a-z_]+)/` over rendered prose, so a wording change in how the executor renders an error would have zeroed every refusal count, every dead end and the whole failure-by-action table while the report still read like a clean run. It now reads `CuDebugRecord.error`, which is the executor's own code written beside the result rather than inside it. The regex stays as a second source for older journals, and a result that reads like a failure but yields no code from either is counted as `unclassified` and printed loudly rather than passed over as a success. A corpus in which no file carried a single call now exits 2. Nothing on main writes `MAKA_CU_DEBUG_LOG` yet, so that is the state this reports today, and it must not read as "no problems found". There was no test file at all. There is one now, wired into `test:scripts` so it runs on every change under `scripts/`. Each shape is asserted twice — once on a trajectory that must trip it, once on a neighbour that must not — because a counter that only ever goes up is not a measurement. Two files leave this branch. `cu-prune-eval.mjs` and its test import `packages/runtime/dist/computer-use-observation-text.js`, which does not exist on main, on this branch, or on any open pull request. The evaluator cannot be imported, so its safety net — the byte-for-byte re-render comparison, the round trip, the negative control — cannot run, and wiring it into CI would only make CI red. It is held for the change that lands the renderer. `cu-ax-oracle.swift` moves to the desktop-harness branch, where its callers are. It had no consumer here, and shipping a witness on one branch and the harness that depends on it on another is how that harness came to skip every scenario and exit 0.
The analyser was written against a Computer Use surface this repository does
not have, and every disagreement failed in the direction that reports a clean
run.
`carriesObservation` looked for `observation_id=`. The renderer emits JSON:
`observationText` in packages/runtime/src/computer-use-tools.ts writes
`{"observation_id":"obs-1",...}`, and the `=` form has zero occurrences on
main. So the flag was false for every real observation, and `blindCalls` reads
that flag to decide whether an action handed back a replacement tree. Measured
against real renderer output, a correct run of four clicks that each returned
a fresh observation reported BLIND 3. The counter was flagging the product
working.
The MUTATING and OBSERVING regexes were a second copy of the wire enum, and
they had drifted off it entirely. Matching neither: left_click, right_click,
left_mouse_down, left_mouse_up, left_click_drag, mouse_move, cursor_position,
type, key, hold_key, wait, zoom. In the regexes and not on the wire: click,
type_text, drag, scroll_element, element_sequence, window_action, launch_app,
wait_for_text, wait_for_text_gone. A trajectory of twenty coordinate clicks
with no observation at all measured BLIND 0, because `left_click` was neither
observing nor mutating and the walk stepped over every one of them.
So the vocabulary now comes from the product. `@maka/core` gains one list of
the wire actions and the observing/mutating partition, the runtime's wire enum
and its two lease predicates are built from that list instead of restating it,
and the analyser imports it. Adding an action to the tool now classifies it
here the day it lands, and an action nobody has heard of is refused rather
than counted.
`classify` defaulted a record it could not read to `{}` and `''`, and the only
guard was `calls === 0`. Twenty records shaped `{kind:'call', arguments:{…},
result:'…'}` — arguments under the wrong key, result under the wrong key,
action not on the wire — reported `{"calls":20,"refusals":0,"blind":0,
"actions":["?"]}` and exit 0: a clean bill of health for a corpus it had not
read one field of. Every field is now read strictly, an unparseable line is
counted rather than dropped, a file that cannot be opened is distinguished
from a file that is empty, and any of the three makes the run exit 3.
That guard matters more than it looks, because nothing writes this journal.
`MAKA_CU_DEBUG_LOG` and the `CuDebugRecord` shape have no producer anywhere in
this repository — not on main, not on any open branch. The record shape read
here has therefore never been checked against a writer, and the first writer
to land will disagree with it somehow. The header says so, and the disagreement
now exits non-zero instead of printing zeroes.
`signature()` was key-order sensitive, so three byte-identical calls emitted
with the keys in three orders reported `repeated: 0, thrash: click_element×3
shapes` — verbatim repetition diagnosed as schema-guessing, which is the exact
confusion the file's header claims to have fixed.
The test file's fixtures were the same guesswork: `observation_id=obs_1` and
calls with no result at all. They are now the renderer's own output, both
branches of it — the fresh-observation form and the "Observation consumed"
form. Each fix is negative-controlled: reverting it one at a time turns
exactly the assertions that cover it red, and nothing else.
Also brings scripts/cu-ax-oracle.swift over from the desktop-harness branch,
so the analyser and the independent witness ship together. It walks the
accessibility tree in its own process through the system framework, which is
the only way to check an observation without asking the thing that produced it.
Two states it used to report as an ordinary empty world: a depth cut left
`truncated` false, and every consumer of this file makes negative assertions
off that field; and without Accessibility permission every attribute read fails
silently, which produced a well-formed `{"window_count":0,"element_count":0}`.
Both are now distinct, and every state the oracle cannot see through exits 1,
so a caller cannot walk on with a file that says the tree is empty because the
oracle was blindfolded.
1ce25d9 to
d2065b8
Compare
1 participant
Three tools that exist because looking at a Computer Use run and forming an opinion about it does not work.
cu-prune-eval.mjs— try a rendering change offline before a real machine sees itReplays recorded observations through the real renderer and reports what a pruning rule would have hidden.
It imports
packages/runtime/dist/computer-use-observation-text.jsrather than carrying its own copy of the rendering. Twice before, an evaluator with its own copy reached the opposite conclusion from the shipped renderer.Two negative controls ship with it — a rule that hides nothing, a rule that hides everything — so a result in between means the measurement actually ran.
This is how the naive pruning rule was rejected: it read as an obvious win, and it hid 1,023 elements a model could have acted on.
cu-trace-analyse.mjs— count shapes, not success ratesrepeatedthrashdead endblindabandonedEach is measurable evidence that the tool surface is hard to use, rather than an opinion about a model.
cu-ax-oracle.swift— an independent witnessA model sees the screen through the tool and reports through the same tool. On that one path, a click that never landed and a report saying it did are indistinguishable. This reads Accessibility directly, in its own process, so there is a second path to check against.
Verification
node scripts/cu-prune-eval.test.mjs— 8/8, run locally against current main. These are plain scripts with no workspace build dependency beyondpackages/runtime/dist.