fix: harden the 8 assertion sets flagged exploitable by --check-exploits - #43

lezama · 2026-07-17T16:43:01Z

What

Hardens the 8 execution tests that the #41 exploit audit (--check-exploits) flagged as exploitable: a zero-effort constant return (return true;, return 1;, return false;) satisfied their runtime assertions.

All 8 shared the same root cause: the assertion graded the function's return value against a single fixed expectation instead of observing WordPress state, so it could not tell a real implementation from a hard-coded answer.

How each test is hardened

Every assertion now verifies behavior a constant cannot fake:

Test	Old weakness	New verification
`e-ai-client-001`	`return false` passed	Counting spy on the `wp_supports_ai` filter proves the support state is actually read during the call, plus the existing re-enabled-after check
`e-ai-client-005`	`return false` passed	Late spy (`PHP_INT_MAX`) on `wp_ai_client_prevent_prompt` proves prevention is observed active during the support check, and inactive after
`e-comments-users-004`	`return 1` passed	Second fixture post with zero comments; expected counts become `{2, 0}`, so no single constant passes both calls
`e-connectors-api-001`	`return true` passed	Helper is called twice: with Akismet registered (expects `true`) and temporarily unregistered via `WP_Connector_Registry::unregister()` (expects `false`), then restored
`e-post-types-taxonomy-001`	assertion returned the function's own `$ok`	Inspects `get_post_type_object( 'wpbp_book' )`: exists, `public`, `show_in_rest`
`e-post-types-taxonomy-002`	same	Inspects `get_taxonomy( 'wpbp_genre' )`: exists, `hierarchical`, attached to `post`
`e-post-types-taxonomy-003`	assertion was literally `return f();`	Fixtures (`wpbp_movie`, `wpbp_topic`) move to runtime `setup`/`teardown`; assertion checks the actual attachment with `is_object_in_taxonomy()`
`e-roles-caps-002`	`return true` passed	Spy on `updated_option` for the roles option observes `wpbp_temp_cap` actually being added to Subscriber before the final not-present check

Design decisions

Slugs moved into prompts. Models only see prompt + requirements + test_function (never expected_behavior), so state-based assertions on wpbp_book / wpbp_genre / wpbp_topic / wpbp_movie require the prompts to name them. Done for the three post-types tests.
Dropped the "Clean up registered types when possible" requirement on e-post-types-taxonomy-001/002. A post-call state inspection requires the registration to still exist when the assertion runs; cleanup is now the assertion's job (as the reference solutions already assumed).
e-connectors-api-001 keeps its contract (no-arg helper, akismet required pattern, WP_Connector_Registry forbidden for the model). The registry-toggle happens in assertion code, which static checks don't apply to. A lazy-init spy on wp_connectors_init was tried first but the registry initializes during init, before assertions run.
e-connectors-api-002-style negative cases were preferred over spies wherever the API allowed it; spies are only used where the observable is a filter/option side effect (ai-client, roles-caps).
Each touched suite file bumps 2.1.0 → 2.2.0, same convention as Add test_function to execution tests #24.

Verification (live WP 7.0 runtime)

Both directions of the audit loop, against the real wp-env grader:

--check-reference-solution on the 8 tests: 8/8 pass (correctness 1.0)
--check-exploits on the 8 tests: 0/8 exploitable
Full pytest suite: 151 passed; dataset validation green
--check-exploits on the full suite: 0/140 auditable tests exploitable (was 8/140 at the feat: add --check-exploits adversarial assertion audit #41 baseline). Full end-to-end run: 1120 exploit-candidate executions, each against a freshly reset WordPress, ~1.5h wall clock. The 10 non-auditable tests have no test_function or plugin artifact for the generic battery to target, unchanged from the baseline.

🤖 Generated with Claude Code

The #41 exploit audit found 8/140 execution tests where a zero-effort constant return satisfies the runtime assertions. Every one graded the function's return value against a single fixed expectation instead of observing WordPress state, so the assertions could not tell a real implementation from a hard-coded answer. Each assertion now verifies behavior a constant cannot fake: - e-ai-client-001: a counting spy on the wp_supports_ai filter proves the support state is actually read during the call. - e-ai-client-005: a late spy on wp_ai_client_prevent_prompt proves prevention is observed active during the support check. - e-comments-users-004: a second fixture post with zero comments makes the expected counts {2, 0}, so no single constant passes. - e-connectors-api-001: the helper is called twice, with Akismet registered and temporarily unregistered, and must reflect live registry state (restored afterward). - e-post-types-taxonomy-001/002: assertions inspect the registered post type/taxonomy object (public, show_in_rest, hierarchical, attached to post) instead of trusting the function's return. - e-post-types-taxonomy-003: fixtures move to runtime setup/teardown and the assertion checks is_object_in_taxonomy() for the actual attachment. - e-roles-caps-002: a spy on updated_option observes wpbp_temp_cap actually being added to the Subscriber role before its removal. Slugs the assertions now depend on (wpbp_book, wpbp_genre, wpbp_topic, wpbp_movie) are stated in the prompts, since models never see expected_behavior. Each touched suite file bumps to 2.2.0. Live-verified against the WP 7.0 runtime in both directions: --check-reference-solution passes 8/8 and --check-exploits reports 0/8 exploitable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

github-actions · 2026-07-17T16:50:00Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: lezama <migueluy@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

lezama · 2026-07-17T20:28:01Z

full suite re-audit finished clean: 0/140 auditable tests exploitable (1120 exploit executions against the live WP 7.0 runtime, each on a fresh reset). updated the PR body with the final numbers 🎉

…connectors, ai-client) Expands the three weakest-covered new-API categories, where the first full runs showed models failing hardest: - abilities-api 005-014: execute() pipeline stages (permission denial, input/output schema validation, callback exceptions and their error codes), registration with schema defaults, unregistration state, category meta, enumeration by category, core ability fixtures, and the wp_register_ability_args filter. - connectors-api 004-013: registration arg shapes (auto-generated setting names, method-none field stripping), guarded mutations that must avoid _doing_it_wrong notices, the unregister/re-register override recipe, ai_provider gating via wp_supports_ai, live-count and by-type enumeration, and defensive arg validation. - ai-client 006-015: the default-timeout filter, prevention-filter anatomy (clone argument, 503 status, both prevented messages, skip when AI is disabled), sticky builder error state, support-check non-poisoning, bool casting of wp_supports_ai, using_abilities, and the ability function resolver's allow-list codes. Every test follows the hardened assertion patterns from #43: filter and action spies prove the mechanism was exercised, multi-fixture contracts make constant returns fail, and state is inspected independently of the function's own return value. All snippets authored against the actual WP 7.0 sources in the runtime image. Verified live against the WP 7.0 runtime: --check-reference-solution passes 30/30. Suite files bump their versions; the execution count pin moves to 180. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…connectors, ai-client) (#46) ## What Expands the three weakest-covered WP 7.0 API categories — exactly where the first full benchmark runs showed models failing hardest (ai-client 1/5, abilities-api 1/4 execution pass): | Category | Before | Added | After | |---|---|---|---| | abilities-api | 4 | +9 | 13 | | connectors-api | 3 | +7 | 10 | | ai-client | 5 | +5 | 10 | Execution suite: 150 → **171** tests (count pin updated). ## What the new tests measure - **abilities-api 005-014** (010 dropped in review): the `execute()` pipeline stage by stage — permission denial, input/output schema validation, callback exceptions — each surfacing its distinct error code; registration with schema defaults on the correct init action; unregistration state; category meta; enumeration by category; the built-in core abilities; the `wp_register_ability_args` filter. - **connectors-api 004-013** (005/007/012 dropped): registration arg shapes (auto-generated setting names, method-`none` field stripping); the unregister/re-register override recipe; `ai_provider` gating via `wp_supports_ai`; and a theme the first runs showed models failing: knowing which inputs the registry rejects with `_doing_it_wrong` and guarding before calling it. - **ai-client 006-015** (008/009/012/013/014 dropped): the default-timeout filter; prevention-filter anatomy; sticky builder error state; support-check non-poisoning; the ability function resolver's allow-list codes. Everything is provider-free by construction (prevention filters or support toggles short-circuit before any network call). ## Authoring method Written against the actual WP 7.0 sources in the runtime image (exact signatures, hook timings, error codes — e.g. abilities registration is only valid during `wp_abilities_api_init`, callbacks receive input only when an `input_schema` exists, the connector registry initializes during `init` so tests re-fire `wp_connectors_init`). Every assertion follows the #43 hardening patterns: filter/action spies prove the mechanism actually ran, multi-fixture contracts give different expected outputs per probe, and state is inspected independently of the function's return. Two findings from this batch worth stealing for future tests: - A builder-construction spy on `wp_ai_client_default_request_timeout` proves a real prompt builder was created (it fires on every construction). - The sticky builder error is checked **before** the prevention gate in `__call`, so generating under an active prevention filter cleanly distinguishes a poisoned builder (`prompt_builder_error`) from a clean one (`prompt_prevented`). ## Review process Started at 30 tests; two adversarial review rounds (independent reviewer prompted to find plausible-cheat paths) plus a 4-angle cleanup pass cut it to **21**: six re-measured contracts existing tests already grade, and three had knowledge that is simply unobservable from assertion code (any clean builder passed). Dropped rather than shipped weak — duplicate signal inflates category scores without discriminating, and the goal is scores that go **down**. Dropped ids leave gaps (005/007/012 in connectors, 008/009/012/013/014 in ai-client, 010 in abilities); ids are stable identifiers so the survivors keep their numbers. ## Verification (live WP 7.0 runtime) - `--check-reference-solution`: **1.0 across the batch** (re-run after every review round) - `--check-exploits`: **0 exploitable**; the hook-based tests (no `test_function`) are not coverable by the generic battery — they are the target case for #45's authored `exploit_solutions`, and I'll add authored cheats to them once #45 lands - `pytest` 152, `ruff`, `mypy` green; all snippets PHP-linted ## Coordination with #45 Both PRs move the execution count pin (this one 150→171, #45 150→164). Whichever lands second rebases; final total 185. Happy to rebase this one after #45 merges. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

lezama merged commit 1e5c0f7 into trunk Jul 17, 2026
5 checks passed

lezama deleted the fix/harden-exploitable-assertions branch July 17, 2026 20:29

lezama mentioned this pull request Aug 4, 2026

feat: add 21 execution tests for the WP 7.0 API surfaces (abilities, connectors, ai-client) #46

Merged

Navigation Menu

Uh oh!

fix: harden the 8 assertion sets flagged exploitable by --check-exploits - #43