feat: add 21 execution tests for the WP 7.0 API surfaces (abilities, connectors, ai-client) - #46
Merged
Merged
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
This was referenced Aug 4, 2026
lezama
added a commit
that referenced
this pull request
Aug 4, 2026
## What CI's "Python lint, types, tests" job went red on **every open PR** (#45, #46) over files those PRs never touched. Cause: the dev extra declares `ruff>=0.6` unpinned, CI installs latest on each run, and ruff 0.16 changed its default rule set. ## Fix - **Pin `ruff>=0.16,<0.17`** so lint results are deterministic; future default-set changes land as deliberate bumps, not surprise CI breakage. - **Apply the 0.16 autofixes** (import sorting, modern typing syntax) across the harness — mechanical, no behavior change (26 files, all green under pytest/mypy). - **Exempt `typer.Option` from B008** (`flake8-bugbear.extend-immutable-calls`): typer's call-in-default pattern is its documented API. - **Fix the four genuine findings** the new rules caught: useless conditional in `cli.py`, `dict()` call → literal in a test, result-file timestamps now explicitly UTC (timezone-independent artifact names), and an intentional broad `except` in cost estimation annotated with its rationale. ## Verification `ruff check python` clean under 0.16.1, `mypy python` clean, 152 tests pass. Once merged, #45 and #46 just need a CI re-run (the lint job runs on the merge ref) — no rebase strictly required.
lezama
force-pushed
the
feat/expand-wp70-api-tests
branch
from
August 4, 2026 20:59
ab26228 to
e37fdc5
Compare
…connectors, ai-client) Expands the three weakest-covered new-API categories, where the first full runs showed models failing hardest: - abilities-api 005-014: execute() pipeline stages (permission denial, input/output schema validation, callback exceptions and their error codes), registration with schema defaults, unregistration state, category meta, enumeration by category, core ability fixtures, and the wp_register_ability_args filter. - connectors-api 004-013: registration arg shapes (auto-generated setting names, method-none field stripping), guarded mutations that must avoid _doing_it_wrong notices, the unregister/re-register override recipe, ai_provider gating via wp_supports_ai, live-count and by-type enumeration, and defensive arg validation. - ai-client 006-015: the default-timeout filter, prevention-filter anatomy (clone argument, 503 status, both prevented messages, skip when AI is disabled), sticky builder error state, support-check non-poisoning, bool casting of wp_supports_ai, using_abilities, and the ability function resolver's allow-list codes. Every test follows the hardened assertion patterns from #43: filter and action spies prove the mechanism was exercised, multi-fixture contracts make constant returns fail, and state is inspected independently of the function's own return value. All snippets authored against the actual WP 7.0 sources in the runtime image. Verified live against the WP 7.0 runtime: --check-reference-solution passes 30/30. Suite files bump their versions; the execution count pin moves to 180. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Drop the six tests that re-measured contracts already covered (connectors 005/007/012, ai-client 012/014, abilities 010): duplicate signal inflates category scores without adding discrimination. The batch lands at 24 tests; ids keep their numbers, gaps documented. - Remove the eight dead connectors teardown blocks (the registry is in-memory and dies with the per-test PHP request; matches the file's own 001-003 convention) plus the $GLOBALS snapshot plumbing in 010 that existed only to feed one. - Drop the three static required_patterns that sat below API altitude (preg_match, is_callable, instanceof): each rejected legitimately different correct implementations the runtime assertions already grade behaviorally. Also drop the str_word_count requirement wording that leaked the reference implementation. - Point fixes: consistent instanceof guard in abilities-009, redundant clause in connectors-013, subsumed remove_filter in ai-client-006, cheaper has_filter cleanup check in ai-client-008. - Consolidate incidental fixture category slugs onto wpbp-fixtures / wpbp-ai-fixtures where the slug is not part of the test contract. Re-verified live after the edits: --check-reference-solution 15/15 on every touched test, --check-exploits 0 exploitable (13 auditable; the hook-based tests have no gateway function for the generic battery and are the target case for #45's authored exploit_solutions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ds 1-3) Two review rounds against codex surfaced plausible-cheat paths beyond the generic exploit battery; each was either closed with a stronger observable or the test was dropped as unverifiable: - Dropped e-ai-client-008/009/013: their knowledge (clone argument identity, prevention-filter skip while disabled, ability attachment) is not observable from assertion code, so any clean builder passed. - ai-client-011/015 now return the builder and the assertion distinguishes poisoned from clean builders by generating under an active prevention filter: the sticky error is checked before the prevention gate in __call, so a poisoned builder surfaces prompt_builder_error while a clean one surfaces prompt_prevented. - ai-client-006 narrowed to the filter contract (void) with two-value probes on runtime-owned builders, eliminating both the hardcode hole and a same-priority spy collision that failed correct solutions. - ai-client-007 gained construction and observed-disabled spies. - connectors-008/011 assertions inspect the stored connector data; 011 adds uppercase-only and period ID probes; 013 adds an invalid authentication-method probe and narrows its no-notice promise to the probed paths; 009's fixture callback reads a mutable global so both toggle states on the same connector force real invocation. - abilities-007 asserts the schema type is exactly string. Batch lands at 21 tests (execution pin 171). Live-verified after every round: --check-reference-solution 1.0 and --check-exploits 0 exploitable on all edited tests. Codex final verdict: ship. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generic exploit battery cannot target tests without a gateway function; #45's exploit_solutions field exists exactly for them. Adds two verified wrong-variant cheats to each hook-based test in this batch (abilities 007/009, connectors 004): hardcoded greeting and callback-side default for the schema-default test, missing meta and wrong label for the category test, explicit setting_name and wrong auth method for the auto-setting-name test. --check-exploits now covers all three (previously not auditable): 0 exploitable, every authored cheat fails its assertions live. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lezama
force-pushed
the
feat/expand-wp70-api-tests
branch
from
August 4, 2026 21:11
e37fdc5 to
508d371
Compare
Second cleanup sweep over the post-rebase diff. Efficiency, reuse, and altitude angles came back clean (single-assertion remains the dataset convention at 91%, the exploit format matches #45, the 185 pin is verified); the simplification angle found four residuals: - connectors-011: drop the four dead post-state checks (the sandbox turns any _doing_it_wrong into a failure before they can decide) and probe 'wpbp invalid' so the space case is isolated from uppercase. - ai-client-011: drop the fluency clause; fluent methods return the builder instance whether or not it is errored, so the check could never distinguish solutions. - ai-client-007 and connectors-013: expected_behavior drift fixed to match what the final assertions enforce. Kept knowingly: the akismet probe in connectors-009 (exercises a real core connector; the runtime stably ships it inactive). Live re-verified the two edited assertions: references 1.0, exploit audit 0 exploitable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Expands the three weakest-covered WP 7.0 API categories — exactly where the first full benchmark runs showed models failing hardest (ai-client 1/5, abilities-api 1/4 execution pass):
Execution suite: 150 → 171 tests (count pin updated).
What the new tests measure
execute()pipeline stage by stage — permission denial, input/output schema validation, callback exceptions — each surfacing its distinct error code; registration with schema defaults on the correct init action; unregistration state; category meta; enumeration by category; the built-in core abilities; thewp_register_ability_argsfilter.nonefield stripping); the unregister/re-register override recipe;ai_providergating viawp_supports_ai; and a theme the first runs showed models failing: knowing which inputs the registry rejects with_doing_it_wrongand guarding before calling it.Authoring method
Written against the actual WP 7.0 sources in the runtime image (exact signatures, hook timings, error codes — e.g. abilities registration is only valid during
wp_abilities_api_init, callbacks receive input only when aninput_schemaexists, the connector registry initializes duringinitso tests re-firewp_connectors_init).Every assertion follows the #43 hardening patterns: filter/action spies prove the mechanism actually ran, multi-fixture contracts give different expected outputs per probe, and state is inspected independently of the function's return. Two findings from this batch worth stealing for future tests:
wp_ai_client_default_request_timeoutproves a real prompt builder was created (it fires on every construction).__call, so generating under an active prevention filter cleanly distinguishes a poisoned builder (prompt_builder_error) from a clean one (prompt_prevented).Review process
Started at 30 tests; two adversarial review rounds (independent reviewer prompted to find plausible-cheat paths) plus a 4-angle cleanup pass cut it to 21: six re-measured contracts existing tests already grade, and three had knowledge that is simply unobservable from assertion code (any clean builder passed). Dropped rather than shipped weak — duplicate signal inflates category scores without discriminating, and the goal is scores that go down. Dropped ids leave gaps (005/007/012 in connectors, 008/009/012/013/014 in ai-client, 010 in abilities); ids are stable identifiers so the survivors keep their numbers.
Verification (live WP 7.0 runtime)
--check-reference-solution: 1.0 across the batch (re-run after every review round)--check-exploits: 0 exploitable; the hook-based tests (notest_function) are not coverable by the generic battery — they are the target case for Add gb-block-markup execution category (14 tests) and wire authored exploit_solutions #45's authoredexploit_solutions, and I'll add authored cheats to them once Add gb-block-markup execution category (14 tests) and wire authored exploit_solutions #45 landspytest152,ruff,mypygreen; all snippets PHP-lintedCoordination with #45
Both PRs move the execution count pin (this one 150→171, #45 150→164). Whichever lands second rebases; final total 185. Happy to rebase this one after #45 merges.
🤖 Generated with Claude Code