test(storybook): prove the autoplay contract the smoke harness stands on - #2071
Merged
Astro-Han merged 1 commit intoAug 4, 2026
Conversation
FIDELITY.md's play guidance rests on Storybook executing play functions by default: a throwing play fails the smoke run only while plays run at all. If an upgrade ever turned autoplay off, plays that no longer run could no longer fail, and nothing in CI would notice (apache#1981's follow-up after the doc itself was corrected). Add the positive proof: a Design System harness story whose play flips a DOM marker, and a `play-executed` manifest check that fails the smoke run when the marker stays pending. The surface joins REQUIRED_PRODUCT_SURFACES so it cannot be dropped by deleting its manifest entry, and FIDELITY.md points at it. Verified both directions locally: smoke passes with the contract (68 checks), and a sabotaged no-op play fails the run with "play function did not execute — the Storybook autoplay contract (FIDELITY.md) is broken" (exit 1). Closes apache#1981.
Astro-Han
approved these changes
Aug 4, 2026
Astro-Han
left a comment
Contributor
There was a problem hiding this comment.
Approve. Two follow-ups for the record, neither blocking:
- The guard is only as durable as its manifest entry: deleting the surface's
checksfield passesvalidateCoverageManifest, so the guard can vanish silently. A REQUIRED_PRODUCT_CHECKS map (surface to required checks) would pin the wiring the way REQUIRED_PRODUCT_SURFACES pins the surface. - The negative direction is verified manually only. The predicate lives inside a
page.evaluatestring; extracting it as an exported pure function would make it unit-testable in Node and catch a sabotaged predicate before it ships.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The remaining item from #1981. The doc half was already fixed on
main— FIDELITY.md's play section now correctly states that plays execute and a throwing play fails the smoke run. What was still missing is the guard the issue asked for: that failure path only exists while plays actually run. If a Storybook upgrade turned autoplay off, throwing plays would stop failing, silently.A literally-throwing story can't ship (it would keep CI permanently red), so the contract is proven positively instead:
harness-play-contract.stories.tsx(Design System/Harness Contracts, exempt namespace): renders adata-play-proof="pending"node; itsplayflips it toexecuted.play-executedproduct check instorybook-visual-smoke.mjsfails the run when the marker never flips, with a message that names the broken contract.REQUIRED_PRODUCT_SURFACES, so it cannot be switched off by deleting its manifest entry — the same durability rule the other required surfaces get.Closes #1981.
Verification
Both directions, locally against a real
build-storybookoutput:smoke:storybookpasses —Product Storybook smoke passed (68 manifest check(s), 73 catalog render(s)).[design-system-harness-contracts--play-executes @ wide] play function did not execute — the Storybook autoplay contract (FIDELITY.md) is broken. Restored and re-verified green after.tsc -p tsconfig.storybook.json, biome format/lint,check-story-annotationsall clean.