Skip to content

test(storybook): prove the autoplay contract the smoke harness stands on - #2071

Merged
Astro-Han merged 1 commit into
apache:mainfrom
UncertaintyDeterminesYou4ndMe:test/storybook-play-contract
Aug 4, 2026
Merged

test(storybook): prove the autoplay contract the smoke harness stands on#2071
Astro-Han merged 1 commit into
apache:mainfrom
UncertaintyDeterminesYou4ndMe:test/storybook-play-contract

Conversation

@UncertaintyDeterminesYou4ndMe

Copy link
Copy Markdown
Contributor

Summary

The remaining item from #1981. The doc half was already fixed on main — FIDELITY.md's play section now correctly states that plays execute and a throwing play fails the smoke run. What was still missing is the guard the issue asked for: that failure path only exists while plays actually run. If a Storybook upgrade turned autoplay off, throwing plays would stop failing, silently.

A literally-throwing story can't ship (it would keep CI permanently red), so the contract is proven positively instead:

  • harness-play-contract.stories.tsx (Design System/Harness Contracts, exempt namespace): renders a data-play-proof="pending" node; its play flips it to executed.
  • A new play-executed product check in storybook-visual-smoke.mjs fails the run when the marker never flips, with a message that names the broken contract.
  • The surface joins REQUIRED_PRODUCT_SURFACES, so it cannot be switched off by deleting its manifest entry — the same durability rule the other required surfaces get.
  • FIDELITY.md's play section points at the contract.

Closes #1981.

Verification

Both directions, locally against a real build-storybook output:

  • Positive: smoke:storybook passes — Product Storybook smoke passed (68 manifest check(s), 73 catalog render(s)).
  • Negative (the actual acceptance): with the play sabotaged into a no-op (marker never flips, simulating an autoplay regression), the run exits 1 with [design-system-harness-contracts--play-executes @ wide] play function did not execute — the Storybook autoplay contract (FIDELITY.md) is broken. Restored and re-verified green after.
  • tsc -p tsconfig.storybook.json, biome format/lint, check-story-annotations all clean.

FIDELITY.md's play guidance rests on Storybook executing play functions
by default: a throwing play fails the smoke run only while plays run at
all. If an upgrade ever turned autoplay off, plays that no longer run
could no longer fail, and nothing in CI would notice (apache#1981's follow-up
after the doc itself was corrected).

Add the positive proof: a Design System harness story whose play flips
a DOM marker, and a `play-executed` manifest check that fails the smoke
run when the marker stays pending. The surface joins
REQUIRED_PRODUCT_SURFACES so it cannot be dropped by deleting its
manifest entry, and FIDELITY.md points at it.

Verified both directions locally: smoke passes with the contract
(68 checks), and a sabotaged no-op play fails the run with
"play function did not execute — the Storybook autoplay contract
(FIDELITY.md) is broken" (exit 1).

Closes apache#1981.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Two follow-ups for the record, neither blocking:

  • The guard is only as durable as its manifest entry: deleting the surface's checks field passes validateCoverageManifest, so the guard can vanish silently. A REQUIRED_PRODUCT_CHECKS map (surface to required checks) would pin the wiring the way REQUIRED_PRODUCT_SURFACES pins the surface.
  • The negative direction is verified manually only. The predicate lives inside a page.evaluate string; extracting it as an exported pure function would make it unit-testable in Node and catch a sabotaged predicate before it ships.

@Astro-Han
Astro-Han merged commit 54a8fe2 into apache:main Aug 4, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(storybook): correct FIDELITY play-function guidance

2 participants