[experimental] A language-agnostic test runner built on Pkl. Generalizes the retry / sharding / retry-on-fail machinery that playwright test provides for one ecosystem to any kind of test — shell, HTTP, browser, SQL, and whatever you teach the runner next. First-class support for spec-driven authoring, property-based testing, fuzzing, snapshot, and differential testing across language implementations.

Start with the quick-start guide:

• Quick Start
• クイックスタート

amends "./pkspec/Test.pkl"

tests {
  new {
    name = "login_smoke"
    specRef { "LOGIN-001" }
    steps {
      new { http = new HttpRequest { url = "http://localhost/login"; method = "POST"; body = "..." }
            expectStatus = 200 }
      new { name = "judge_message"
            http = new HttpRequest { url = "http://localhost/login/welcome" }
            expectAi = new AiAssertion {
              prompt = "the response acknowledges the user in English"
              cmd = "claude --no-stream"
              snapshotName = "login-welcome"
            } }
    }
  }
}

pkspec exec -f Test.pkl --shard=2/4         # 4-way history-balanced split
pkspec exec -f Test.pkl --rerun-failed      # only previously failed tests
pkspec timings -f Test.pkl --shard=2/4      # preview the shard without running
pkspec spec tests/**/*.pkl                  # render SPEC.md from Scenario tags

Tests are typed values, not bash scripts or YAML. Pkl gives the schema:

• Static checks at author time: a Test with both cmd and steps is rejected before the runner ever starts.
• Composition: a step body is just a Step value — reuse it across scenarios, parameterize it via Pkl import, generate it from a property-based input.
• Language-independent: the schema lives in pkl/, the runner in Go (cmd/pkspec/). Low-level step kinds can still be Go executors, while whole native runners are described by the Pkl adapter DSL so every ecosystem does not become a hard-coded core dependency.
• Reusable with pkl test: pkspec rendered output is a Pkl module; Pkl's own facts / examples / snapshot machinery still applies, and pkspec run wraps pkl test so its unreliable exit code becomes CI-trustworthy.

The features playwright test ships as built-ins (--retries, --shard=K/N, last-failed re-runs) generalized to every kind:

Sharding uses an append-only .pkspec/timings.jsonl history, median of the most recent 5 runs per test, Longest-Processing-Time bin-packing with deterministic tie-breaking. The same input produces the same shard assignment on every machine. See docs/notes/timing-shard.md, including the GitHub Actions matrix recipe.

A new low-level Step kind is three things:

1. a Pkl class on the Step (<Kind>Spec)
2. a Go executor under internal/executor/<kind>.go
3. a value for Step.kind (the computed discriminator that drives dispatch)

See docs/notes/runner-design.md for the architectural sketch, and the per-kind notes: playwright / playwright-test / http-dsl / cassettes / sql / shell output assertions.

For existing native runners, prefer the Pkl adapter DSL over adding one Go executor per ecosystem. pkl/Adapter.pkl defines an abstract Adapter, and built-ins are ordinary Pkl subclasses:

• pkl/adapters/Vitest.pkl
• pkl/adapters/Playwright.pkl
• pkl/adapters/NodeTest.pkl
• pkl/adapters/GoTest.pkl
• pkl/adapters/MoonTest.pkl

Projects select and specialize adapters with extends:

amends "./pkspec/Adapter.pkl"

import "./pkspec/adapters/Vitest.pkl" as Vitest

local class WebVitest extends Vitest.Vitest {
  configPath = "packages/web/vitest.config.ts"
  include = new { "src/**/*.test.ts" }
}

suites {
  new {
    name = "web-unit"
    adapter = new WebVitest {}
    overlays {
      ["src/parser.test.ts::empty input"] = new CaseOverlay {
        specRef { "parser.empty" }
      }
    }
  }
}

Run adapter modules with pkspec adapter -f Adapter.pkl. The runtime executes the generic protocol (discover JSON, manifest run, JSONL events) and post-run coverage collectors. Native shim commands are installed as pkspec-adapter-vitest, pkspec-adapter-playwright, pkspec-adapter-node-test, pkspec-adapter-go-test, and pkspec-adapter-moon-test; built-in adapters select those commands from Pkl instead of a Go registry. See docs/notes/adapters.md.

Three layers, from low to high:

Test.pkl (low) — declare concrete subprocess / HTTP / browser invocations with explicit expectations.

Spec.pkl (mid) — BDD-style Given / When / Then scenarios that desugar to Tests. A scenario tagged spec with an empty body is auto-pending — the description is the spec, the body lands later without renaming the test. pkspec spec renders Markdown SPEC.md from the scenarios.

expectAi (orthogonal) — fuzzy natural-language assertions on response bodies, delegated to an external judge command (typically an LLM wrapper). The verdict is cached by sha256(prompt + body) under .pkspec/ai-snapshots/; identical inputs reuse the cached verdict and never spawn the judge.

expectAi = new AiAssertion {
  prompt = "the response acknowledges the user in English"
  cmd = "claude --no-stream"
  snapshotName = "greeting-acknowledges-user"
}

Spec knowledge graph + Goals — Spec.pkl scenarios are nodes in a graph: each carries a stable id, lifecycle (reviewStatus draft/review/approved + deprecated), severity, edges (dependsOn / supersedes / replacedBy / parent for sub-specs), an append-only decision log, and a list of open questions. Goals are sibling user-value statements with no test of their own; scenarios point at them via contributes. Test.pkl implements scenarios via specRef. The runner prints (verifies AUTH-001) on each test line, and the default pkspec spec Markdown includes a per-spec implementation index so reviewers can scan Scenario.id back to active tests or implementedAt pointers. The review/CI surface is exposed as top-level commands:

• pkspec check — CI gate: exit 1 on any non-draft non-deprecated spec without an implementing test
• pkspec coverage — declared vs implemented, broken down by severity and review-status
• pkspec graph — graphviz dot of the knowledge graph, including test/code/doc implementation backlinks
• pkspec decisions — newest-first Markdown decision log
• pkspec goals — Goals listed by priority with per-Goal coverage
• pkspec milestones — release/planning Milestones with Goal progress rollups
• pkspec next — unimplemented specs ranked by Goal priority + severity ("what to work on next")
• pkspec implementations — the reverse index only: spec id → active tests / code / doc pointers
• pkspec orphans — active tests that still need a specRef
• pkspec lint — broken/deprecated spec links and authoring invariants
• pkspec docs --audience X — audience-specific Markdown projection from audience { "X" } or audience:X tags, with implementation details hidden by default

// Spec.pkl
goals {
  new Goal {
    id = "GOAL-SECURE-AUTH"
    name = "users can authenticate securely"
    priority = 90
    reviewStatus = "approved"
  }
}

scenarios {
  new {
    id = "AUTH-001"
    name = "valid credentials"
    severity = "critical"
    reviewStatus = "approved"
    contributes { "GOAL-SECURE-AUTH" }
    dependsOn { "SESSION-001" }
    decisions {
      new Decision {
        date = "2026-03-01"
        author = "mizchi"
        summary = "lock the spec to cookie-based auth"
      }
    }
    tags { "spec" }
  }
  new {
    id = "AUTH-001a"
    name = "valid credentials happy path"
    parent = "AUTH-001"      // sub-spec refines the parent
    contributes { "GOAL-SECURE-AUTH" }
    tags { "spec" }
  }
}

// Test.pkl
new Test { name = "login_happy_path"; specRef { "AUTH-001" }; cmd = "..." }

Shared setup across scenarios uses the top-level prelude (Cucumber Background:). See docs/notes/spec.md / docs/notes/ai-assertion.md / docs/notes/spec-id.md / docs/notes/spec-graph.md and examples/spec-graph/. For project-level local gates and task-runner contracts, see docs/notes/project-gates.md. For advanced Goal progress methods and Milestone rollups, see docs/advanced/goals-and-milestones.md.

• QuickCheck-style PBT — Test.iterations, Test.inputs (abstract Input schema with concrete IntInput, …), seed-deterministic generation in Pkl, input-space shrinking in Go. Works with every kind, so generated inputs can drive a shell cmd, an HTTP body, or a SQL parameter. See docs/notes/quickcheck.md.

• Snapshot testing — reference bytes under .pkspec/snapshots/<name>.bytes, written on first run, committed to git. Inline snapshots (inlineStdout) get rewritten in-place via --update-inline-snapshots. Mid-port, the reference implementation IS the spec — pkspec runs it, captures the bytes, asserts every port matches. See docs/notes/snapshots.md.

• Differential testing across language implementations — two or more impls of the same spec, the same input, the same expected bytes. Snapshots make this trivial: capture from the reference once, every port must match.

Beyond the per-test plumbing:

• Hooks — before { all { ... }; each { ... } } and after, scoped (all / each), LIFO for after, with stdout-capture into env vars. See docs/notes/hooks.md.
• Backgrounds — long-running auxiliary processes with readyProbe and optional portEnv for dynamic-port allocation.
• Ephemeral workdirs — Test.ephemeralWorkdir = true for an auto-temp dir cleaned at test exit.

pkspec is distributed as prebuilt MoonBit binaries — the pkspec CLI plus its five adapter shims (pkspec-adapter-{vitest,playwright,node-test,go-test,moon-test}). Supported platforms: linux-amd64, linux-arm64, darwin-arm64. Intel macOS and Windows are not supported (the MoonBit toolchain has no x86_64 macOS / Windows target).

curl -fsSL https://raw.githubusercontent.com/mizchi/pkspec/main/install.sh | sh

Installs all six binaries into ~/.local/bin (override with PKSPEC_INSTALL_DIR). Pin a version with --version 0.4.0. You also need the Pkl CLI on PATH.

Download pkspec-<plat>.tar.gz from the releases page, verify it against the published .sha256, extract the six bare-named binaries, and put them on PATH.

nix run github:mizchi/pkspec/v0.4.0 -- init --dir pkspec
nix run github:mizchi/pkspec/v0.4.0 -- exec -f path/to/Test.pkl
nix profile install github:mizchi/pkspec/v0.4.0

The flake fetches the prebuilt release tarball and wraps the binaries so the native Pkl CLI is on PATH automatically — no Go toolchain and no source build. That Pkl CLI is the upstream native binary, not the Java/JAR build from nixpkgs.

In a home-manager flake:

{
  inputs.pkspec.url = "github:mizchi/pkspec/v0.4.0";
  inputs.pkspec.inputs.nixpkgs.follows = "nixpkgs";

  outputs = { self, nixpkgs, home-manager, pkspec, ... }:
  let
    system = "aarch64-darwin";
    pkgs = import nixpkgs { inherit system; };
  in {
    homeConfigurations.example = home-manager.lib.homeManagerConfiguration {
      inherit pkgs;
      modules = [
        pkspec.homeManagerModules.default
        {
          programs.pkspec.enable = true;
        }
      ];
    };
  };
}

programs.pkspec.enable = true installs both pkspec and pkl-native by default. Set programs.pkspec.installPkl = false if you only want the wrapped pkspec binary and do not want a standalone pkl command in home.packages.

After installing, generate the local schemas and author test modules against them:

pkspec init --dir pkspec

amends "./pkspec/Test.pkl"

tests {
  new {
    name = "smoke"
    cmd = "true"
  }
}

A setup-only composite action lives at the repo root:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: mizchi/pkspec@v0.4.0
        with:
          init-schema-dir: pkspec
      - run: pkspec exec -f Test.pkl

The action installs pkspec and the Pkl CLI, then adds both to PATH. init-schema-dir is optional; set it when the workflow should materialize local Test.pkl / Spec.pkl / QuickCheck.pkl / Adapter.pkl schemas and built-in adapter modules.

Inputs:

pkspec init --dir pkspec                  write Test.pkl / Spec.pkl / QuickCheck.pkl / Adapter.pkl schemas

pkspec exec -f Test.pkl                    run all tests in a module
pkspec exec -f Test.pkl --tag spec         filter by Test.tags (repeatable, OR)
pkspec exec -f Test.pkl --only login       filter by name substring (repeatable, OR)
pkspec exec -f Test.pkl --shard=K/N        run only the K-th shard of N (LPT)
pkspec exec -f Test.pkl --rerun-failed     only tests whose latest record is non-pass
pkspec exec -f Test.pkl --total-timeout=5m abort run after wall-clock cap
pkspec exec -f Test.pkl --junit-reports DIR write JUnit XML

pkspec run [pkl test args...]              wrap `pkl test` with a trustworthy exit code

pkspec adapter -f Adapter.pkl              run adapter discover/run protocol and collectors
pkspec adapter -f Adapter.pkl --dry-run    discover and print merged adapter cases

pkspec spec tests/**/*.pkl                 render Markdown SPEC.md from Scenario tags
pkspec spec tests/**/*.pkl --output SPEC.md
pkspec spec tests/**/*.pkl --tag spec
pkspec docs --audience pm Spec.pkl         render PM-facing docs from audience metadata
pkspec docs --audience end-user --output docs/USER.md Spec.pkl
pkspec check Spec.pkl Test.pkl             CI gate: declared specs vs implementing tests
pkspec coverage Spec.pkl Test.pkl          coverage report by severity / review-status
pkspec graph Spec.pkl Test.pkl             graphviz dot of spec edges + implementation backlinks
pkspec decisions Spec.pkl Test.pkl         newest-first Markdown decision log
pkspec goals Spec.pkl Test.pkl             user-facing Goals + their contributing-spec coverage
pkspec milestones Spec.pkl Test.pkl        release/planning Milestones + Goal progress
pkspec next Spec.pkl Test.pkl              unimplemented specs ranked by Goal priority + severity
pkspec implementations Spec.pkl Test.pkl   spec id -> tests/code/doc implementers
pkspec orphans Test.pkl...                 active tests with no specRef (spec-link backlog)
pkspec lint Spec.pkl Test.pkl...           convention checks: broken/deprecated refs, descriptions, ...
pkspec lint --lint-disable lint.X          suppress one or more rule ids (comma-separated)
pkspec spec --template scenario|goal|module  print a Pkl skeleton (no input files needed)
pkspec spec --discover                     auto-walk for Spec.pkl / Test.pkl / specs/*.pkl
pkspec check --strict                      verify implementedAt paths exist on disk
pkspec check --goal goal.X                 filter review commands to one Goal
pkspec check --severity critical           filter review commands to one severity

pkspec timings -f Test.pkl                 per-test runs / median / p90 / latest / kind
pkspec timings -f Test.pkl --failing       only tests whose latest record is non-pass
pkspec timings -f Test.pkl --shard=K/N     preview which tests would land in shard K/N

PKSPEC_TIMING_ENV=ci-linux pkspec exec ... tags timing records with an explicit environment so CI history doesn't poison local-machine shard balancing (or vice-versa).

For JUnit report semantics and CI publishing notes, see docs/notes/junit.md.

Project maintenance tasks are defined in Taskfile.pkl and run with pkfire:

pkf list
pkf run build:pkspec       # build the six native release binaries
pkf run moon:test          # MoonBit unit gates (evaluator + model)
pkf run conformance        # candidate vs frozen goldens (71/71 strict)
pkf run preflight          # pre-commit gate (check/test + pkl + action-lint)

pkspec is implemented in MoonBit at the repo root (src/, moon.mod.json); nix develop includes the MoonBit toolchain and the native pkl CLI. To create the release tags locally (no push):

pkf run tag --version=0.4.0   # creates pkspec@0.4.0 + v0.4.0

Pushing pkspec@<ver> (and v<ver>) triggers the Release workflow (.github/workflows/mbt-publish.yml). It builds the six binaries on a matrix of linux-amd64 / linux-arm64 / darwin-arm64 runners, packages each platform into one pkspec-<plat>.tar.gz + .sha256, uploads them to the GitHub Release, and opens a follow-up PR syncing nix/pkspec-release.json. v-tags.yml then keeps the action-friendly v<ver> / v<major> tags in sync.

Active development, frequent API churn. v0.4.x is the first MoonBit-native line (the Go implementation has been retired), shipping prebuilt binaries via the install script, GitHub Actions, and Nix; expect schema and CLI changes before a stability promise.

For decision history per phase, see findings.md; the time-ordered raw log. For thematic deep dives, see docs/notes/ and docs/advanced/.

If you are looking for a real task runner rather than a test runner, see mizchi/pkfire; pkspec is its testing-focused sibling.

The Scenario.openQuestions field and the open-questions-policy recipe were prompted by reading NyxFoundation/speca, a specification-anchored security-audit framework. pkspec borrows a small idea — keeping unresolved authoring questions first-class on the spec so they cannot be silently rolled over — and does not import speca's broader proof-attempt pipeline or security framing.

MIT.