Agentic Workflow Platform

A governed, inspectable multi-agent evaluation platform for fully synthetic loan-servicing tickets. It compares a single prompt, retrieval grounding, and a typed planner/specialist/reviewer workflow while recording quality, latency, cost, escalation, fairness, redaction, and every hand-off.

No Nelnet, borrower, or production data is included. The committed results are deterministic offline test-double measurements, not resume-ready model claims.

The reproducible 150-case suite currently records:

These values are stored in baselines/offline.json and gated in CI with a maximum 3-point accuracy drop. They validate the harness, not real-model performance. Run the live sweep before using any number externally.

flowchart LR
    T["Synthetic ticket"] --> G["PII redaction gate"]
    G --> P["Planner"]
    P --> R["Retriever"]
    P --> A["Analyst"]
    P --> D["Drafter"]
    R --> V["Reviewer"]
    A --> V
    D --> V
    V -->|confidence ≥ τ| X["Accept"]
    V -->|confidence < τ| H["Human review queue"]
    G -. every call .-> S[("PostgreSQL audit trail")]
    X --> S
    H --> S
    S --> API["FastAPI registry"]
    API --> UI["React evidence dashboard"]
    S --> M["Markdown + plot reports"]

• Every graph edge is validated with a strict Pydantic v2 contract.
• Retriever, analyst, and drafter run concurrently through LangGraph.
• A single governed model client handles redaction, token/cost accounting, stable rehydration, latency, payload hashes, errors, and audit writes.
• Exactly five registry tables preserve experiments, runs, steps, case results, and reports.
• The dashboard exposes accuracy, cost/latency, fairness, and case-level traces.

Docker Compose starts PostgreSQL 16, migrates the schema, serves the API and dashboard, and runs one offline three-arm sweep:

cp .env.example .env
docker compose up --build

• Dashboard: http://localhost:8080
• API docs: http://localhost:8000/docs
• Worker progress: docker compose logs -f worker

The worker exits successfully after seeding the registry; refresh the dashboard when it finishes. Stop the stack with docker compose down.

For local development:

uv sync --frozen --extra dev
npm --prefix web ci
uv run alembic upgrade head
uv run uvicorn agentic_workflow.api:app --reload
npm --prefix web run dev

# Deterministic, free, 150 cases × 3 arms
uv run agentic-workflow eval --provider offline

# Recompute the full arm and enforce the committed 3-point budget
uv run agentic-workflow gate

# Real OpenAI calls; requires OPENAI_API_KEY and the optional PII dependencies
uv run --extra pii python -m spacy download en_core_web_sm
uv run --extra pii agentic-workflow eval --provider openai

Model names come from .env; token prices are versioned in api/agentic_workflow/pricing.json. Each run emits a Markdown report and Matplotlib plots under reports/.

The golden set contains exactly 150 deterministic cases across late-payment, payment-allocation, and identity-verification workflows. data/review-log.csv deliberately marks every item pending: no human correction or judge-agreement claim is made until a reviewer signs that ledger. Scoring is deterministic fact-level recall so CI is stable; live-model and human-review results belong in a new labelled baseline.

make lint
make test
make gate
npm --prefix web run build
docker compose config --quiet

The CI workflow runs those backend, frontend, dataset-reproducibility, and container-configuration checks independently. See docs/superpowers/specs/ for the design rationale and docs/superpowers/plans/ for the implementation record.

• The bundled provider is a deterministic test double; live OpenAI metrics are intentionally absent until credentials are supplied.
• The synthetic golden set is generated and machine-verified, not yet hand-corrected. Do not claim otherwise.
• Stable string matching makes the CI scorer reproducible but is stricter than a calibrated LLM judge. Add a blinded 30-case human agreement study before making judge-quality claims.