Your TTS provider just died.The episode still ships.
CastIron is a self-healing audio-episode factory: hand it a script and it fans out narration, music, and cover art in parallel. When a voice provider goes dark mid-render, a cross-provider failover ladder (ElevenLabs → LMNT → Hume) steps down a rung — and the episode still lands on Backblaze B2, hash-verified, Object-Locked, with provenance sealed inside the MP3.
$ OFFLINE=1 .venv/bin/python bench.py- 0/96
- episodes shipped hash-verified
- 0
- dropped — healthy + forced-outage runs
- 0
- tests passing · 100% line coverage
- 0ms
- p95 failover orchestration (offline)
* Benchmarked by the checked-in bench.py, seeded (SEED=42), exits non-zero on any correctness failure. Latency is OFFLINE orchestration — vendor synthesis excluded. Full methodology in DEMO.md.
01 / The failure mode
Generative pipelines break in exactly one place that matters.
The vendor call. Everything upstream of it is your code; everything downstream is your reputation. CastIron was built for the moment between the two — when the provider you paid for simply stops answering.
Vendors go dark without notice
TTS providers throttle, return 500s, and disappear mid-render. The vendor call is the one brittle joint in every generative-media pipeline.
One outage kills the whole episode
A naive pipeline drops everything with it — narration, music, cover, the entire run. On a schedule, that's dead air for your daily brief or podcast.
Nothing proves the file is untouched
Once an episode is produced, most pipelines can't show it wasn't silently altered afterward. No tamper-evidence from render to storage.
> CastIron treats provider failure as the expected case — not the exception.
02 / The outage drill
Kill the provider yourself.
This is a scripted replay of CastIron's real /console SSE rail — the same beats as the checked-in demo. Flip the chaos toggle and watch the ladder step down a rung without dropping the episode.
narration · failover ladder
- 0standby
elevenlabs
primary
- 1standby
lmnt
failover
- 2standby
hume
last resort
parallel fan-out
manifest — the actual provider, not the requested one
provider_used = "lmnt"
fallback_rung = 1
verify() = … · dropped = 0
ALL GREEN — 0 dropped episodes. outage caught on rung 0 · shipped via rung 1
Proven 48/48 forced-outage trials in the seeded benchmark — the primary provider is killed on every failover trial.
Run it live at api.castiron.edycu.dev/console03 / What makes it unkillable
Failure is the default path — so every stage is armored.
Cross-provider failover ladder
Every narration render tries distinct vendors in order — ElevenLabs → LMNT → Hume. One dies mid-render, the ladder steps down a rung and the run keeps moving. CastIron's own primitive: Genblaze's built-in fallback_models is in-provider only (gap filed upstream).
Provenance hashed into the file
The manifest is embedded inside the MP3 (ID3). Edit one byte and verify() → False. Tamper-evidence travels with the episode.
Immutable publish
Episodes land in Backblaze B2 under a real Object Lock — GOVERNANCE +30d, proven by reading the retention back from a live bucket. Published means provably unaltered.
Self-healing quality gate
An AgentLoop evaluator iterates the narration until it passes a LUFS band, silence ratio, and duration drift — and transient failures resume with a single charge, not a resubmit.
Budget hard-abort
A run projected over MAX_RUN_COST_USD aborts before spending, with a typed BUDGET_ABORT — no surprise invoices from a retry storm.
Always-green OFFLINE mode
Mock providers + a local backend give a zero-network, zero-credential path — the dev loop, the deterministic regression net, and the demo-day disaster fallback are the same code.
$ OFFLINE=1 verify_offline.py → "ALL GREEN — 0 dropped episodes"04 / The engine room
The SDK is the engine — not decoration.
Nine load-bearing Genblaze surfaces and three distinct Backblaze B2 capabilities, each one doing real work in the run path. Remove either half and the differentiator collapses.
- 01
POST /runs
script in, chaos optional
- 02
fan-out ×3
narration · music · cover
- 03
gate + verify
AgentLoop → manifest → ID3 embed
- 04
B2 event
HMAC → stage machine
- 05
object lock
immutable publish
Pipeline.astream(max_concurrency=3)pipeline.pyObjectStorageSink · HIERARCHICALpipeline.pyread_manifest(verify=True)pipeline.pySmartEmbedder — in-file ID3 manifestmedia.pyPipeline.resume_step / aresume_stepresume.pyAgentLoop · CallableEvaluator + ThresholdEvaluatorgate.pyObjectLockConfig(mode=GOVERNANCE)publish.pyStorageBackend subclass + ProviderComplianceTestsbackends.pyRetryPolicy per rung · S3StorageBackend.for_backblazeladder.py
Storage plane
HIERARCHICAL object layout over the S3 API — runs/{date}/{run}/… plus manifest.json for every asset.
Control plane
B2 Event Notifications, HMAC-signed, drive an idempotent render→mix→verify→publish stage machine that converges under duplicate and reordered delivery.
Immutability plane
Object Lock (GOVERNANCE, 30 days) on ci-published/ — verified live by reading back get_object_retention, not claimed.
Why only this pair: the resilience thesis needs both halves. Genblaze's provider abstraction is what makes a cross-provider ladder and manifest-verified resume possible; B2 Object Lock is what turns "published" into "provably unaltered." Without Genblaze there is no uniform provider/manifest layer to fail over across — without B2, tamper-evidence stops at the file and never reaches storage.
05 / Receipts
Testimonials from things that cannot flatter you.
A weekend project doesn't have customers — it has evidence. Every quote below is a real artifact of the pipeline, and every one can be regenerated from a fresh clone.
“HEADLINE: 96/96 episodes shipped hash-verified across healthy + forced-outage runs — 0 dropped.”
“provider_used = lmnt · fallback_rung = 1 — the vendor that actually rendered, not the one requested.”
“Mode=GOVERNANCE · RetainUntilDate=+30d — retention read back from the live bucket, not claimed.”
“Flip one byte of the sealed provenance and verify() returns False. Every time.”
“175 passed. 100% line coverage on castiron/. Ruff clean.”
“Quality → Security → Build → E2E → Performance → Deploy: green. CodeQL + Dependabot + secret scanning: 0 open alerts.”
06 / Interrogation
The questions a skeptical judge should ask.
OFFLINE=1 .venv/bin/python bench.py on a fresh clone: fixed seed (SEED=42), zero config, no API keys, and the script exits non-zero if any correctness gate fails. 48 healthy trials + 48 forced-outage trials, every episode hash-verified. Full per-scenario table in DEMO.md.provider_used: lmnt, fallback_rung: 1), not the requested one, and the episode ships hash-verified. The live SSE rail shows the rung step in real time.fallback_models retries within a single provider — if the vendor itself is down, every fallback model is down with it. CastIron's LadderTTSProvider fails over across distinct vendors and persists the rung into the run manifest. The gap was filed upstream as a dossier issue.verify() to False — the tamper-evidence travels with the file. Second, the published object sits under a real B2 Object Lock (GOVERNANCE, 30 days), so the stored copy can't be overwritten either.get_object_retention. Drop B2 and the tamper-evidence stops at the file instead of reaching storage.OFFLINE=1 swaps in mock providers and a local storage backend, no network, no credentials. It's the dev loop, the deterministic regression net behind all 175 tests, and the demo-day disaster fallback — the same code path.MAX_RUN_COST_USD hard-aborts with a typed BUDGET_ABORT before a cent is spent. And transient failures use Pipeline.resume_step — resume, single charge, not resubmit.Still skeptical? Good. Read the methodology in DEMO.md — limitations included.
07 / Your move
Kill a provider.Ship anyway.
Open the live console, flip the chaos toggle, and watch a real run step down the ladder — or clone the repo and make the benchmark prove it to you.
$ OFFLINE=1 .venv/bin/python bench.py- Zero API keys needed offline
- Seeded — same numbers every run
- Exits non-zero on any failure
