CastIron

Script to published episode — even when your TTS provider dies. A self-healing audio-episode factory on Genblaze + Backblaze B2.

Comment

Inspiration

Generative-media pipelines are brittle in exactly one place that matters: the vendor call. TTS providers throttle, return 500s, and go down — and when they do, a naive pipeline drops the entire episode: the narration, the music, the cover, the whole run. For anything on a schedule — a daily news brief, a podcast, an automated show — a single upstream outage means dead air. And once an episode is produced, most pipelines can't prove it wasn't silently altered afterward.

We wanted to build the pipeline you could actually put on a cron job: one that treats provider failure as the expected case, not the exception — and that can prove every episode it ships is authentic and unaltered.

What it does

CastIron turns a script into a published podcast episode and keeps shipping even when a provider dies mid-render. Hand it a script; it fans out narration, music, and cover art in parallel. The moment the primary voice provider goes dark, it steps down a cross-provider failover ladder (ElevenLabs → LMNT → Hume) and still lands a hash-verified, immutably-published episode — with the manifest recording the actual provider used, not the one requested.

The promise is one measurable number: zero dropped episodes.

Four pillars make that real:

  1. Cross-provider failover ladder — every narration render tries distinct vendors in order; one outage steps the ladder down a rung and the episode still ships.
  2. Provenance hashed into the file — each asset's manifest is embedded inside the MP3 (ID3); edit one byte and verify() flips to False. Tamper-evidence travels with the episode.
  3. Immutable publish — finished episodes land in Backblaze B2 under a real Object Lock (GOVERNANCE, 30 days). "Published" means "provably unaltered."
  4. Self-healing orchestration — an AgentLoop quality gate (loudness / silence / duration), transient resume (single charge, not resubmit), a budget hard-abort, and an always-green OFFLINE fallback keep the run alive.

How we built it

  • Engine — Genblaze 0.4.1 drives the whole pipeline, not a single decorative call. We use Pipeline.astream(max_concurrency=3) for the parallel fan-out and typed event stream, ObjectStorageSink (HIERARCHICAL) + read_manifest(verify=True), SmartEmbedder for the in-file ID3 manifest, Pipeline.resume_step for transient single-charge resume, the AgentLoop CallableEvaluator + ThresholdEvaluator for the quality gate, ObjectLockConfig(GOVERNANCE) for immutable publish, and we subclass StorageBackend + run the ProviderComplianceTests conformance kit. The cross-provider ladder is our own primitive built on Genblaze's provider abstraction (its built-in fallback_models is in-provider only).
  • Storage + control plane — Backblaze B2 is both: HIERARCHICAL object layout via the S3 API, Event Notifications (HMAC-signed) that trigger the publish stage machine, and Object Lock that makes the published episode provably immutable — verified live by reading back get_object_retention.
  • App — a FastAPI backend with a live Server-Sent-Events stage rail (/console), background episode runs, and a SQLite event log as the durable source of truth. Audio synthesis/mixing via ffmpeg; ID3 embedding via mutagen. Packaged with uv, containerized with Docker.
  • Rigor — 175 tests, 100% line coverage, a 6-stage GitHub Actions CI/CD pipeline, CodeQL + Dependabot + secret scanning, and a one-command reproducible benchmark.

Challenges we ran into

  • Genblaze's fallback is in-provider only. To fail across vendors we had to build our own LadderTTSProvider that records the actual rung into the manifest metadata (we verified this persists into the run manifest) — and we filed the gap upstream as a feature request.
  • Object Lock is easy to think you've set and not have. Our first publish passed string args that boto3 silently ignored — nothing was actually locked. We fixed it to a real ObjectLockConfig and proved immutability by reading back the enforced retention (mode=GOVERNANCE, RetainUntilDate).
  • The tamper story has to reach storage. Manifest.verify() checks the provenance record but not the audio payload bytes — so we embed the manifest inside the MP3 and verify the file, then seal it under Object Lock so authenticity survives all the way to B2.
  • Honest, reproducible numbers. We built an OFFLINE mode (mock providers + a local backend, zero network) so the whole pipeline — including the failover — is deterministic and always-green, and wrote a seeded benchmark so every metric is reproducible on a fresh clone.

Accomplishments that we're proud of

  • 96/96 episodes shipped hash-verified across healthy + forced-outage runs — 0 dropped. The primary TTS provider is killed on every failover trial and the ladder still lands a verified, publish-ready episode every time.
  • Real Backblaze B2 Object Lock, proven — not claimed. Immutability verified by reading back the enforced retention on a live bucket.
  • Production-grade rigor for a hackathon: 175 tests (100% coverage), a green 6-stage CI/CD pipeline, CodeQL + Dependabot + secret scanning with 0 open alerts, and a one-command benchmark (bench.py) that anyone can reproduce.
  • Deep, honest SDK usage — 8+ real Genblaze surfaces and 3 distinct B2 capabilities (S3 storage, Event Notifications, Object Lock), each load-bearing.

What we learned

  • Treating failure as the default path — not an afterthought — changes the whole architecture: every stage becomes a try-ladder, every artifact carries its own provenance, and "done" means "verified and sealed."
  • Deep integration beats decoration. Genblaze's provider/manifest abstraction is exactly what makes a cross-provider ladder and manifest-verified resume possible; B2 Object Lock is what turns "published" into "provably unaltered." Remove either half and the differentiator collapses.
  • If you claim a number, ship the script that proves it. An OFFLINE, seeded, deterministic harness made honesty cheap.

What's next for CastIron

  • Close the live loop end-to-end: deploy the B2 Event-Notification receiver at a public URL and run one real ObjectCreated round-trip driving the publish stage machine.
  • Real-provider ladder run on live keys (ElevenLabs → LMNT → Hume) to complement the deterministic OFFLINE proof.
  • Payload-level authenticity (re-hash the decoded audio, not just the provenance record), a wider chaos matrix, and Slack/RSS delivery on the publish event.

Built With

  • amazon-s3-api
  • backblaze-b2
  • codeql
  • docker
  • elevenlabs
  • fastapi
  • ffmpeg
  • genblaze
  • github-actions
  • hume
  • lmnt
  • mutagen
  • python
  • railway
  • server-sent-events
  • sqlite
  • uv
Share this project:

Updates

Submission history