Model Canary

Your coding agent can ship. Model Canary can say absolutely not.

Comment

Inspiration

ML systems are getting more automated, and coding agents can now make changes, run tests and deploy with surprisingly little human input.

The problem is that they usually understand the code they are editing, but not the full production ML system and context around it.

A tiny upstream data transformation can break a feature, affect several models and put a production deployment at risk, even if the code itself looks fine and the local tests pass.

I built Model Canary around that gap: an independent safety layer that looks at the whole ML dependency graph before letting a bad change quietly propagate.

What it does

Model Canary watches DataHub for failed data-quality and freshness assertions. When it sees a new failure, it turns it into a typed incident signal, deduplicates it and starts an investigation automatically.

The investigation is built as a bounded LangGraph workflow. It collects context, generates a controlled set of possible hypotheses, decides which read-only tools are needed to test them, executes those tools against DataHub and then evaluates the evidence. (The investigation has hard limits on iterations and tool calls, so it cannot just keep exploring forever.)

The evidence evaluation is deterministic. Model Canary looks at structured facts such as assertion results, metric types, dataset URNs, recent metadata changes and timestamps. Evidence can support, contradict or stay neutral to a hypothesis. It also deduplicates repeated observations of the same underlying fact, so seeing the same failed assertion through two different tool calls does not artificially make a hypothesis look twice as strong.

Once Model Canary has a supported explanation, it calculates the ML blast radius through DataHub:

dataset -> feature -> model -> deployment

It then applies a deterministic promotion policy. If a candidate model shares an unhealthy upstream dependency, promotion is blocked.

The LLM is not allowed to make that safety decision. It is used to turn the structured investigation into a useful human explanation and it only receives metadata, never raw dataset rows, model weights, or credentials.

Developers can access all of this directly from Claude Code through the Model Canary MCP server and project skill. They can ask what went wrong, inspect the blast radius or check whether a model is safe to promote without leaving their coding workflow.

The web console is the second surface. It is essentially a debugger for the safety agent. The Canary Graph combines the actual DataHub production lineage with Model Canary's reasoning trail, including signals, tool executions, evidence, hypotheses, policy decisions and proposed actions. This means you can see both what depends on what and why the system reached its conclusion.

How we built it

The backend is written in Python with FastAPI. DataHub is the source of truth for ML metadata, lineage, assertions, ownership and recent pipeline changes.

A background watcher polls DataHub's assertion results and converts new failures into typed IncidentSignals. Those signals are then passed into the LangGraph investigation engine.

The investigation itself is deliberately constrained. The agent layer cannot access DataHubGraph directly. All access goes through a ToolRuntimeContext and an allowlisted executor. The available tools include reading dataset assertions, finding owners, inspecting recent metadata changes, and traversing which features use a dataset. Write tools are blocked during the investigation.

The planner is also deterministic and budget-aware. It maps each hypothesis to the tools that can actually test it, avoids duplicate calls, skips calls when the evidence is already available, and respects the remaining investigation budget.

For evidence evaluation, we built our own deterministic interpreter instead of asking an LLM whether something "looks plausible". For example, a failing value-range assertion on the payments dataset supports a data-quality regression, while a healthy unrelated null-rate assertion stays neutral instead of incorrectly disproving it. A timestamped pipeline_transform metadata change supports a transformation-change hypothesis because it provides evidence of an actual causal event.

The result of every investigation is persisted with SQLModel and SQLite. It stores the original signal, collected evidence, hypotheses and scores, tool execution records, blast radius, budget usage, explanation, and proposed actions. This means the console is rendering a real persisted investigation rather than asking the agent to regenerate its reasoning every time the page loads.

The Canary Graph is built deterministically from that persisted record and rendered with Cytoscape.js and dagre. It has separate Impact, Assets, Reasoning, and Hybrid views depending on whether you want to understand the production dependencies, the investigation logic, or both.

For the developer workflow, the repository contains project-scoped MCP configuration and a Claude Code skill. DataHub tools provide organisational context and lineage, while Model Canary remains the authority for incidents and promotion safety. Claude can investigate and repair code, but it cannot approve writebacks, close incidents or override a promotion decision!

Challenges we ran into

The hardest part was making the demo behave like a real production system instead of a scripted simulation.

DataHub assertion results are eventually consistent, so there were timing issues where a write had succeeded but a read still returned the previous result. We had to make the verification and investigation flow handle that properly rather than relying on fake state changes.

Another challenge was deciding how much autonomy the coding agent should have.

We deliberately ended up drawing a hard boundary: the agent can investigate, repair code, and redeploy, but it cannot approve its own safety state or close the incident that blocks it.

That made the system more interesting, but also much more realistic.

Accomplishments that we're proud of

Figuring out how DataHub works was a task in itself, and then weaving Model Canary into its assertions, lineage, ML entities, metadata changes and writeback mechanisms was a challenge on top of a challenge...

I am also really proud that the final workflow feels agent-native. The developer does not need to constantly watch another dashboard. They can stay inside Claude Code, make a change, deploy it and ask whether a model is safe to promote. But if they want a dashboard experience, they get it too.

What we learned

The strongest architecture ended up being a separation of responsibilities. So:

DataHub provides the shared system context. Model Canary produces the safety verdict. The coding agent reacts to that verdict and repairs the system when necessary. The human remains responsible for consequential approvals.

What's next for Model Canary

The main next step is expanding Model Canary beyond data-quality regressions.

I initially wanted to tackle feature drift, schema changes, training-data issues, freshness failures, and model-performance degradation, etc.... But there is only so much you can accomplish during a hackathon.

Built With

  • claude
  • datahub
Share this project:

Updates