Innkeeper | Devpost

Overview: the problem and the autopilot at a glance (three disagreeing ledgers, a 2 AM cron, a signed close).
Architecture: 3× MCP fetch → qwen3-vl-plus extract → deterministic matcher → qwen3.7-max adjudicate → E[loss] gate → signed Merkle close.
A night run: fetch → extract → match → adjudicate → gate → signed close, including the `m-07` auto-clear citing OTA line 7 ($5.67).
The morning report: what auto-cleared vs. what queued, with the `m-12` $310 no-folio case carrying both hypotheses at 0.60 / 0.40.
Zero-key replay: `innkeeper replay` re-derives every verdict, root & signature byte-for-byte (I4); a one-byte tamper fails verify-chain.
Live on Alibaba Cloud: the Function Compute `/verify` endpoint re-verifying all 30 signed closes in the cloud, offline and key-free.
The load-bearing number: 277/281 auto-cleared, 0.9964 classification accuracy, and 0 false auto-clears across the whole τ-sweep.
Social/OG card (branding closer).

Inspiration

A husband-and-wife inn runs 14 rooms. Every night one of them stays up past midnight squinting at three statements that disagree by $6.67 — the property-management folios, the card processor's settlements, and the OTA's PDF — because the last time they let a small gap slide, it was $2,300 by month-end. The night audit is exactly the kind of work the Autopilot track asks about: repetitive, unattended, high-stakes, and full of "should I clear this or ask a human?" moments. We wanted to hand that whole ritual to an agent that runs while the owners sleep — and, crucially, one that knows when it is not allowed to decide alone.

What it does

Innkeeper is an autopilot night auditor. In production an Alibaba Function Compute timer trigger fires the audit at 2:00 AM — the cron log, not a button click. Each run fetches three sources through three MCP servers (PMS folios, processor settlements, and the OTA statement), and the OTA source is deliberately what it is in real life: a PDF, read by qwen3-vl-plus with bounding-box citations in two passes (temperatures 0.0 / 0.4) that must agree or the figure escalates — never silently averaged (invariant I5).

A deterministic 3-tier matcher (reference → fuzzy → amount+date window) clears ~77% of transactions at zero model cost; only the mismatch residue reaches qwen3.7-max.
qwen3.7-max adjudicates with thinking: a typed Verdict carrying a classification (timing / fee / FX / duplicate / true error), evidence citations from at least two systems (each bound by a resolvable sha256), competing hypotheses, and a confidence.
The human-in-the-loop gate — the track's "critical decision points" — is math, not vibes: auto-clear ⟺ confidence ≥ 0.85 ∧ materiality ≤ $50 ∧ class ≠ true_error, generalized as E[loss] = amount × (1 − confidence) ≤ τ, with the τ-sweep curve published in bench.py. A true_error classification (I1) and a two-pass disagreement (I5) can only ever queue.

In the demo, mismatch m-07 auto-clears — the verdict cites OTA line 7 (bbox) and the PMS folio: $189.00 − $183.33 = $5.67, exactly the 3% OTA commission, confidence 0.94. The $310 charge with no folio (m-12) lands in the morning queue carrying both hypotheses at 0.60 / 0.40 — the agent knowing what it doesn't know.

Every night closes cryptographically: verdicts form a Merkle tree, the root is Ed25519-signed and chained to the previous night — a tamper-evident ledger of ledgers. innkeeper replay --night 2026-07-04 re-derives every verdict from stored evidence with zero API keys; flip one byte and verify-chain fails, by test.

The load-bearing number: across a seeded 30-night month (1,199 transactions), 277 of 281 mismatches auto-clear (98.6%), classification accuracy is 0.9964, and zero of the four escalations is a false clear — the invariant "zero false auto-clears on planted true errors" is a pytest, not a promise.

How we built it

Python 3.12, shipped as a Typer CLI plus three MCP (JSON-RPC stdio) mock servers (PMS, card processor, OTA-PDF), not a web app. The data is a committed, deterministically-seeded 14-room month with real reportlab-rendered OTA PDFs and planted, ground-truth-labeled discrepancy archetypes. Pydantic v2 makes every Verdict typed so the gate is math over typed fields.

Built on Qwen Cloud — the transport is swappable but the reasoning is Qwen-only. qwen3-vl-plus reads the 8-pt OTA PDF table with bbox citations, two-pass (0.0 / 0.4), where agreement becomes confidence and disagreement escalates. qwen3.7-max with thinking adjudicates the mismatch residue over evidence and ranks competing hypotheses. Structured output gives the E[loss] gate typed fields to compute over; function calling over the three MCP servers turns every tool call into a typed, logged entry in the replayable decision log.

The transport switch is honest and explicit: FakeQwen is the default — deterministic, offline, no key; it computes each verdict from the mismatch's own arithmetic and memos (the same signals qwen3.7-max reasons over), makes no VL call, and never reads the ground-truth labels. LiveQwen runs only under --live with a DASHSCOPE_API_KEY (DashScope's OpenAI-compatible endpoint, openai + pypdfium2); it rasterizes the committed OTA PDF and calls the real models. That live path is wired and verified with a real DashScope smoke call, and stays key-gated — a full end-to-end live-keyed run is not yet captured, so the graded, deployed, and demo path is the offline-deterministic FakeQwen engine, byte-for-byte replayable.

Deployed live on Alibaba Cloud Function Compute: a thin WSGI handler (infra/fc/) runs the audit engine on managed python3.10 in ap-southeast-1, deployed straight from source with Serverless Devs (no container registry), at https://innkeeper-temfmzpqug.ap-southeast-1.fcapp.run. Every endpoint runs offline on the committed fixtures/ + ledger/ — zero keys, no network, no vision model — so a judge can verify the signed books from a browser: /health (liveness), /verify (re-verifies the 30 signed closes in the cloud — roots, Ed25519 signatures, evidence sha256 bindings, a byte-identical replay of 2026-07-04, and a one-byte tamper caught — I2 / I3 / I4), and /run?night=2026-07-04 (reproduces one deterministic FakeQwen night audit). The 02:00 timer trigger is configured in infra/fc/s.yaml; a captured console recording of the cron firing on its own is the one remaining artifact — we do not claim that recording here. Quality and security are gated in CI (ruff + mypy, 404 pytest tests at 100% coverage, invariants I1–I5 parametrized across all 30 nights, a zero-key offline-replay judge path, CodeQL, Dependabot + pip-audit, TruffleHog, build verification), versioned by python-semantic-release.

Challenges we ran into

Reading a real-world PDF, not a clean API. OTA statements arrive as PDFs; we made qwen3-vl-plus return bounding-box citations and ran two passes that must agree, so a shaky extraction escalates instead of silently corrupting the books (invariant I5).
A human-in-the-loop gate that is defensible. "When should the agent stop and ask?" is the whole Autopilot risk surface. We made it expected-loss math (amount × (1 − confidence) ≤ τ) with a published τ-sweep, and hard-wired that true_error and two-pass disagreement can only queue — then proved zero false auto-clears across 30 planted-error nights.
Cheap where it can be, smart where it must be. The deterministic 3-tier matcher clears ~77% at zero model cost so only the ~23% residue touches the LLM — the difference between a demo and something a 14-room inn could actually afford to run every night.

Accomplishments that we're proud of

Zero false auto-clears on planted true errors across a 30-night seeded month — as a test, with 0.9964 classification accuracy and 277/281 auto-cleared.
A tamper-evident ledger of ledgers — Merkle + Ed25519, chained night-to-night, replayable with zero keys.
404 tests at 100% coverage, invariants parametrized across all 30 nights.
A live Alibaba Cloud Function Compute deployment whose /verify re-verifies all 30 signed closes in the cloud, offline and key-free.
Three seedable MCP mock servers that are a genuine developer asset for anyone building reconciliation agents.

What we learned

For an autopilot, the interesting engineering isn't the automation — it's the refusal to automate. Turning "should I clear this?" into expected-loss math with a couple of inviolable queue-only rules is what lets the agent run unattended and still be trustworthy. And keeping the cheap deterministic matcher in front of the LLM is what turns a clever demo into something a tiny business could actually run every night.

What's next for Innkeeper

Capture a console recording of the 02:00 timer trigger firing on its own, and provision ApsaraDB for cloud persistence (the deployment and /verify are already live).
One full live-keyed Qwen run against the rendered PDF (qwen3-vl-plus two-pass + qwen3.7-max adjudication) — the model path is wired and smoke-verified with a real DashScope call, not yet captured end-to-end.
Publish innkeeper-audit to PyPI.