Delivery Tracker

Inspiration

We started with the obvious question — does bad delivery drive customers away? — and killed it in ten minutes. Olist's repeat-purchase rate is 3.12%, and it's flat across review scores: customers whose first order got 1 star come back 3.16% of the time; 5-star customers, 3.22%. There is no churn signal in this data to explain. We'd have shipped a beautiful chart of nothing. So we went looking for a decision the marketplace actually makes, and found it in the one number every checkout shows and nobody audits: the delivery date.

What it does

The Promise Engine decides the delivery date a marketplace shows at checkout — one it can actually keep — and tells ops, lane by lane, whether to pad the promise or fix the lane.

Then it does the thing we didn't expect to build. Break a promise by 8+ days and 70% of customers leave 1 star (against a 6.6% baseline), so we optimized the promise for reviews — the only outcome Olist's data can measure. The optimizer came back unbounded: promise +20 days to everyone, every order lands early, reviews max at 4.29/5, late rate hits zero. Push to +200 and it scores identically. The data never says stop.

It says stop. The cost of a 60-day promise is a customer who sees it and never orders — and Olist has no clickstream, no sessions, no cart events. That cost isn't unmeasured, it's structurally unmeasurable. So the agent refuses its own optimum and falls back to a criterion that needs no data it doesn't have: is a lane's gap distance (median — irreducible, padding is honest) or variance (p95 − median — recoverable, the lane is broken)?

That distinction found Rio. Rio de Janeiro's median delivery is 12 days. Its p95 is 38. Rio isn't slow — it's unpredictable. Olist promises 27 days and breaks its word to 1 in 7 customers, in its #2 market, sitting right next to São Paulo. Padding Rio to 38 days would keep the letter of the promise while making Olist's second-biggest market look worse than Pará. Verdict: fix it. Pará, genuinely 2,500km away, gets PAD — there, padding is honest.

Three surfaces: a checkout simulator (per-seller, per-destination quote, decomposed into seller handling + lane transit + lane tail), an ops work-queue ranked by gap × volume (Rio tops it at 135,850 orders at risk — 6× the next lane), and a live investigation you can watch: probe → probe → trap → refusal → resolution.

Every SQL query came from CRAFT's generate_sql. We hand-wrote none.

How we built it

CRAFT's MCP server (generate_sql → execute_query → get_result_page) against the Olist Snowflake database. We asked it seven natural-language questions — lane percentiles, seller handling times, carrier transit, the review-damage curve, monthly seasonality, and two falsification queries — and recorded every response as a committed JSON fixture. The repo therefore runs end-to-end with zero credentials, tests are deterministic and offline, and the live CRAFT path stays real.

The load-bearing architectural decision: numbers never pass through the LLM. The model chooses which questions to ask and writes the prose; all math — the p95 promise, the gap, the pad/fix verdict, the trap curve — is deterministic Python. Every tool call records every number it emits into a computed set, and a narrative guard walks the model's output and raises HallucinatedNumber if it states a figure no tool produced. A model that invents a delivery date fails the build instead of reaching a customer.

That guard fired for real twice during the build. Once when a draft narrative said "68%" where the tool had recorded 0.68. Once when the trap's verdict text contained invented literals. Both times we fixed the narrative to source its numbers from tools — we never loosened the guard.

Challenges we ran into

A code review caught a silent 100× data-corruption bug on our trust boundary. Our late-rate loader inferred its unit from its magnitude — value / 100 if value > 1. A lane genuinely 0.85% late would arrive as 0.85, fail the test, and get stored as 85% late. No exception. And the LLM would have narrated it confidently. The only thing standing between us and it was a HAVING COUNT(*) >= 500 filter that happened to exclude every lane below 5% late. The loader now reads the unit from the column name, and every Lane validates its own invariants (0 ≤ median ≤ p95, finite fields, 0 ≤ late_rate ≤ 1) at construction.

We had to retract our own headline claim. We'd written that the 0.60 variance threshold "derives the finding rather than being fitted to it." It doesn't. Sorted by score, the lowest FIX lane (Ceará, 0.600) and the highest PAD lane (Pernambuco, 0.594) are 0.006 apart — Ceará sits exactly on the cut. Rather than bury that, the engine now reports a flip_distance for every lane: Rio's verdict survives an 8.1-day error in its p95; Ceará's flips on 0.1 days. Both are FIX, but only one is safe to bet on — and it's the one the whole story rests on. Saying it first is better than being asked.

Our own metric was lying to us. We called it variance_share, but (p95 − median) / p95 contains no variance — it reduces to 1 − median/p95, which mechanically rewards fast lanes. São Paulo scored 0.650, higher than two of our three FIX lanes. Only an unrelated short-circuit stopped SP being flagged. Renamed to tail_fraction and documented as meaningful only for lanes that actually have a gap.

Accomplishments that we're proud of

The agent refuses its own optimum, and can show its work. It computes the review-maximizing promise, proves it's unbounded, and declines — citing the specific missing data (no clickstream) that makes the metric untrustworthy. An agent that knows the limits of its own evidence is a better agent than one that doesn't, and we didn't set out to build that; the data made us.

Four hypotheses tested, two killed, before a single recommendation. Churn (dead — 3.12%, flat). "A few bad sellers" (dead — a volume artifact: the top-30 "worst" sellers are 9.39% late against a 7.87% baseline, and not one seller in the marketplace exceeds 40% late. They're big, not bad. You cannot fire your way out of it). The rule also found a lane we'd missed by hand — Rio Grande do Sul, tail_fraction 0.606, 5,345 orders.

The whole thing runs on a laptop with no API keys. 112 tests, offline, deterministic, against real recorded CRAFT responses.

What we learned

The metric you can measure is not the metric you should optimize. Olist's data contains a gradient that points straight off a cliff, and a competent, well-intentioned, "data-driven" agent will follow it all the way down and produce a dashboard full of green. The most valuable thing our agent does is decline. We think that generalizes far past delivery dates.

Also: execute_query does not return rows — it returns an artifact_fqn, and you call get_result_page with it. That one cost us twenty minutes and we're told it costs everyone twenty minutes.

What's next for Delivery Tracker

A TIGHTEN verdict. We have FIX, PAD, and OK — but nothing for a lane that's over-promising and needlessly slow-looking. In Olist it isn't a live problem (the largest over-promise is 0.8 days), but a fuller product needs it, and its absence is the honest hole in our rule. Ship the seasonality axis properly. It's built and it's a second, independent instance of the same blindness: on Black Friday 2017, volume rose 63%, delivery slowed to 15.1 days, the late rate tripled — and the promise Olist showed customers got shorter (23.2 vs 23.7 days). It made a harder promise exactly when it was least able to keep it. Sell the seller scorecard. The promise decomposes into handling + transit, so we can already tell a seller "your handoff adds 3 days to every promise we show your customers." Handling p95 ranges from 1.3 to 41 days across 461 sellers. That's a product Olist could charge for. Confidence intervals on p95. Five lanes have under 1,000 orders and their percentiles have real uncertainty that is_borderline only gestures at.

Video Link: https://www.loom.com/share/ebe11728892d4fa7b0dbceea77c10d39

Built With

• css
• emergence-craft-mcp
• fastapi
• html
• javascript
• nebius-token-factory
• nemotron-3-super-120b
• openai-sdk
• pydantic
• pytest
• python
• rich
• snowflake
• uv
• uvicorn