Squawk

Squawk is Cursor for air traffic controllers: an AI copilot that plans the sky, listens to the radio, and makes sure every instruction is heard and flown correctly.

Inspiration

Air traffic control still runs the way it did in 1970. Planes fly fixed highways in the sky, one person juggles a dozen aircraft by hand, and every instruction goes over a crackly shared radio. Pilots repeat each instruction back, and the controller is supposed to catch a wrong readback. Busy controllers miss some. Research puts readback errors at 1 to 2 percent of transmissions, and a missed one means a plane at the wrong altitude or heading. Meanwhile the US is thousands of controllers short and traffic is forecast to double.

We wanted to build the second set of ears that never misses, and then go further: if the listening side is reliable, you can plan tighter, more efficient routes and trust that they get flown.

What it does

Squawk runs against a flight simulator we built, with AI pilots that answer by voice.

• Plans a conflict-free path for every flight at the same separation margins used today (5 NM, 1,000 ft), using search and geometry, no ML.
• Replans the moment anything changes: drop a fighter jet, a storm, or a drone into the sector and only the affected flights reroute, live.
• Listens to the radio with a Whisper model we fine-tuned on real ATC audio. Stock Whisper is nearly useless on tower radio; ours is not.
• Validates every pilot readback against the instruction at the concept level, not word level, and alerts on wrong value, wrong runway, wrong aircraft, missing readback, and more.
• Verifies on radar that each plane actually does what it read back. A correct readback followed by wrong flying still gets caught.
• Investigates the messy cases with an agent. When audio is garbled or two callsigns sound alike, an agent re-listens, searches the radio history, checks the radar track and nearby traffic, and commits to alert, dismiss, or watch, with its reasoning shown on screen.
• Real traffic. Load an actual hour of flights over Western Europe, the UK, Toronto, or the US Northeast from adsb.lol and replay it through Squawk.

The plane obeys what the pilot said, not what the controller meant. Turn Squawk off, let a wrong readback slip, and you watch the plane fly into trouble on the map.

The agent (Rox: Best AI Agent)

Most of Squawk is a fast, deterministic pipeline: speech to text, normalize, parse, compare, under 2 seconds per transmission. That handles the clear cases. The agent exists for everything else, and it is where the messy real-world data lives.

Radio audio is noisy, clipped, accented, and shared by everyone on frequency. Two Air Canada flights with one digit apart in their callsigns are both listening. Pilots shorten, paraphrase, and mumble waypoint names that speech models have never seen. Radar can contradict what the radio said. The agent's job is to decide, in under 5 seconds, whether a busy human needs to be interrupted, and to be right.

It is a hand-rolled tool-calling loop on a Baseten-hosted model, no agent framework. It picks its own tools based on what the last one returned:

• relisten: re-transcribe the clip and get alternative hypotheses.
• frequency_history: what was this aircraft told, ranked by how well each past message matches the garbled reply.
• aircraft_track: what has this plane's altitude and heading actually done over the last 30 seconds.
• nearby_aircraft: who is within 30 NM, so a similar callsign can be checked.
• sanity_check: is this value plausible, and if the waypoint name is garbled, which real fix is closest.
• Terminal actions: raise_alert, dismiss with a reason, mark_uncertain, or watch the plane on radar and let the evidence come to it.

It never ends silently. It is capped at 4 tool calls. Every step is emitted to the screen as it happens, so the judge sees the investigation, not just the verdict. In one live run tonight the model chose to query the radar track on a 0.44-confidence readback, got back "climbing over 30 s, 35025 to 35775 ft", and used that to decide. Nobody scripted that order of steps.

Baseten (Best Use of Baseten)

Both models we trained live on Baseten, and every language model call goes through Baseten's OpenAI-compatible API. No other inference provider.

• Whisper fine-tune on H100s. We trained on the public jacktol/atc-dataset, real European tower radio, using Baseten's training jobs. Word error rate on 1,000 held-out real clips went from 0.708 stock to 0.159 tuned. A proving run of whisper-small took 8 minutes end to end; the full run followed. The tuned model is deployed as a Truss with faster-whisper and the app uses it when ASR_MODEL_URL is set, with a stock-versus-tuned toggle on screen for the same clip.
• Readback checker. A RoBERTa cross-encoder trained on synthetic controller/pilot pairs: 0.894 accuracy, 0.085 false alarm rate, 7 ms per pair. Job config for Baseten is in the repo.
• The resolver and the world-builder agent run on GLM-5.3-Fast through Baseten, with a smaller model for structured extraction. We chose small tuned models over one large prompted one for latency and cost, and we measure both.
• A simulator that generates its own training data. Every AI pilot exchange logs the true clearance, the spoken text, the injected error, and the audio, so the next fine-tune has more labelled radio.

Elasticsearch (Elastic: Find the Signal)

Elasticsearch is the agent's context layer. Every transmission, clearance, verdict, resolver step, and radar frame streams into Elastic as it happens, one document per aircraft per second while the sim runs. The agent's tools query it directly instead of reading in-process lists:

• BM25 over the radio log (multi_match with fuzziness: AUTO), ranked against the garbled readback, so the exchange the pilot was probably answering comes first.
• geo_distance on a geo_point around the aircraft's latest position, collapsed by callsign, for who is nearby.
• Time-range queries over radar frames, reduced to an altitude and heading trend, for what the plane is actually flying.
• Fuzzy matching on waypoint names, so a misheard "estr" resolves to ESTIR.

This is retrieval that closes the loop with an action: the agent searches, then alerts, dismisses, or watches. Every step it takes is itself indexed, so the audit trail is queryable in Kibana with ES|QL, and a second screen shows readback errors by type from one STATS query. We deliberately did not add vectors: ATC phraseology is short and fixed, BM25 plus fuzziness finds the right thing, and a vector call would add latency to a 5-second budget. With no ELASTIC_URL set, the app falls back to in-process state and behaves identically, so Elastic is a real dependency in the demo and never a single point of failure.

OpenAI

Whisper is the ear of the whole system. We started from OpenAI's open-weights Whisper checkpoints and fine-tuned them on real ATC radio, which took word error rate on 1,000 held-out clips from 0.708 stock to 0.159. Every transmission on the frequency, controller and pilot, goes through Whisper before anything else sees it, and the agent's relisten tool re-runs Whisper with alternative hypotheses when the first pass is unclear. Whisper is also the reason the fine-tune matters: stock Whisper hallucinates loops of digits on short noisy clips and cannot hear made-up waypoint names, and the tuned model fixes both.

We also used Codex during the build to move faster on the parts that were not the core idea: scaffolding the FastAPI and Next.js layers, writing tests against fake services so the pipeline could be verified offline, and drafting the WebSocket protocol docs that kept the frontend and backend in sync across four people.

How we built it

• Speech: Whisper fine-tuned on Baseten H100s, served on Baseten, faster-whisper locally as fallback. Silero VAD, a phraseology prompt, callsign snapping against the active list.
• Readback checker: RoBERTa cross-encoder plus a deterministic rules layer. Grammar parser first, LLM only as fallback.
• Resolver agent: hand-rolled tool loop on Baseten, Elasticsearch as memory, 4 calls, 5 seconds, always one terminal action.
• Planner and simulator: flat-plane kinematics, prioritized planning with a vectorized numpy conflict test, replanning with a frozen window and an emergency layer, Monte Carlo evaluation across three arms (fixed routes, Squawk without validation, Squawk with validation).
• AI pilots: template readbacks with eight injectable error types, ElevenLabs voices through a narrow-band radio filter. The pilot never reads text; it only hears.
• Backend: Python, FastAPI, WebSockets. Frontend: Next.js, deck.gl on MapLibre for a tiltable 3D map with altitude, trails, and a flight strip per aircraft.

Challenges we ran into

• Stock Whisper hallucinates on short noisy clips and cannot hear made-up waypoint names like ESTIR. Callsign snapping, a phraseology prompt, fuzzy fix search in Elastic, and the fine-tune.
• False alarms cost trust. A naive checker flags every mishearing. Four defences: the n-best rule, callsign snapping, radar verification, and the agent.
• The agent has 5 seconds. With a real model each step costs one to two, so it has to choose its searches well, and Elastic queries had to come back in milliseconds.
• Our first efficiency number was wrong. Simulated scenarios showed 7 to 8 percent fewer miles, but real cruise traffic already flies nearly straight. On the replay it is about 0.5 percent. We report both and say which is which.
• Sim audio is not real radio. The fine-tuned model was worse on our synthetic pilot voices at first, so we mixed simulator audio into training.
• Serverless Elastic makes new documents searchable a few seconds after writing. One refresh call after loading a scenario, and the background stream accepts the lag.

Accomplishments that we're proud of

• Whisper WER on 1,000 held-out real ATC clips: 0.708 stock to 0.159 fine-tuned, on Baseten.
• Monte Carlo, demo scenario, 20 runs: fixed routes 0.34 losses of separation per flight hour, Squawk 0. Dense scenario at 5 percent readback errors: fixed 154, Squawk without validation 1, with validation 0. That last row is the thesis: validation is what makes a tighter plan safe.
• Readback checker: 0.894 accuracy, 0.085 false alarm rate, 7 ms per pair.
• A live agent that chose to search Elasticsearch on its own and used the answer, with the trace on screen and in Kibana.
• The whole loop is real audio. Nothing reads the other side's text.
• Every number on the scoreboard is measured live in that session. None are seeded.

What we learned

• Concept-level comparison beats word-level. "One eighty to DME four" can be a correct readback of a completely different sentence.
• Surveillance data rescues speech recognition. Knowing who is on frequency fixes most callsign errors before any model sees them.
• Short, fixed phraseology is where keyword search plus fuzziness wins over vectors.
• Small fine-tuned models on the right platform beat a large prompted one on both latency and accuracy for a narrow domain.
• A tool-calling loop in a hundred lines beats an agent framework when latency is the constraint.

What's next for Squawk

• Separation and density sliders that show the efficiency-versus-margin trade-off curve honestly.
• Feed the agent's verdicts from Elastic back as training data for the checker on Baseten.
• Replay full days across regions and mine the Elastic log for where airspace wastes fuel: which shortcuts get granted, which holds recur.
• Data-link mode for bulk instructions so voice is reserved for what matters.
• A trainee mode, since AI pilots that make deliberate mistakes are exactly how controllers are assessed today.

Squawk does not replace the controller. It is the second set of ears that never gets tired.

Built With

• baseten
• deck.gl
• elasticsearch
• elevenlabs
• fastapi
• huggingface
• kibana
• maplibre
• nextjs
• numpy
• pydantic
• python
• pytorch
• react
• roberta
• typescript
• websockets
• whisper