feetball

We took real World Cup tracking data and taught virtual players how professionals actually move — then built a live 3D broadcast around them, with AI commentary and 1,100 autonomous agents betting on the result.

Not scripted. Not animated. Learned.

The hook — five things that make this cool

Players play like their real selves. Swap Gakpo into your XI and he finishes like Gakpo — his pace, passing, and shooting are all derived from real tracking data. 829 real players, real attributes.
Thirteen ML models run per player, every tick, in your browser — while holding 60 fps in a 40,000-seat stadium. Movement, passing, shooting, tackling, goalkeeping: each one behaviourally cloned from real football.
The sim improves itself. A headless optimizer plays thousands of matches and retunes the AI until it measurably converges toward real-football statistics — pass completion, pass length, off-ball spacing, possession balance.
1,100 AI agents in the stands are betting live. Click any one and inspect their wallet, positions, and reasoning. They watch the same event stream you do and trade a real prediction market on top of it.
Broadcast-quality AI commentary at 0 ms latency. "…he shoots… GOAL! …It's Gakpo!" — stitched live in the browser, with near-zero cost per event.

All of it grown from one match of PFF FC's FIFA World Cup 2022 tracking data (Senegal vs Netherlands, 30 positions/second).

Inspiration

Sports games fake it. FIFA runs on hand-tuned animation logic, and broadcast "AI stats" are mostly marketing. We wanted the opposite: take real tracking data — every player's position 30 times a second — and actually teach virtual players how professionals move, pass, shoot, and defend. Then build a whole broadcast around them: a live 3D stadium, an AI commentator, and stands full of autonomous agents betting on the outcome.

The goal was simple and a little absurd: grow an entire football broadcast out of data.

Powered by ElevenLabs & Freesolo

Two pieces of tech do heavy lifting, and both are used in ways that play to exactly what they're best at.

ElevenLabs — the live AI commentator.

How we use it: We don't call the API per event — a pass fires ~once a second and the ball is in the net ~400 ms after a shot, so a live round-trip would always lose the race. Instead we use ElevenLabs offline to pre-generate reusable clip banks — reaction phrases ("…he shoots… GOAL!") and per-player name clips — then concatenate them in WebAudio at runtime. The natural broadcast pause hides the seam. A Next.js route lazily synthesizes and caches name clips for the long tail of the 829-player database.
Why it's good and suitable: ElevenLabs' voices are broadcast-quality and expressive enough to sell a real football-commentator feel — the thing that makes the whole broadcast believable. Precomputing against its API gives us 0 ms runtime latency and near-zero per-event cost while keeping that quality, and lazy synthesis means we get full 829-player name coverage without generating everything up front. It's the rare TTS that's convincing enough to carry a live sports broadcast.

Freesolo — the AI betting agents' brains.

How we use it: Named market agents can optionally run a Freesolo SFT→GRPO betting policy — a trained policy that reads the same live event stream a human would and decides what to bet, with every decision schema-constrained before it touches the ledger. It's opt-in per agent, with a deterministic rules policy as the fallback so the market never depends on it.
Why it's good and suitable: Freesolo's SFT→GRPO pipeline is built for exactly this — training a decision policy that improves against a reward, which maps cleanly onto "bet well against the market." Making it opt-in per agent keeps 1,100 bettors cheap (one event never fans out into 100 paid calls), and the schema-constrained output plus rules fallback means a learned policy can be dropped in safely behind a hard validation boundary. It gives our agents genuinely trained judgement without putting the ledger at risk.

What it does

A full 3D stadium in the browser. Procedurally generated MetLife Stadium — ~40k instanced seats, crowd, roof, LED ad boards — at 60 fps. Click any seat to sit in it, fly the camera anywhere, or grab a player and control him yourself (WASD + charge-and-kick).

22 data-driven players. Movement, passing, pass accuracy, shot selection, shot placement, tackling, goalkeeping, and off-ball positioning are each a small learned model, run per player per tick.

Movement, off-ball runs, and defending → neural nets (small MLPs).
Passing, pass accuracy, shots, tackles, and goalkeeping → tiny logistic/linear regressions.

829 real players with real attributes. Pace from top tracked speed, passing from completion %, shooting from conversion — all derived from event data and shrunk toward role priors.

A self-improving loop. A headless Cross-Entropy Method (CEM) optimizer simulates thousands of matches, scores how realistically each plays against targets from the real data (pass completion ~72%, mean pass length ~12 m, possession balance, off-ball dispersion…), and retunes the sim's behaviour weights toward how real teams actually play.

AI broadcast commentary (ElevenLabs). Pre-generated clip banks stitched in the browser at 0 ms latency, with lazy server-side synthesis for the long tail of player names.

A prediction-market economy. A dependency-free Node backend runs five market types (match winner, total goals, exact score, next goal, player to score), an authoritative double-entry credit ledger, event-driven settlement, risk controls, and deterministic replay. 1,100 autonomous agents — 100 named, risk-profiled personas plus 1,000 lightweight behavioural agents — watch the live event stream and bet.

An agent-first API. External AI agents discover a JSON-Schema manifest (GET /api/agent/manifest) and act through one validated endpoint (POST /api/tools/call) with idempotency keys and stale-price protection. The market was built to be played by agents.

Plus: replay of the real Senegal–Netherlands match with auto-detected highlights, live decision overlays (each ball-carrier's shoot/pass/dribble scoring + vision cones), and an admin ops dashboard.

How we built it

Data pipeline (Python + NumPy). We stream the 819 MB tracking file once, offline, into a compact binary replay (~8 MB: 10 fps, int16 centimetre-quantized, 121 bytes/frame) plus per-player attribute tables. Fifteen scripts in tools/ — one preprocessor plus a training/analysis script per skill — behaviour-clone each behaviour. The movement policy takes a player's egocentric, attack-normalized view of all 22 players and the ball, and predicts his velocity over the next 2 seconds. Pure NumPy MLPs, no PyTorch, exported as JSON weights.

Simulation + 3D (Next.js 16, TypeScript, three.js / React Three Fiber, zustand). Config-driven procedural geometry from a single source of truth. React handles structure; refs and useFrame handle the hot path, so nothing re-renders during play. The engine sits behind a MatchSource seam, making the mock source, the real-match replay (PffMatchSource), and the live ML sim (SimMatchSource, ~2,000 lines) interchangeable with zero changes to the 3D layer.

Self-improvement. The browser sim and a headless harness share one SimConfig. A CEM optimizer tunes its behaviour weights against realism metrics from the real data and writes the winner back to public/models/sim_config.json, which the game picks up automatically.

Commentary (ElevenLabs). Calling TTS per event is wrong on both cost (a pass fires ~once a second) and latency (the ball is in the net ~400 ms after the shot). So we pre-generate reusable reaction and name clip banks once and concatenate them in WebAudio — the natural broadcast pause hides the seam perfectly. A Next.js route lazily synthesizes and caches name clips for the long tail of the 829-player DB.

Markets (Node, zero runtime deps except ajv). Event-sourced match feed with idempotent sequence keys, a full market lifecycle (draft → open → suspended → settled/voided), a double-entry VirtualLedger as the single authority on balances, seeded agent personas across eight strategies, and a provider registry with deterministic mocks for every external boundary (Stay22, ElevenLabs, MongoDB, Solana…). Named agents can optionally run a Freesolo SFT→GRPO betting policy (schema-constrained, opt-in per agent) with a deterministic rules fallback.

Testing. node:test across the backend, vitest for the sim math, and a Playwright e2e suite (desktop + mobile Chromium) that boots an isolated server and validates the agent-facing API contract.

Challenges

Challenge	Solution
819 MB of tracking vs the browser	Streaming O(1)-memory preprocessor, 30→10 fps downsampling with client-side interpolation, int16 quantization → 100× smaller, zero visual cost
Cloned agents all chase the ball	Naive cloning gives 22 kids swarming one ball. We attack-normalized features, added off-ball dispersion + "distinct possessors" to the realism reward, and let CEM tune against structural metrics, not just imitation loss
Live TTS is physically too slow	A commentary round-trip loses to a 400 ms shot every time → concatenative clip banks push synthesis fully offline, runtime cost to 0 ms
1,100 bettors must stay cheap	One event must never fan out into 100 paid LLM calls → model reasoning is opt-in per agent, everything falls back to a deterministic policy, every decision is schema-validated before touching the ledger
Money math has no "close enough"	Concurrent bets, settlements, voids, refunds forced a real double-entry ledger with an immutable audit trail, not a balance field
Freesolo's CLI is Unix-only (imports `fcntl`)	Routed the whole training workflow through WSL

Accomplishments we're proud of

Players that visibly play like their real selves — real pace, real finishing — from a single match of tracking data.
Thirteen learned models running per player, per tick, holding 60 fps in a 40k-seat stadium.
A training loop whose reward is behavioural realism — we can watch the sim measurably converge toward real-football statistics.
A complete, auditable prediction-market economy with 1,100 autonomous participants and essentially zero backend dependencies.
An agent-first API: any external LLM can discover the contract and trade safely.
Broadcast-feel commentary with zero runtime latency and near-zero per-event cost.

What we learned

Behavioural cloning gets you 80% of the way. The last 20% — division of labour, off-ball structure — needs explicit realism objectives and black-box search.
Quantization and downsampling are magic. 1 cm precision is invisible; 100× smaller changes what's possible in a browser.
For live audio, when you synthesize matters more than how well. Precompute beats streaming.
Clean seams (MatchSource, provider registry, SimConfig) let three of us work on 3D, ML, and markets in parallel without stepping on each other.
Double-entry bookkeeping exists for a reason.

What's next

Full deep-RL self-play on top of the cloned priors, plus multi-match training data.
Contextual LLM commentary — form, rivalries, stakes.
Real identity + authorization for the market, and a Solana settlement provider behind the existing ledger boundary.
Mapbox real-world geo mode (already built, currently off): drop the stadium anywhere on Earth.
Head-to-head: your tuned SimConfig vs mine.