Apex FSD | Devpost

Sample frames w/ game telemetry
Cleaning Input for Model
Single Lap Recording Output
Large Recording Dataset Output
Recording grind

Inspiration

Most AI driving projects end at a function call. We wanted ours to end with a steering wheel turning on its own. Self-driving used to mean hand-built pipelines: one system for lane detection, another for path planning, and another for obstacle avoidance. Deep learning collapsed all of that into one network. In 2016, NVIDIA's PilotNet showed a single convolutional neural net could learn to steer straight from camera pixels. Tesla's modern FSD v12 has since taken the same end-to-end idea onto real roads. One of us is a racing fanatic with a force-feedback sim rig under their desk, and a hackathon is where big ideas get chased. So we asked: could we build that same pixels-to-steering brain in a weekend and have it physically turn a real wheel, one a human can grab back at any moment?

How we built it

Model: a CNN based on NVIDIA's PilotNet architecture: 5 convolutional layers that learn road features (edges, lane lines, curves), then fully connected layers that turn them into driving outputs. We added custom late fusion of Forza's live telemetry: the car's speed joins the image features right before the decision layers, because the same corner needs different steering and braking at different speeds.

road crop (66×200) ──► 5 conv layers ──► image features ─┐
Forza speed (telemetry) ─────────────────────────────────┴─► FC 100 → 50 → 10 ──► steering, throttle, brake

Wheel loop: Forza can only read one controller. HidHide hides the real wheel from Forza, so the game only sees vJoy, a virtual wheel that the neural net can control and also mirror and intake data from the wheel. This allows us to switch between manual driving and FSD without pause, just like on real autonomous vehicles.

TMX wheel ──(angle, pedals)──► our runtime ──► vJoy ──► Forza
    ▲                              │
    └──── motor turns the wheel ◄──┘ ◄── AI prediction (laptop, over Ethernet)

Stack: Python, PyTorch, DXcam, OpenCV, PySDL2, vJoy + HidHide, Forza Data Out telemetry, two PCs over a direct Ethernet link.

Challenges we ran into

Syncing frames with telemetry. Every frame, wheel reading, and telemetry packet is timestamped on one clock. We measured that the screen shows the game 66 ms after telemetry reports it by matching gear changes on screen against gear changes in the telemetry.
Hot-lap data doesn't teach recovery. We mostly recorded fast laps, minimizing lap time and chasing apexes. So the model never saw what to do once it drifted off the line, and it couldn't recover after going off the road. It also never learned to use the road markings as guidance but more or less memorized the patterns around each corner.
Overfitting. We tuned the driving (steering gain, throttle ramping, and a corner speed limit) and added DAgger: when the AI makes a mistake, we rewind, take over, and drive it properly, and those corrections become new training data.
Driving at signs instead of lanes. The network learned a shortcut: steer toward the big "HORIZON" signs rather than follow the lane markings. Crash run-ups in our recordings had taught it that. We cut them from the data automatically. -Lack of Data. We trained all of our dataset during the hackathon, which accumulated up to around 3 hours of driving on one race/map. The model's ability to generalize is poor, but we did try testing it on races it had never seen before, and it was able to steer around some tight corners and perform decently on straights.

Accomplishments that we're proud of

The wheel actually moves. A neural network's steering decision ends up as a real Thrustmaster TMX turning by itself in front of you, driven by a 100 Hz PD loop with static-friction compensation. Getting motor tracking error from 134° down to 9° by throttling commands to the wheel was the moment the whole thing clicked.
A real end-to-end pipeline in a weekend: screen capture → telemetry sync → recording → automatic data cleaning → PilotNet-style training → inference over Ethernet → vJoy → Forza → physical wheel → human takeover → DAgger corrections back into training.
The human is always in charge. Grab the wheel, tap a pedal, or press a button, and control is yours within a fraction of a second, and every disengagement is logged with its cause. Stale frames, telemetry, or network? The AI backs off on its own.
Honest data engineering. We measured the screen running 66 ms behind telemetry by matching gear changes and built automatic crash/rewind/pause detection with adaptive buffers so the model stops learning from our mistakes.
A model that beats its baselines on day one. Our first real GPU run cut held-out steering error well below both the always-zero and training-mean baselines, with a live HUD showing the AI's intended path over the game view.

What we learned

Hot laps don't teach recovery. We recorded fast, clean laps, so the model never saw what to do after drifting off the line. DAgger (rewind, take over, drive it properly, retrain) was the fix, and it reframed the whole project: the dataset is the product.
Speed belongs in the model. Late-fusing telemetry into PilotNet made a real difference: the same corner needs different steering and braking at 80 km/h than at 200 km/h.
Keep two machines honest. A clean contract between the inference laptop and the game/wheel PC, plus per-failure logging, saved us hours of guessing over an Ethernet cable.
In machine learning, the first 99% comes quickly, and the last 1% takes all the work. It was difficult to optimize our inputs and weights through trial and error given the tight time constraints to retrain the model.

What's next for Apex FSD

Interactive Voice for Driving Agent. You can command it to drive faster or slower or turn more aggressively in real time as it dampens or strengthens the inference controls inputs. -More advanced inputs. Right now, the model is only trained with the speed and wheel input for every frame. We haven't tried giving it more inputs, like the wheel turning acceleration or feeding it the speed for the past 5 frames, giving it more context.
Take it out of the game. The whole stack is already camera-in, steering-angle-out. The next step is swapping Forza's screen capture for a dash cam and the game's telemetry for a car's OBD-II speed, then running the model in shadow mode on real roads: it predicts, the human drives, and we measure the gap. Same recorder, same DAgger loop, real data.
From shadow to assist. Once shadow-mode predictions agree with a human driver, the same force-feedback controller that turns the TMX can become a gentle steering nudge, with the same grab-to-take-over logic we already have.
Better model, more data. Fix the right-turn underestimation we saw in the baseline, add training-time mirroring, weight steering loss properly, and record on more than one circuit so we're evaluating on routes the model has never seen.
Longer horizon. Lane changes, overtaking, and navigation inputs, then a reinforcement learning layer on top of the imitation baseline.