We gave ChatGPT, Claude, and Grok control of a real Toyota Corolla 🚗, steering/gas/brakes, no human driving (just a foot over the brake).
Only one model was able to complete our entire driving course. Introducing DrivingBench. 🔥⌛️🏁
Help us consider how to evaluate Jev by @typesafeai on DrivingBench! Since it can't do perception/vision it needs to be paired with some (fast) protocol to turn the car camera into a structured state or text for Jev to handle
Trying to evaluate Jev! But it’s not easy since Jev doesn’t natively take vision inputs (so the demos you’ve seen of it playing games etc required giving whole state… which we can’t do.) Trying to build out a pipeline that utilizes it and its speed well, pairing it with Luna for