DriverQ

A semantic search engine for autonomous driving videos

Comment

Inspiration

With more and more autonomous vehicles being deployed in the real world, it is important to reliably evaluate their safety.

Safety engineers need to go through terabytes of driving videos and training data to find cases of the scenarios they want to test.

If only there were a way to rapidly search through these driving videos to find exactly the clip you are looking for...

What it does

DriverQ lets you search for a driving clip matching a scenario you describe in natural language (e.g. "driving in a school zone in daylight with pedestrians nearby").

You get a ranking of the best matching results, as well as the vehicle data collected for the clip (speed, turning data, etc).

How we built it

  • I leveraged the multimodal capabilities of the Lllama 4 Maverick Model to generate textual descriptions of the driving scenes. Fast inference was achieved by using this model via the Groq API.
  • I used ChromaDB to store embeddings of these textual descriptions to help narrow down the search space of driving clips to consider when finding the best matching results.
  • The nuScenes public driving dataset was used for the example clips for this demo. The dataset contains the videos as well as the corresponding vehicle data gathered from sensors onboard. These help augment the quality of the search results and provide more filters.

Challenges we ran into

  • Computing the speed data of the cars to tag each video clip with; this was important so that users can do speed-related filtering/searches too
  • Achieving faster search speeds - a lot of big data to search through!

Accomplishments that we're proud of

  • Building a project that helps ensure the future of autonomous driving is a safe one
  • Achieving fast searches by incorporating Groq for fast inference and ChromaDB for fast retrieval

What we learned

  • Self-driving car datasets are huge
  • The multimodal capabilities of Maverick are impressive; I was able to get accurate scene descriptions for each driving clip. Groq's low latency makes this super usable for my app.
  • Vector databases can help funnel out less relevant datapoints and drastically improve my app's search speed, which is doubly important for large self-driving car datasets

What's next for DriverQ

  • Deploying it on even larger datasets
  • Incorporating multiple camera views
Share this project:

Updates