Patch | Devpost

Patch Thumnail
Patch Rover

Inspiration

Ever leave the house and wonder, "Did I leave the stove on? Did I forget to feed the cat?" or spent twenty minutes hunting for you keys?
Patch reports back with an answer, giving you peace of mind whether you're in the room next door, or across town.
Instead of controlling a robot through rigid commands, we wanted to build something that could see its surroundings, listen to what someone says, ask questions, and act on natural human intent

What it does

Patch is a small mobile robot that can see, hear, and understand what you ask, then go and act on it.
Find missing things: The rover searches the room using its camera and reports back on where it spotted the item.
Check on your home: Ask "Is the fridge closed?" or "Did I leave the stove on?" The rover drives over, looks, and gives you a clear answer.
Talk to it naturally: Uses natural-language instructions from the user are used to interpret intent and guide the robot’s actions.

How we built it

Patch is a Raspberry Pi 5-powered rover with cameras, motor controllers, and servos, and a multimodal AI model acting as its brain.

Raspberry Pi 5: The main computer. It streams live camera video and audio to the AI, then turns the AI's decisions into actions.

Huawei OMNI multimodal AI API: The intelligence layer connecting what the robot sees, hears, and understands. It looks at the live camera feed and the user's speech, identifies objects and where they are in the room, and figures out what the user wants. When a request is ambiguous, it asks follow-up questions.

ESP32 motor controllers and servos: The Pi sends movement commands to the ESP32s, which drive the wheels and servos so the rover can move, aim its camera, and get a good look at things.

Camera: Give the rover its eyes for both searching and status checks.

The result is a real-time loop: vision → speech → language reasoning → physical action. Instead of fixed commands or pre-programmed responses, the rover reacts to what's actually happening in front of it.

Challenges we ran into

Integrating and debugging software and hardware components together, one of the motors was weaker than the other
Qwen-Omni makes every actual match decision in our detection loop, but it's a multi-second, rate-limited call — not something you can fire ten times a second. Polling on a fixed timer and calling OMNI on every frame worked, but wasted calls on empty frames with no cheap way to tell "something's here" from "nothing's here."
We added a local YOLOv8n model as a pre-filter: it runs continuously and cheaply, firing OMNI only on a new detection or after a short cooldown, instead of blind polling. YOLO never makes the match decision itself — only OMNI does — since its confidence is about a generic object class, not the user's specific item.
The catch: YOLO only knows COCO's 80 categories, and most real search targets (keys, a specific mug) aren't in it. So we map free text to a COCO class when possible and gate on YOLO; when it doesn't map, we transparently fall back to unaccelerated timed sampling — OMNI still does the whole job either way.
One surprise: YOLO alone spiked CPU to ~750% from OpenMP/MKL threads spin-waiting between calls, not call frequency. Forcing passive waits (OMP_WAIT_POLICY=PASSIVE, KMP_BLOCKTIME=0) dropped that under 40% with no behavior change.