Project AMPM | Devpost

AM and PM
Minimap Tracking
Human ReID
Electronics
UI Pathfinding
Camera View

Inspiration

We wanted to build something in swarm robotics, not just a single clever robot. Most hackathon robotics projects are one machine doing one trick. We were more interested in the harder, more useful problem: how multiple cheap robots coordinate as a team to solve something a single robot cannot do alone.

Search and rescue, firefighting, and security all share the same shape of problem. A large area needs to be covered quickly, a person or hazard needs to be found, and no single robot can be everywhere at once. LiDAR and SLAM based localization can solve part of that, but the hardware cost and setup complexity put it out of reach for a weekend build. We wanted proof that a coordinated swarm could work with nothing more exotic than a camera and a field of printed markers, so the idea stays accessible instead of requiring a lab budget.

What it does

Each car in the swarm carries a single camera and localizes itself by detecting ArUco/AprilTag markers placed at known positions around the arena, the same way a person might navigate by street signs. Once a car knows where it is, it can compute the position of a person (marked via ReID) and plan a route to them around the obstacles (pylon cones) using A* pathfinding, then drive there with steering and throttle handled by a real time microcontroller. A dedicated travel router keeps every car talking to each other and to a base station on a private network to send information via UDP and TCP. A manual override mode lets a human take direct control at any time.

How we built it

Hardware: We used a Raspberry Pi 5 for high-level computing, an ESP32 for real-time vehicle control, and an ESC and steering servo for throttle and steering. We used a Logitech C920 webcam on one car and a Raspberry Pi Camera Module 3 Wide on another to compare USB and native Pi camera setups. A TP-Link TL-WR1502X router connected the cars and base station on a private network.

Software: We used Python for vision, localization, and path planning; OpenCV for AprilTag detection and pose estimation; YOLOv11 for obstacle detection and re-identification; and pyserial with JSON to send throttle and steering commands to the ESP32 at 20 Hz. The ESP32 converted these commands into PWM signals and used a watchdog to return the car to neutral if communication was lost.

QNX

-For a realtime system, all information is transferred over UDP to prioritize speed -Since all of our AI/ML is local, there is 0 service reliability -Hyper optimized to keep everything local (ex. tensor rt for reID, computer vision optimization with multithreading) -Cool use and combination of AI to identify a person in all angles, even when they leave frame and re-enter in a different position

Challenges we ran into

Nearly every challenge came from the gap between what a spec said and what the hardware actually did. Our steering and throttle pins were originally documented as GPIO 18 and 19, but the car was physically wired to GPIO 25 and 26. Flashing the documented pins against the real wiring looked exactly like a dead servo and a dead ESC, and cost real debugging time before a multimeter and some patience found the mismatch.

Detection performance was another surprise. The same ArUco tuning that ran comfortably at full frame rate on a desktop dropped to about six frames per second on the Raspberry Pi 5, and it took per stage timing, separating camera read time from detector time, to realize the bottleneck was the detector's threshold search, not the camera. Decimating the detection resolution fixed it without giving up range. Then every other form of computer vision had to be hyper optimized to run completely locally on the limited compute. We made conversions to tensorRT, CPU allocation manipulation, dynamic batching to optimize throughput.

On the mechanical side, our steering servo developed a visible jitter that turned out to be caused by rewriting the same PWM duty cycle every control loop, including every watchdog neutral command, thousands of times a second. Our keyboard driven manual control also stuttered until we accounted for the roughly half second delay before a terminal starts repeating a held key. We also discovered late that two different, incompatible serial protocols had been built in parallel by different people on the team, which needed to be reconciled before manual control could be wired into the autonomy stack. Even our travel router gave us a scare: it takes 9 or 12 volt USB PD or QC power specifically and will not turn on from a plain 5 volt power bank, which looked identical to a dead unit until we traced it back to the power source.

What we learned

We learned to trust instrumentation over intuition. A single frames per second number hid the real problem on the Pi until we looked at read time and detect time separately. We learned that documentation drifts from reality fast on a hardware project, and that pin assignments, protocols, and calibration values need to be treated as living facts to verify on the bench, not settled decisions to copy from a spec. We also learned that building for a swarm, not a single robot, changes the design from the start: the network, the shared coordinate frame, and the task allocation logic all need to be first class parts of the system, not something bolted on after a single car works.

What's next

We want to close the gap between demo and field readiness: fusing wheel encoders into localization so a car is not blind between marker sightings, moving from a few coordinated cars to real decentralized task allocation across a larger swarm, and reducing how much of the environment needs to be pre marked with tags. Longer term, the part worth building on is not any single robot but the shared coordination layer itself, which is what would let this become a real pilot with a search and rescue, fire, or security team rather than staying a hackathon demo.