Rescuebot | Devpost

Inspiration

When a building is on fire, a room is filled with smoke, or a structure might collapse, someone still has to go in and look for people. We wanted to build a robot that goes in first and scouts the space before rescuers do.

Our goal was a robotic scout that can be driven into a dangerous space, find people with on-device AI, and report what it sees back to the team outside, built from affordable off-the-shelf parts that work together as one system.

What it does

Drives anywhere on the floor: a mecanum-wheel chassis moves forward, sideways, and diagonally, and rotates in place, controlled from a browser with the keyboard.
Finds people in real time: a YOLO11n model runs directly on the Raspberry Pi AI Camera's own chip, so detection doesn't overload the Raspberry Pi 5's CPU.
Shows a live operator view: a browser dashboard streams annotated video with detection boxes, a people count, drive state, sensor readings, and system health.
Speaks alerts out loud: an onboard speaker announces warnings such as "Person detected on the left!" using ElevenLabs voices, with an offline text-to-speech fallback for when there's no internet.
Signals survivors and lights the way: when it spots a person, the robot sounds a buzzer and flashes its light so the person knows help is coming. In the dark, the light turns on automatically.
Maps the space: the lidar builds a live map of the area as the robot drives, and each person the camera finds is marked on it.
Logs every run for review: each detection (with its angle and lidar distance) and every 360° lidar scan are timestamped into one folder per run, and detection logs can be replayed in the dashboard.
Fails safe: the Stop button overrides everything, the robot disarms if the browser disconnects or goes quiet, and the motor controller has its own 500 ms watchdog.

How we built it

Compute: Raspberry Pi 5 for vision, the dashboard, and logging; an Adafruit QT Py ESP32-S2 for motor control, the IMU, and accessories, talking to the Pi over USB serial
Vision: Raspberry Pi AI Camera (Sony IMX500) running YOLO11n on its own chip
Sensing: Slamtec RPLIDAR C1 and Adafruit BNO055 + BMP280 (orientation, pressure, temperature)
Motion: Hiwonder mecanum chassis with the Adafruit Motor/Stepper/Servo Shield v2.3, powered by 6× AA NiMH cells (7.2 V)
Alerts: MAX98357A I2S amplifier and speaker, a buzzer, and an Adafruit NeoPixel Jewel
Software: Python, FastAPI, and a WebSocket dashboard; Picamera2 and OpenCV; Slamtec's RPLIDAR SDK; Arduino/PlatformIO firmware; ElevenLabs; ROS 2 Jazzy with slam_toolbox in Docker for mapping

Technical details

Mecanum drive. The dashboard sends forward, sideways, and turn inputs, and the ESP32 mixes them into four wheel speeds (positive means forward, right, and clockwise). The Python mock and the firmware must produce the same outputs for a shared set of test cases, so the dashboard and the real robot always agree.

Reliable detection. A detection only counts once it appears in 3 of the last 5 camera results, which filters out one-frame false alarms. People get a lower confidence threshold than rarer objects, and boxes older than 1 second disappear.

Locating people. Each detection's position in the image is converted to an angle using the camera's 66° field of view. The lidar reading at that angle gives an estimated distance to the person, and that's what places them on the map.

Automatic accessories. The camera also estimates light level. When a person is confirmed, the robot beeps for 2 seconds and flashes 5 times; if it stays below 10 lux for 5 seconds the light turns on, and it turns off again above 40 lux.

Challenges we ran into

Power: the Pi 5 wants 5 V/5 A, but our power bank gave 5 V/3 A, which caps the Pi's USB ports at 600 mA. Starting the camera and lidar with the ESP32 attached caused undervoltage and USB dropouts until we switched to a supply the Pi recognizes as 5 A.
Camera bring-up: the AI Camera wasn't detected until we reseated its ribbon cable, and we had to get the YOLO11n model into the IMX500's format ourselves.
Wheel mapping: some motors were wired reversed or on different ports than expected, so the robot turned the wrong way until we tested each wheel and remapped them in firmware.
Strafe drift: two of the wheels are weaker than the other two at our capped motor power, so the robot drifts when moving sideways.
Corrupted commands: the ESP32's own sensor output was overwriting parts of incoming drive commands, so we enlarged its receive buffer and added status reporting to find the problem.
Accessories: the buzzer turned out to be passive and needed a generated tone, and the NeoPixel Jewel wouldn't reliably read the ESP32's 3.3 V signal while running on 5 V, so we powered it from 3.3 V instead.
Lining up lidar and camera: the lidar's 0° didn't point where the camera looks, so early distance readings came out at 10 m for someone standing right in front of it.
Integrating across branches: our detection output didn't match the format the dashboard expected, so we standardized one shared format and added tests to keep it that way.

Accomplishments that we're proud of

Real-time person detection running entirely on a camera chip, with live annotated video in the browser
A full control path from browser to Pi to ESP32 to motors, with safety stops at every layer
A robot that reacts on its own when it finds someone: a spoken alert, a buzzer, and flashing lights
A live lidar map with every detected person marked on it
Detections and lidar scans logged together into timestamped runs that can be replayed in the dashboard

What we learned

How to split work between a Raspberry Pi and a microcontroller, and connect motors, a lidar, a camera, and alert hardware into one system
Running AI on the hardware: model formats, confidence tuning, and filtering noisy detections over time
Power budgeting, shared grounds, logic levels, and soldering
How much sensor fusion depends on calibration, since two sensors are only useful together if they agree on direction

What's next for Rescuebot

Calibrating lidar-to-camera alignment so every detection comes with an accurate distance
Autonomous search: the robot explores and navigates on its own with ROS 2 Nav2 (already running in simulation)
Hazard sensors (flame, gas) on the ESP32, with spoken warnings
Gemini descriptions of each detection snapshot (e.g., "a person lying near a doorway")
A ruggedized chassis for smoke-filled or collapsed buildings ```

Two things to check before you submit:

Camera bring-up: I changed "create it ourselves" to "get the YOLO11n model into the IMX500's format ourselves." If you actually trained or exported the model yourselves, go back to the original wording.
Strafe drift: the text says the drift still happens. If you fixed it, for example with per-wheel trim, add one line saying how.