Fallback | Devpost

The problem

Falls are the number one killer on construction sites , more workers die from falls than from anything else. When a worker goes down alone, nobody knows until someone happens to walk by. The clock is the whole problem.

Every worker-monitoring system we looked at sends video to the cloud and waits for an answer. That means the alarm depends on the network. On a construction site, the network is the least reliable thing there.

What it does

Fallback puts the safety decision on the camera itself.

A node running QNX on a Raspberry Pi 5 watches the site, runs pose estimation locally, confirms a fall, and fires a GPIO output — that pin is a klaxon in production. The decision never touches the network.

Separately, a cloud layer serves the humans. A responder dashboard shows the worker's position on a site map, an A* rescue route from the entrance around obstacles, the frozen frame from the moment of detection, and a grounded AI briefing.

Two tiers, on purpose. The safety decision cannot wait. A supervisor can.

How we built it

Detection. YOLOv8n-pose, 17 keypoints, on-device. Fall logic is rule-based and derived from real captured data — we labeled standing, kneeling, and fallen frames from our own camera and computed the thresholds from the actual distributions.

State machine. UPRIGHT → DOWN → FALL_CONFIRMED after 5 seconds sustained → RECOVERED. Frames where the detector loses the person are UNKNOWN, never "upright" — a body on the floor gets occluded constantly, and treating those frames as safe would break the timer.

The alarm latches. Once confirmed, it stays confirmed until a human clears it. During a real incident the frame fills with responders and detection falls apart; a spurious "recovery" must never un-fire an alarm.

Localization. Four taped floor markers, homography, ankle keypoint → real-world metres. Routing. A* on a 5 cm occupancy grid with obstacle inflation for responder clearance. Briefing. Gemini 2.5 Flash, strictly grounded. It receives measured facts and the detection frame, and is explicitly forbidden from inventing injuries, vitals, or geometry. Cloud. FastAPI + Server-Sent Events, Next.js dashboard, deployed on a DigitalOcean droplet.

Challenges

Thresholds cannot be guessed. Our first rule was the textbook one — bounding box wider than tall. It fired on zero percent of real falls in our camera geometry. We threw it out and derived new thresholds from labeled data.

The tracker followed the wrong person. At low confidence, the model detects furniture as people. Worse: a bystander standing near the camera has a larger bounding box than a worker lying on the floor. Our system was watching the narrator instead of the victim. We had to lock the subject once a fall begins.

Kneeling was harder than falling. A crouching worker must not trigger the alarm — a system that cries wolf gets switched off on day one. We measured the longest false-positive window on real kneeling data (3.4 seconds) and set the confirmation threshold above it.

A known blind spot, honestly kept. A body lying head-toward the camera is foreshortened and geometrically indistinguishable from a crouch. There is no threshold that separates two things the pixels don't separate. We loosened recall and raised the confirmation window rather than fabricate a boundary that isn't in the data.

What we're proud of

Every threshold in this system came from data we captured and labeled ourselves. Nothing was tuned by feel.

The AI briefing refuses to guess. It will describe a worker's visible body position; it will not claim they are unconscious or injured. A fabricated medical assessment handed to a first responder is worse than no briefing at all.

The full loop runs over the public internet: camera → on-device decision → cloud → a responder's phone showing MAN DOWN, the map, the route, and the briefing.

What we learned

The safety decision and the coordination layer have different requirements, and conflating them is the mistake the industry makes. Milliseconds on-device. Seconds in the cloud. Never the other way around.

What's next

Fine-tuned classifier to close the head-toward-camera blind spot. Multi-node fleet — one node per zone, cost is a Pi and a camera. A second camera at 90° eliminates the foreshortening ambiguity entirely. PPE detection on the same pose backbone.