Ironminer

Hot take superintendents have a hard job they're constantly putting out fires while having to plan out massive structures and evaluate workers. Yet many of this can be augmented to allow a superintendent to put on spotify and get work done 10x faster, if they had a 2nd brain with accesss to all parts of the system. So we built that 2nd brain.

Inspiration

In construction, cameras are everywhere, but managers are not manually reviewing hours of footage to find safety issues, delays, or coordination problems. Most video is stored and only checked after an incident happens. We saw an opportunity to turn passive video into active intelligence that surfaces what truly matters on a job site.

What it does

• Is the openclaw for superintendents
• AI powered spatial intelligence system built specifically for construction environments
• Analyzes live and recorded site footage
• Detects unsafe behaviors, equipment conflicts, restricted zone violations, and workflow inefficiencies
• Replaces raw video review with structured insights and clear outputs
• Delivers incident summaries and real time alerts so managers can act quickly
• Helps prevent issues before they escalate on site
• Automatically sorts and assigns teams for dynamic tasks
• Creates immersive 3D Gaussian environments for rapid site review by management
• Provides quality and performance assessments for each worker using profile data and video evidence
• Enables live tagging of objects and people in real time for future reference

How we built it

We started by experimenting with GPUs to run vision-language models locally and tried Meta's SAM 3 for segmentation and scene understanding. After iterating on what worked best for construction footage—where occlusion, dust, and dynamic layouts make raw segmentation noisy—we switched to Pegasus by Twelve Labs for video understanding. Pegasus gave us reliable, contextual summaries of site activity that we could then feed into our safety and productivity agents. The pipeline today: video is summarized with Pegasus (or a cached summary from it), we extract structured zones and events from that narrative, run deterministic OSHA-style safety checks and congestion analysis, and use an LLM to generate executive summaries and recommendations. Live streaming is supported via LiveKit so managers can monitor feeds and talk to workers in real time.

Challenges we ran into

Spatial intelligence in construction environments is extremely complex because sites are dynamic, cluttered, and constantly changing. We had to handle occlusion from equipment and materials, depth estimation from standard cameras, dust and weather conditions, and the challenge of distinguishing safe proximity from real risk. Teaching the system to understand context rather than just detect objects required significant iteration and real-world testing.

Accomplishments that we're proud of

We built a real-time system that transforms construction video into actionable intelligence instead of archived footage. We successfully automated detection of key safety and operational events and reduced the need for manual review. Most importantly, we created a system that helps managers make faster, more informed decisions without adding complexity to their workflow.

What we learned

We learned that detection alone is not enough and that context is everything on a construction site. Managers care about outcomes, not footage, and reliability matters more than flashy features. We also learned that building AI for real-world environments requires constant iteration and close collaboration with end users.

What's next for Ironminers

Next, we want to take the project to scale with a data layer that is robust enough to handle real construction scenarios. We plan to improve the vision pipeline for diverse site conditions, add more safety and productivity rules, and integrate with existing project management and reporting tools so insights flow directly into the tools site teams already use.

Built With

• claude
• fastapi
• gemini
• javascript
• pegasus
• python
• react