• 

  GITIRL

• 

  Sentry Fixing Bugs

• 

  Elasticsearch 3D Geohash

• 

  Elasticearch Geohash Search

• 

  Telemetry

• 

  Detecting Items

• 

  BracketBot grabbing a bag of chips

GitIRL

Your room as a Git Repo.

Inspiration

Something is never where you left it and nobody moved it, and there is no way to check.

Software has had this solved for twenty years. git diff tells you what changed, git revert puts it back, git blame says who did it. Rooms have none of that.

We didn't want a database that imitates git. There is an actual .git directory on the laptop, each file in it is an object on the table, and the thing that applies the patch is a balancing robot with an arm.

What it does

GitIRL puts a room under real version control. Objects are YAML files in a git repo, the robot's stereo cameras stage changes, and the arm is git apply.

Git is the write path and the source of truth. Everything upstream of the robot is actual git rather than a reimplementation of it, so when we print a diff that is git diff output.

Elasticsearch is the read path, because git cannot query. A commit graph knows what changed but has no answer for "where did I leave my keys".

Sentry covers all three machines, which fail in completely different ways: a laptop, a Raspberry Pi that can physically fall over, and a web tier.

In the demo we hand a judge an object and let them put it anywhere they want. room status goes amber, room diff prints the coordinates they picked, and the robot puts it back. There is nothing scripted about it since a stranger chose where the object went.

How we built it

Perception. Stereo cameras feed point cloud, voxel map, instance segmentation, named objects, scene graph. A capture quality gate throws out any scan taken while the robot is correcting its balance. Poses quantize to a 10 mm grid before serialization so that a commit records a real move instead of sensor noise.

Git. The scene graph writes one YAML file per object under zones/ in a real repository. A revert produces a normal diff, and the executor turns that diff into an ordered list of pick and place operations. It has to topologically sort them, because you cannot put the mug back if something is already sitting in its spot, which is the same constraint as applying a patch out of order.

Elasticsearch. Every commit indexes into six stores: three indices (room-objects, room-voxels, room-clouds) and three time series data streams (room-observations, room-events, robot-telemetry) carrying 50 Hz robot telemetry. Hybrid search is why typing "mug" finds the cup.

Observability. Every process starts tracing at init, so one Sentry trace covers the laptop, the Pi and the web tier. Every span carries capture_id and every Elasticsearch document carries sentry_trace_id, so you can get from a slow trace to the documents behind it, or from a bad diff back to the waterfall that produced it.

49k lines of Python, 84 test files, four people, 36 hours.

Challenges we ran into

The robot spent most of the weekend off the network and Sentry is how we know exactly how badly. Three failures were worth the whole integration.

1. The arm was fine. The shutter was the bug. Grasps kept slipping and we were about to rewrite the IK and recalibrate the gripper. The Sentry issue carried the last 40 telemetry samples as breadcrumbs, and reading them, balanced dropped 1.0 to 0.0 and motor current hit 2.0 A in the 200 ms before the shutter fired, with tilt_rate_max = 0.239 rad/s on the capture span. Looking up the same capture_id in Elasticsearch, the three cameras disagreed about the object by 49 mm against a 10 mm quantization quantum. The robot was mid lean recovering from a balance correction and the grasp target had been computed off a cloud captured during the tilt, which is not something a stack trace was ever going to tell us. We added a capture gate: skew_ms < 25, tilt_rate_max < 0.05, coverage > 0.60.

2. Session Replay recorded black rectangles and returned HTTP 200. Segments uploaded, the replay count climbed, nothing errored. The only symptom was segment size, around 2 KB, which is far too small for a WebGL scene. replayCanvasIntegration defers its snapshot to the next animation frame and by then WebGL has cleared the drawing buffer, so it dropped the blank frames and produced replays that were valid and completely empty. We moved the snapshot into the render loop and made it synchronous. Without that we would have found out during judging.

3. The robot was starving itself, and it only showed up on the robot. Load average 41, 139 MB free. Our own server was using 16.5% CPU with no clients connected. We were polling five topics at 200 Hz, roughly a thousand syscall pairs a second, underneath a telemetry tap sampling at 50 Hz and a SLAM publishing at 28. The map reader also kept two copies of a slot for as long as it lived, so reading the 36 MB map once pinned 72 MB for good. Neither of those costs anything on a laptop.

We also lost hours believing tracing was gated by our plan tier because the Performance tab was empty. The transactions were in the spans dataset the whole time, and on top of that the default inbound filter was dropping every /api/health transaction, which was the endpoint we had been testing with.

Accomplishments that we're proud of

The robot files its own bug reports and closes them by tidying up. A grasp slips and it opens a grasp_slipped issue with the camera frame attached. The executor retries, the object lands on the committed pose, a rescan comes back clean, and the robot marks the issue resolved through the Sentry API.

01:42  grasp_slipped, gripper closed to 2mm, [camera frame attached]
01:43  robot retried, rescan clean
01:43  resolved by robot

Issue trackers assume a person reads the error and patches the code. In this case the robot that caused the error is the one that clears it, by moving an object.

We also got real git rather than something git-shaped. Every status, diff, commit and revert runs the actual tool, so branching, history and merge conflicts came free, including a conflict you resolve by moving furniture.

The join between Sentry and Elasticsearch is the part we would build again. Going from a 3 second span to the exact point cloud that produced it, and back, is what made debugging three machines in 36 hours possible at all.

What we learned

We started out treating this as a 30 FPS perception problem and ran out of compute and bandwidth almost immediately. Reframing it as commits fixed most of both. A scan happens at a moment, on purpose, and everything between two commits is throwaway.

The one architectural rule we kept was that the Pi does I/O and the laptop does thinking. Nothing goes on the Pi that you would have to reflash to debug, because you will be debugging at 4am with the robot on its side.

Two smaller things. Never verify tracing with a health check, since that is the one endpoint Sentry filters by default. And an error that returns HTTP 200 is the expensive kind: two of our three worst bugs reported success, and we caught both by noticing a number that looked wrong, a 2 KB replay segment and a 16.5% idle CPU. Worth instrumenting sizes and rates and not only failures.

What's next for GitIRL

Robot Session Replay, rebuilding the robot's motion from encoder and pitch telemetry and playing it next to the trace of the same moment, so you can scrub a physical action the way you scrub a web session.

git merge already produces a real conflict when two people commit incompatible rooms, and right now a human resolves it. We want the robot to propose a resolution, stage it, and open a PR you approve from your phone.

room blame already knows which commit moved something and who authored it. We want it to say so out loud in the kitchen when someone asks.

Built with

bracketbot, raspberry-pi, vla, python, numpy, scipy, opencv, yolo, pytorch, clip, gaussian-splatting, opensplat, git, elasticsearch, fastapi, websockets, tailscale, react, three.js, webgl, javascript, vercel, openai, sentry, rerun

Built With

• bracketbot
• elasticsearch
• fastapi
• gaussian-splatting
• git
• google-cloud
• javascript
• mongodb
• numpy
• open3d
• openai
• opencv
• opensplat
• python
• pytorch
• raspberry-pi
• react
• sentry
• tailscale
• three.js
• vercel
• vla
• webgl
• websockets
• yolo