Dex | Devpost

Logo
Landing Page
LiDAR Scan
Point and Talk Tool
Sample AR result
3D Blueprint
3D Blueprint
2D Blueprint

Inspiration

Designing a room on a flat screen means guessing how it will feel in person. We wanted to work inside the space itself: point at a corner, say what you want, and see it at full size.

Meet DEX: an agent that helps you redesign a room while standing in it.

What it does

Scan once. Capture walls, floors, openings and furniture with LiDAR, marking unseen areas as inferred.
Point and talk. Move, replace or resize furniture. A spatial solver checks collisions, clearances and door swings, then explains any adjustments.
Mask and hide. Select unrecognised objects and fill their appearance in the camera view.
Furnish instantly. Place textured objects from a local catalog, or generate custom assets on demand.
Restyle the room. Request a theme, apply a layout that fits, and export a scaled PDF floor plan.

How we built it

DEX combines Expo, React Native, React Three Fiber, a Swift module, Fastify and a Python worker.

OpenAI Realtime powers voice over WebRTC. The agent requests actions; a TypeScript spatial engine computes positions, validates changes and handles undo.

Furniture comes from procedural templates, a textured catalog or Meshy generation. Every asset passes through the same bounds and clearance checks. PBR materials give simple geometry convincing surfaces, while Segment Anything and LaMa support object removal.

Scripted tool-call replays make agent behaviour testable without a live model.

Challenges we ran into

Asynchronous actions: Delayed responses targeted the wrong object until we bound work to its originating turn.
Generation latency: Custom models take about a minute. We pre-built common objects and show geometry before textures finish.
Spatial drift: Keeping content attached to an AR anchor corrected placement errors caused by tracking updates.
Disconnected inventory: The planner overlooked generated assets until we unified its catalogs.
Unpredictable assets: Generation added unwanted decorations, and dark textures made recolouring muddy.

Accomplishments that we're proud of

Unknown targets trigger clarification instead of an action.
Catalog and generated furniture share one spatial validation pipeline.
Inferred dimensions and visual substitutions are disclosed.
Object removal has a deterministic fallback when models are unavailable.
A known reconstruction marker landed within 0.3 cm of ground truth.

What we learned

Let the model express intent and let the solver own geometry. Validate generated assets before using them, and bind asynchronous work to the request that created it.

Our biggest surprise: materials often matter more than meshes. A good texture improves every existing object immediately; generation takes time and improves one.