The Depth Module API now allows caching of depth frames, enabling pixel-to-3D projection even after a delay. This is especially useful for cloud-based vision models — once the results are returned, you can map image-space coordinates back into world space using the cached depth data.

This example lens demonstrates that workflow using the spatial reasoning capabilities of Gemini 2.5 Pro Preview, but the same approach can be applied to any vision model that outputs pixel or image-space coordinates.

NOTE: This project will only work for the Spectacles platform and Lens Studio.

Designing Lenses for Spectacles offers all-new possibilities to rethink user interaction with digital spaces and the physical world. Get started using our Design Guidelines

• Lens Studio: v5.15.0+
• Spectacles OS Version: v5.64+
• Spectacles App iOS: v0.64+
• Spectacles App Android: v0.64+

To update your Spectacles device and mobile app, please refer to this guide.

You can download the latest version of Lens Studio from here.

To obtain the project folder, clone the repository.

IMPORTANT: This project uses Git Large Files Support (LFS). Downloading a zip file using the green button on GitHub will not work. You must clone the project with a version of git that has LFS. You can download Git LFS here.

This example uses ASR to listen for voice input when the microphone button is pinched or tapped. The transcription is sent to Gemini Pro via the Remote Service Gateway package.

Before sending an image to Gemini we have to cache the depth data. DepthCache.ts listens to a stream of depth frame data (using the Depth Module API) and also keeps a small history of camera frames, aligning them with timestamps. When we call saveDepthFrame() it returns an ID that we can use to get the depth frame data and associated camera frame from that point in time.

When we get a result from Gemini we can call getWorldPositionWithID() from DepthCache.ts to convert the pixel coordinates (center of a bounding box) to 3D world coordinates from that cached information.

In GeminiAPI.ts you can see the response format that we are requesting from Gemini. It returns a text response that we display to the user, as well as a data object that contains the bounding boxes and labels.

For testing purposes there is DebugVisualizer.ts which will plot points over a camera frame texture. This can be turned on in SceneController.ts by checking showDebugVisuals to true in the hierarchy. Now when you make a request you will see the points Gemini found in the image.

1. Open the Preview panel in Lens Studio.
2. Use Interactive Preview, WASD (+QE for elevation), LMB and RMB to move around scene.
3. Input your key for the Remote Service Gateway in the hierarchy.

1. Build and deploy the project to your Spectacles device.
2. Follow the Spectacles guide for device testing.

If you have any questions or need assistance, please don't hesitate to reach out. Our community is here to help, and you can connect with us and ask for support here. We look forward to hearing from you and are excited to assist you on your journey!

Feel free to provide improvements or suggestions or directly contributing via merge request. By sharing insights, you help everyone else build better Lenses.

Built with 👻 by the Spectacles team