Tree-D | Devpost

Inspiration

The project was inspired by the "Scalar Gap"—the disconnect collectors experience when viewing art online. It is often impossible to gauge the physical presence, texture, or true scale of a painting through a flat thumbnail. We wanted to bridge the sensory gap between digital archives and physical reality. (Track 4 TRACK 4: AI for Virtual Viewing & Decision Support)

What it does

Tree-D Studio converts 2D paintings into depth-aware 3D reliefs. By integrating with the Met Museum API, users can search for masterpieces and instantly view them in a 3D space with accurate real-world dimensions. The app generates AI-driven normal and displacement maps to simulate texture and allows users to export these models as USDZ or GLTF files for placement in their own homes via Augmented Reality (AR).

How we built it

We built the platform using Next.js 14 and TypeScript for a robust frontend. The 3D heavy lifting is handled by Three.js, which renders our 3d models in an efficient manner. For the "depth," we utilized Marigold AI for normal map generation and procedural logic for displacement and roughness. We experimented with other AI solutions, but we found Marigold to be the one that worked the best (given the time constraint haha). Combining the data of Marigold with our Three.js model, we were able to display the roughness and roughly simulate how the painting should look like with this added artificial distortion.

Challenges we ran into

Constant pivoting, we originally wanted to make an AR-powered app for mobile where users could place paintings in their rooms using AR. However, due to the significant delays when it comes to building and developing on iOS and Android, we decided to pivot to a web-app instead. Our team also did not possess a Macbook so we were unable to make use of iOS' strong AR capabilities natively.

Accomplishments that we're proud of

We managed to create a tool that displays paintings in people's environments, though not directly through a smooth pipeline.

What we learned

We gained deep insights into the nuances of spatial computing—specifically how important metadata (like physical dimensions) is when moving from 2D web screens to 3D environments. We also sharpened our skills in combining AI-generated assets with traditional procedural textures in a Three.js environment.

What's next for Tree-D

Image recognition: We want to add a feature that allows users to snap photos from museums and directly have a 3D model on their phone for usage.

Native development: Given more resources and time we would love to create a native app that can render the paintings in a more controlled AR environment, allowing for the detection of walls, etc.