RealityMOD | Devpost

riding through reality from Sketch to MakotoShinkai
use a PS-like UGC tool to customize realitys
chat with LLM to create reality you want
spacecut to experience colorblind
an reality portal generated by AI voice input
use a portalgun to ride through different realitys
use a lazergun to mod reality colors , lights
check all reality UGC

Inspiration

As an indie XR developer, RealityMOD was sparked by a rebellious idea: "What if our reality were a game—then we could add MODs to it." Inspired by the portal gun from Rick and Morty, we created a tool that uses this concept to "shoot" portals and step into different visual realities.

What it does

RealityMOD features several functional modules: MOD RealityRide: A portal gun that opens gateways to VST realities with distinct visual styles—Sketch, Matrix, Anime, Movie, even Colorblind simulation. MOD RealityShop: A Photoshop-like workshop where users manually adjust parameters to create and define their own reality. MOD ChatSeek: By combining the headset's passthrough camera API with an LLM, users describe a style in words, and the AI intelligently generates and renders the corresponding VST scene in real-time. MOD HueLazer: A color-modifying laser gun that targets and changes the colors of objects in the VST feed in real-time—like altering clothing hues, trying a purple watermelon, or adjusting ambient lighting.

How we built it

Dynamic LUT Generation & Rendering: Built on Unity. Commands from the Photoshop-style interface or the LLM are converted into specific color parameters. Our proprietary Advanced LUT Generator dynamically creates or blends color lookup tables based on these, applying them to the VST video stream via shaders for lag-free, stylized rendering. Intuitive Portal-Gun Interaction: The design adopts the Rick and Morty portal-gun concept. Using the Passthrough Camera API for reprojection and recoloring, we achieved the magical visual effect of walking between different reality portals with an intuitive and immersive operation.

Challenges we ran into

Performance: Achieving real-time complex color transformation and LUT application on mobile XR chips demanded extreme optimization. Design: Creating intuitive interaction. We chose the "portal gun" as the core metaphor, making the abstract concept of switching realities operable and understandable. Technical Integration: Integrating the LLM required extensive prompt engineering to ensure stable, high-quality, and formatted color parameter output for the rendering pipeline.

Accomplishments that we're proud of

RealityMOD is likely one of the first MR applications to fully leverage VST technology for real-time, diverse style rendering. Technically, it uses VST-LUT and the Passthrough Camera API to create distinct "portal realities," LLM-generated realities, and direct object color modification. Conceptually, it proposes "adding MODs to reality" pushing MR beyond gaming/entertainment toward a grander goal: modifying and reshaping our perceived reality.

What we learned

During the intensive one-month development of the new version, we gained deep experience in integrating the Passthrough Camera API with LLMs and explored intuitive interaction and narrative UI design through the "portal gun" mechanic.

What's next for RealityMOD

RealityMOD will be a long-term project:

Build a UGC Platform: For users to upload and share styles created with RealityShop/ChatSeek, and for artists to import LUTs from other software.
Incorporate LBE (Location-Based Entertainment): Enable portals to be persistently anchored to specific real-world locations. Imagine daily life and work shifting between different "reality layers" in a lightweight MR future.
Develop More MODs: Combine the Passthrough Camera API with CV to create environment-aware, AI-generated content beyond LUT filters, such as the planned FaceMOD.

Significant Competition Updates

Successfully integrated LLM and the Passthrough Camera API, realizing the core "AI generates reality from a sentence" feature.
RealityShop: Launched this Photoshop-homage UGC editor, implementing most Photoshop point operations (gradient maps, posterization, etc.), allowing users to create complex reality effects with zero learning curve.
Completely redesigned the interaction based on intuitive and narrative UI principles, formally adopting "portals" as the core mechanic for launching RealityRide.