Story Oracle

The first multimodal reasoning engine for infinite, interactive adventures.

Comment

Inspiration

Story Oracle was born from a simple question: what if AI-generated worlds were not just static frames, but dynamic, interactive canvases for human creativity? While I was deeply inspired by Google’s Genie—a world-building model that proved AI could generate navigable environments—I wanted to take it further by building an engine where the AI acts as a sentient Dungeon Master. My goal was to move beyond pre-scripted narratives to a place where every player's choice has a logical, physical consequence in a living world.

What it does

Story Oracle is a multimodal storytelling engine that allows players to co-create infinite adventures with Gemini.

Genre Selection: Players can choose from multiple settings, including Dark Fantasy, Cyberpunk Sci-Fi, Cosmic Horror, and High Seas Adventure.

Asymmetric Multiplayer: Supports solo play or a party of up to four distinct characters.

Multimodal Interaction: Players can use voice prompts for commands and listen to AI-generated narration.

Fourth Wall Integration: Users can scan real-world physical objects using their camera to add them to their in-game inventory.

Dynamic World Reaction: The engine calculates world reactions based on real physics and current environmental constraints rather than following a script.

How we built it

We built the platform using a sophisticated "Hub-and-Spoke" architecture designed to leverage the specialised strengths of the Gemini family:

The Brain: Gemini 3 Pro serves as our logic hub, utilising thinkingConfig to reason through complex player strategies and physical interactions.

The Narrator: Gemini 3 Flash takes the rigid logical output and generates immersive prose and cinematic image prompts.The Senses: Gemini 3 Vision identifies real-world objects from camera captures, while Gemini 2.5 Native Audio analyses the emotional tone of player voices to influence NPC reactions.

The Visualisation: Gemini 3 Flash-Image renders 16:9 cinematic illustrations and character portraits in real-time.

To ensure logical consistency, the reasoning hub validates player actions $A$ against the world state $S$ and environmental constraints $C$ using a probabilistic check:

$$P(Success | S, A, C) \rightarrow \text{Result}$$

Challenges we ran into

One of the greatest technical hurdles was the "Amnesia Problem," where the model would lose track of critical plot points over long sessions. We solved this by implementing a Lore Summary that updates every few turns, ensuring the AI maintains a "soul" and remembers key NPCs and objectives. Additionally, decoupling Reasoning from Narration was a significant challenge; we had to ensure the logic hub (Gemini 3 Pro) provided enough structure for the narrator (Gemini 3 Flash) without stifling the creative prose.

Accomplishments that we're proud of

We are incredibly proud of the seamless Real-World Inventory integration. Seeing a player hold up a common household item and have it instantly recognised and utilised in a high-fantasy setting feels like magic. We also successfully implemented a God Mode architecture that allows the Host to act as a human-in-the-loop, intercepting and editing AI narratives in real-time to steer the story. This balance of AI power and human control represents a significant step forward in interactive fiction.

What we learned

This project taught us that the true power of Gemini 3 lies in multimodal synergy. We learned that a game engine becomes exponentially more immersive when it can "see" the player's world and "hear" their emotions. Furthermore, we discovered that deep reasoning capabilities are what transform an AI from a simple chatbot into a reliable referee that can handle infinite, unscripted user creativity.

What's next for Story Oracle

We are just beginning to scratch the surface of what is possible with dynamic world interaction. In the future, we envision Story Oracle expanding into entire digital ecosystems that interact dynamically, creating fully persistent, immersive experiences that were previously thought impossible. We plan to deepen the "NPC Soul" memory vectors so that characters can be imported across different genres, maintaining their unique personalities and memories across every adventure.

Built With

Share this project:

Updates