Fireside Tales | Devpost

Fireside Tales

🔥 Fireside Tales: Collective Storytelling Reimagined in AR

Tagline: Your words, visualized. Collective storytelling powered by AI.

Inspiration: The Magic of the Campfire

The inspiration for Marshmallow Tales comes from the timeless, cozy ritual of campfire storytelling. We wanted to replicate that feeling of warmth, shared presence, and intimate co-creation, but break the barriers of physical distance using the immersive capabilities of Spectacles AR. We aimed to move beyond simple video calls and create a truly engaging, shared social experience where the content is generated collaboratively and magically visualized.

What it Does: AI-Powered Shared AR

Marshmallow Tales is a turn-based, multi-user Augmented Reality game that runs on Spectacles, built for friends to create a story together.

Gather: Players join a session and appear as customizable Marshmallow avatars around a virtual bonfire. Avatars track real head movements, simulating presence.

Tell: In turn, a player records a part of the story using the Spectacles microphone.

Visualize: The core magic happens here:

The audio is sent to a cloud function for Speech-to-Text transcription.

The transcribed text is used as a prompt for a Generative AI service (e.g., Image Generation API).

The resulting unique, high-quality artwork (the visualization of that story fragment) is returned to the Lens.

Reveal: The bonfire visually flares up, and a virtual "story leaf" with the AI-generated art flies out for all players to see, complete with audio playback.

Relive: Once complete, the group has a full, visual storyboard of their collective tale.

How We Built It: Snap Cloud & AI Pipeline

The project relies on a critical three-part architecture:

AR Front-end (Spectacles Lens): Handles the avatars, the bonfire, head-tracking (for gaze synchronization), audio recording, and rendering the final AR visuals (the art leaves).

Multiplayer Sync (Snap Cloud): We use Snap Cloud for persistent state management. This is essential for:

Synchronizing the turn-based gameplay logic.

Managing the session, room creation, and user joins.

Reliably distributing the AI-generated content (image URLs/text) to all connected Spectacles in real-time.

Generative AI Pipeline: A dedicated cloud service handles the heavy lifting: Audio $\rightarrow$ Text $\rightarrow$ AI Image Prompt $\rightarrow$ Final Art. Snap Cloud acts as the central hub to pass data to and receive results from this powerful external service, ensuring the Spectacles' performance remains fluid.

Challenges We Faced and Lessons Learned

Challenges:

Multi-Step Pipeline Latency: The largest technical hurdle was managing the latency of the Audio $\rightarrow$ Text $\rightarrow$ AI Image pipeline. Generating unique artwork takes time. We addressed this by implementing robust loading states and visual feedback (the bonfire "processing" the story) to manage user expectations during the asynchronous wait.

Snap Cloud Synchronization: Ensuring all remote players received the AI-generated art and updated turn status simultaneously required careful management of the shared state within Snap Cloud, especially concerning the reliable distribution of generated image URLs.

Head-Tracking Implementation: Accurately mapping Spectacles head rotation data to the Marshmallow avatar's gaze direction in AR was crucial for maintaining the feeling of presence and direct social interaction. (WIP)

Lessons Learned:

Generative AI is a Social Feature: We learned that integrating Generative AI as a collaborative tool (rather than a solitary one) dramatically increases user engagement and surprise, turning an abstract idea into a shared, tangible artifact.

The Power of Snap Cloud: Snap Cloud is the indispensable backbone for complex, high-immersion AR multiplayer. It allows developers to offload intensive tasks (like AI orchestration) while maintaining a high-fidelity, synchronized AR experience on the device.

Design for Immersion: Simple additions, like the avatars turning their "heads" to look at other players, are vital for selling the illusion of presence in a remote AR environment.

This project showcases how Spectacles and Snap Cloud can create profoundly social, shared, and imaginative experiences powered by the latest in generative technology.

https://drive.google.com/file/d/1oGazc7859SM6Mr14EnNFvTFPBXin4Kr5/view?usp=sharing

https://docs.google.com/presentation/d/1Enrj_yTTcavzvL44llkUegqpZWkLKC1eCvReXs5z0H4/edit?usp=sharing