SquaaS | Devpost

Inspiration

Lo! We beheld the mundane streets and screens of our age and dreamt to transmute them into quests of renown. With the spirit of Skyrim upon our hearts and the gravitas of Dark Souls in our souls, we resolved to fashion an overlay whereby the world itself might don parchment and gold, and every footstep be reckon’d as adventure.

Modern live streams are full of content, yet so few feel like stories. We wanted to change that by making the viewer and the streamer alike feel like they’ve stepped into an epic.

What it does

SideQuest as a Service (SquaaS) transforms a live stream into an RPG-style adventure. Using Meta Ray-Bans for real-world streaming and real-time generative overlays, the system reimagines the user’s journey as if it were a fantasy game, complete with objectives, status effects, and danger alerts that are unique to their environment. The result: a real-world Skyrim-style interface for your life.

How we built it

• Hardware: Meta Ray-Bans capture live POV footage.
• Streaming Pipeline: We pipe video from a live WhatsApp call into OBS, maintaining a seamless stream to Twitch.
• Analysis Layer: A lightweight script samples position, video, and audio intermittently to detect environmental cues (e.g., “Entering new area”), NPC-like encounters (via voice recognition), and “danger” or “objective” moments (based on motion + sound analysis).
• Generative Overlay: UI elements and narrative text (titles, quest prompts, and lore-style commentary) render in real time and react to the environment.

Challenges we ran into

• No open access to live streaming blobs: We had to creatively route footage through screen-mirrored WhatsApp calls while keeping inference and broadcast on the same device.
• Prompt engineering for structured world events: Extracting coherent narrative moments from chaotic real-world input was nontrivial.
• Latency hiding with multimodal LLMs: process image to text, speech to text (pending), text to speech in parallel

Accomplishments that we're proud of

• Creating a real-time “RPG camera” from commodity hardware and open APIs.
• Building a working pipeline from Meta Ray-Bans → OBS → Twitch with generative overlays.
• Making the real world feel like a quest log

What we learned

• Meta Ray-Bans are not as developer-friendly as tools like Snap Spectacles; long-term success will require either different consumer hardware or more open APIs from Meta.
• OBS is far more flexible than expected. Its Browser Source can render data from any endpoint, including dynamic quest information from APIs or local web servers.
• Aggregating hyperlocalized events (like pop-ups in SF neighborhoods) is difficult. Social media data is noisy, and platforms like Luma or Partiful lack real-time spontaneity. For our MVP, we focused on curated, San Francisco–specific objectives.

What's next for SquaaS

• Scrape and synthesize local events (e.g., pop-ups, festivals, openings) to generate dynamic, time- and place-specific quests.
• Fuse geolocation + live camera input to dynamically announce “New Areas Discovered” or “Objectives Nearby.”
• Let the audience act have stake in the story by granting buffs, debuffs, or side quests mid-stream.