Comic Quest

Comic Quest is a MR app that redefines comic book reading. By integrating AI, it turns traditional comic stories into immersive, interactive experiences, bringing superhero adventures to life.

Comment

Inspiration

The idea of enhancing the newspaper reading experience was sparked by a 3D Gaussian Splatting recreation of the Francis Scott Key Bridge collapse in Baltimore. This concept intrigued Mika Haak and Lorenzon Carly, leading them to explore how, with the use of new technologies, could transform our understanding and empathy towards global events, while contributing to documenting and journalism. As a proof of concept for augmented 2D reading, they decided to focus on newspaper comics. Our goal for this hackathon became experimenting with how these technologies could enhance the comic book reading experience in a meaningful and satisfying way.

What it does

It processes digitized comic books using AI tools, presenting them in mixed reality (MR) with a parallax effect, resulting in a dynamic and immersive reading experience.

Key Features:

  • Portable Digital Comic Library: Access a wide range of comics on the go.
  • Immersive Reading Experience: Engage with comics in a dynamic MR environment, complete with enhanced visuals and sound.
  • AI-Powered Enhancements: Use AI in existing comics to process the comics and add effects.

Why MR?

  • The virtual comic library is accessible anywhere.
  • The medium offers a safe, private and engaging reading environment with interactive stories.

What makes it unique?

  • AI-enhanced comics that create a more immersive and personalized experience. It utilizes existing comics (both old and new), eliminating the need for additional content creation resources.
  • Modernization of classic comics, giving new life to old stories. This approach connects with new generations while allowing older fans to rediscover their favorite stories in a fresh, innovative way.

For whom is it made?

The comic market is yearly growing and expanding, mainly digitally. There’s always more demand for diverse content, which with help of AI would be easier to achieve.

How could the outcome look?

This project aims to revitalize the comic industry by introducing interactive, AI-enhanced storytelling, leading to greater reader engagement and new forms of narrative.

How we built it

Our initial plan was to process images in real time, but due to time constraints, we pre-processed them, partially using automated AI tools. With more time we could connect the streamline of tools, so this process could be fully automated. A scanned comic (.pdf), gets feeded into Yolof8 to recognize coordinates of panels to give it to a python script, which cuts out the panels of each page and generates a folder. With SAM1 and Stable Diffusion we segmented every panel into different parts (recognising objects) that will be the layers that will be given a different depth. We created a python script that renames the layers and place them in different folders to use this later in Unity. In Unity we created a protocol that identifies per naming the hierarchies of pages, panels and layers and their depth to determine their position. We added hand interactions to navigate and explore the comics in 3D space. We added also the parallax effect to the layers.

Meta SDKs we used:

  • Presence Platform:
    • Passthrough: to be able to experience the comics in MR
    • Hand tracking & Interaction: for hand gestures
    • Meta sample: for swiping interaction
  • SAM1: to segment the comics into the layers that will build the parallax effect

Other technologies/materials/tools we used:

  • Yolof8: panel recognition and splitting, restyling, etc.
  • Roboflow API: with specialized data set for inference
  • Flux: image generation
  • Stable Diffusion: for the creation/edition of visual materials for trailer
  • Florence-2: to process things like refurbishment of visuals
  • Parler-TTS: for the creation of text-to-voice
  • Live-Portrait: for the creation of face animations
  • Sadtalker: text to face animation
  • Open source comics: to use material to experiment with without copyright issues
  • ShapesXR: for the creation of prototypes
  • Blender: for the creation of 3D assets
  • Figma: for the creation of prototypes and assets
  • Unity: for development of the App
  • DaVinci: for the video editing of the trailer
  • ChatGPT: for creating and improving text and code
  • Unreal: to create animations for trailer
  • Python: to pre-process image files with AI tools

Challenges we ran into

  • connecting backend communications because takes time to get that streamline working
  • Training the AI models takes time, it doesn’t work perfect as it is right now
  • Some hand interactions were sometimes not working
  • Detectron2 didn’t work
  • SAM2 was not possible to use in such a short amount of time

Accomplishments that we're proud of

We are proud of creating a round concept with a working prototype that shows the big potential of this idea, that could be worked on. We think this has potential to change the way we experience all kinds of 2D content and not just for entertainment purposes.

What we learned

We learned to use many AI tools that with some input (python) we were soon able to do some parts of the workflows automatically. We also learned how to prototype quickly further ideas that we thought were very valuable to showcase, and also to proof of work before developing.

What's next for Comic Quest

These are some developments we started to explore and know are possible within a timeframe and some extra resources:

  • More AI-Powered Enhancements: Use AI to animate comic panels, add depth, integrate audio for a more engaging narrative & AI generated skyboxes for a VR viewer for a more immersive experience.
  • Story Revival and Creation: Revitalize classic comics with AI-driven refurbishment or create new stories.
  • Personalized experience: Add user-generated content features with the help of AI. For this we could use the Image capture AI kit.
  • Partnerships: Contact comic publishers to work with them
  • Real time Screen capture to Parallax: experiencing also your real-life comic books in an immersive way, in real time, it doesn’t matter which title or which image

Built With

Share this project:

Updates