LiDARcade

LIDARcade: AR-powered arcade action for a better cause!

Comment

LiDARcade 👾

Breathe new life into classic arcade machines by bringing them into the real world with modern technology! LiDARCade is a refreshingly new way to play your favorite games through the lens of Augmented Reality with the purpose of aiding recovering stroke patients.

Inspiration 💭

We took inspiration from the golden age of arcade gaming. More specifically, we sought after the hypnotic pixel patterns of Space Invaders—the simple yet engaging 8-bit action that defined a generation. We asked ourselves: what if we could resurrect these nostalgic experiences but reimagine them through the lens of cutting-edge technology? LiDARcade bridges this gap by transforming ordinary physical spaces into immersive digital playgrounds. By merging the iconic gameplay mechanics of yesteryear with the spatial awareness capabilities of today's LiDAR technology, we're not just paying homage to gaming history—we're evolving it. This fusion of timeless design principles with frontier technology creates an experience that feels both comfortingly familiar and excitingly novel, allowing players to physically move within games that once existed only on flat screens.

What It Does ⚙️

LiDARcade transforms the classic 2D experience of arcade games like Space Invaders into an immersive three-dimensional reality. While the original game confined aliens to a flat screen with players stationary at an arcade cabinet, our application liberates this gameplay into the physical world through advanced augmented reality.

Using LiDAR technology, our app scans and maps real-world environments—whether a hospital room, therapy center, or home living space—and populates them with interactive game elements. Players physically move, aim, and engage with virtual targets that appear throughout their surroundings, creating a dynamic and spatially aware gaming experience that respects and integrates with the physical architecture of their environment.

Beyond entertainment, LiDARcade serves a crucial therapeutic purpose. For stroke recovery patients struggling with limited mobility and coordination, traditional rehabilitation exercises can be repetitive and demotivating. Our application gamifies rehabilitation by encouraging natural movements—reaching, pointing, and reacting to stimuli—that directly contribute to rebuilding neural pathways and strengthening affected muscle groups. The game's difficulty automatically adjusts based on performance metrics, providing an optimized challenge level that maintains engagement while preventing frustration.

How We Built It 🛠️

Frontend - Our user interface was crafted using SwiftUI to create a seamless and responsive experience. We designed an intuitive arcade-style interface with retro visual aesthetics that evoke nostalgia while remaining accessible for users of all ages. The UI features include:

  • A dynamic scoring system with real-time accuracy feedback
  • Custom crosshair aiming interface optimized for AR interaction
  • Arcade-inspired typography and design elements
  • Adaptive layouts for different medical environments
  • Patient and doctor dashboards for tracking progress
  • Environment selection menus for different gameplay scenarios

Backend - The technical foundation of LiDARcade utilizes several frameworks:

  • ARKit integration: Utilized Apple's augmented reality framework to handle spatial tracking, plane detection, and real-world anchoring
  • RealityKit rendering: This provided high-performance 3D content rendering and physics simulation for game elements
  • LiDAR processing: We implemented algorithms to interpret LiDAR depth data for precise environmental mapping and object classification
  • Notification system: We built a custom event-driven architecture to handle game state changes and user interactions.
  • Swift concurrent programming: Asynchronous operations manage resource-intensive scanning and rendering processes without affecting gameplay responsiveness.

Challenges We Faced 👾

Technical Integration Hurdles: Our development approach initially separated frontend and backend responsibilities among team members to maximize parallel productivity. However, this created significant integration challenges when merging the aesthetically-focused UI components with the computationally-intensive AR systems. We had to refactor our notification architecture to ensure seamless connection between these systems.

Swift Learning Curve: None of our team members had extensive prior experience with Swift or Apple's development ecosystem. We faced a steep learning curve understanding SwiftUI's declarative paradigm, RealityKit's entity-component system, and ARKit's spatial mapping capabilities simultaneously under hackathon time constraints. Yet, it was necessary for us to learn as we would need to utilize the iPhone 13 Pro's LiDAR capabilities.

Device Deployment Complexities: Deploying to physical iOS devices proved particularly challenging due to certificate provisioning, entitlement configurations, and privacy permission requirements specific to camera and LiDAR sensor access. Testing on physical devices was essential since the iOS simulator cannot accurately replicate LiDAR functionality.

Spatial Recognition Refinement: Distinguishing between different environmental surfaces and correctly anchoring virtual objects required considerable algorithm refinement to prevent gameplay elements from appearing in physically impossible locations.

Accomplishments We're Proud Of 🏆

We're incredibly proud to have created a fully interactive AR experience that seamlessly merges retro gaming nostalgia with cutting-edge spatial computing in just 24 hours. Despite having to simultaneously learn Swift during development, our cross-disciplinary collaboration successfully integrated complex technologies into a cohesive application that's both entertaining and therapeutic. We developed a functional machine learning component that adapts to individual patient capabilities while maintaining an engaging gameplay loop. Most significantly, we've transformed clinical rehabilitation from a tedious necessity into an immersive experience that patients genuinely look forward to—proving that meaningful healthcare innovation can also be delightful and engaging.

What We've Learned 🤔

This hackathon immersed us in the world of modern AR development and significantly expanded our technical repertoire. We learned Swift's functional programming principles and SwiftUI's declarative syntax while navigating Apple's developer ecosystem. Our team gained hands-on experience with ARKit's spatial anchoring and scene understanding capabilities, particularly in processing and classifying LiDAR point cloud data for real-time environment mapping. Beyond technical skills, we learned valuable lessons about interdisciplinary collaboration—bridging the gap between therapeutic healthcare requirements and engaging user experience design within strict time constraints.

What's Next? 🚀

Here is a snippet of our future ambitions with LiDARcade:

  • Expanded Game Library: Create a diverse collection of nostalgic arcade games reimagined for AR—from maze-runners like Pac-Man to shooting galleries like Duck Hunt—each thoughtfully adapted to leverage spatial computing and therapeutic movements.
  • Advanced Analytics Engine: Further develop a sophisticated machine learning model that analyzes gameplay metrics to predict patient recovery trajectories, identifies areas needing focused rehabilitation, and recommends personalized therapy regimens.
  • Comprehensive Patient Management System: Continue developing a secure database to track individual player profiles, store performance history, and generate detailed progress reports for healthcare providers with visual analytics dashboards. We currently have a Supabase project shell set up for this instance.
  • Environmental Map Marketplace: Build a library of pre-scanned environments including specialized rehabilitation facilities, common household layouts, and public spaces to accommodate patients in various recovery settings.
  • Multiplayer Rehabilitation Sessions: Enable remote collaborative play between patients and therapists or family members, fostering social connection during recovery while maintaining professional oversight.
  • Adaptive Difficulty Scaling: Refine our difficulty adjustment algorithms to respond dynamically to performance fatigue and frustration indicators, ensuring consistent engagement without overwhelming patients.
  • General UX Updates: Add features such as haptic feedback and sound effects for enhanced user experience.

Built With

Share this project:

Updates