AI used

MemoryLens

What It Does

MemoryLens is a mobile application designed to assist Alzheimer’s patients in recognizing people, places, and objects through an AI-powered augmented reality (AR) system. The app allows users to store virtual memories, annotate them with labels and notes, and retrieve them using real-time object and face recognition. It enhances autonomy, reduces caregiver burden, and helps users maintain connections with their surroundings.

Technology Stack

Frontend: React Native (Expo) for cross-platform development.
Authentication: Auth0 for secure user authentication and session management.
Database: Firestore for real-time data storage and retrieval.
Storage: Firebase Storage and Cloudinary for storing images and voice notes.
Face Recognition: FaceNet for identifying individuals.
Object Recognition: YOLOv8 for detecting and labeling user-specified objects.

- Augmented Reality: ViroReact for real-time AR overlays.

How We Built It

User Authentication and Data Management

Auth0 Integration – Secure authentication to manage user sessions.
Firestore Database – Stores user profiles and memory data for seamless retrieval.
Firebase Storage – Used for storing images and voice notes uploaded by users.
Cloudinary Integration – Optimized image storage and transformations.

Memory Storage & Retrieval System

Upload Virtual Memories – Users can store images of people, objects, or places.
Add Labels & Notes – Users can annotate their memories for future reference.
Retrieve Past Interactions – When an object or face is recognized, the app displays previous interactions in AR.
Voice Playback – Users can record and replay voice notes tied to saved memories.

Real-Time Recognition & AR Integration

Face Recognition (FaceNet Model)
Allows users to identify a person when they forget who they are.
Uses a “Who is this?” button for instant identification.
- Stores face embeddings in Firestore for personalized recognition.
Object Recognition (YOLOv8 for Personalized Recognition)
- Users can manually label objects for personalized object recognition.
YOLOv8 enables training custom object detection models for user-specific items.
Real-Time AR Integration (ViroReact)
- Users can scan objects and view saved labels directly in AR.
- Recognized objects trigger overlay labels and voice playback.
Location Recognition ("Where am I?")
- Helps users recognize where they are in their home.
- Uses object positioning and prior labeled data for room identification.

AI-Powered Personalization

Manual Annotation System
Users can train YOLOv8 to recognize specific objects they frequently forget.
User-Trained Object Models – Users can store their trained models for future use.

Challenges We Ran Into

AI Model Integration:
- Faced difficulties in achieving accurate face recognition without a sufficient number of labeled images.
AR Overlay & Navigation:
- Ensuring AR labels remained correctly anchored to moving objects posed a significant challenge. complicating development.
Performance & Latency:
- Experienced variability in Firebase Firestore's syncing speed, impacting data retrieval times.

Accomplishments That We're Proud Of

Optimized AI Integration
Successfully implemented pre-trained AI models, reducing development time.
Developed an efficient fallback system prompting users to label unidentified items.
Integrated image compression to enhance processing speed and reduce latency.

Enhanced User Experience

Designed a UI with intuitive navigation and accessibility features.
Developed real-time voice prompts to assist users when interacting with stored memories.
Implemented a personalized training system, allowing users to refine object recognition accuracy.

What's Next for MemoryLens:

Advanced AR Features:
- Plan to implement more sophisticated AR overlays and navigation aids to enhance user interaction.
Expanded Database:
- Aim to increase the repository of labeled images to improve recognition accuracy and broaden the app's applicability.
Performance Optimization:
- Intend to further refine image processing and API interaction to reduce latency and improve user experience.
User Feedback Integration:
- Commit to actively gathering user feedback to inform future updates and ensure the app continues to meet user needs effectively.