Welcome Page
Memories Notification
Reminders Notification
Add Reminder Page (3)
Add Reminder Page (2)
Add Reminder Page (1)
People Page (2)
People Page (1)
People Page (3)
Home Page

Memoray — Dignified Memory Support with RL‑Ready 3D “Memory Walk”

Memoray is a human‑centered, agent‑powered companion for people living with Alzheimer’s and their caregivers. It combines hands‑free capture (glasses/phone), gentle recall and safety cues, and a Unity WebGL 3D game that turns real photos, places, and relationships into an interactive “Memory Walk” to reinforce identity and routine.

We built this because memory loss doesn’t just remove facts — it erodes confidence, stresses families, and makes everyday decisions harder. Our teammate Kai brought prior eyewear industry experience that shaped our hands‑free ergonomics, capture UX, and privacy signaling (respecting hardware indicators and minimizing cognitive load).

🧠 Problem & Approach

Orientation & safety are daily challenges (wandering risk, time/place confusion).
Caregiver burden grows as reminders and context maintenance become full‑time tasks.
What helps: calm prompts, familiar faces/places, and routine reinforcement — delivered in the moment and without stigma.

Design principles we followed:

Dignity-first: quiet UI, one‑tap “quiet mode,” respectful language, no alarms unless safety is at risk.
Assist, don’t overwhelm: tiny nudges over walls of text; short voice cues.
Privacy by default: local-first processing where possible; explicit consent for sharing.
Care circle transparency: caregivers can opt into summaries; the wearer remains in control.

✨ What Memoray Does

Hands‑free Memory Capture (glasses/phone): photos + short clips + timestamp + location → turned into structured “moments.”
Gentle Recall & Routines: “Who is with me?”, “What did I do this morning?”, medication and appointment nudges, customizable quiet hours.
Safety & Orientation: geofenced safe‑zones (Home/Park/Café), discreet cues when exiting a zone, caregiver notifications with last known context.
Care Circle Updates: calm, opt‑in summaries (e.g., visited the park with Alex; took morning meds at 9:12am).
🎮 3D “Memory Walk” Game (Unity WebGL): people/places/photos become branching waypoints; short narrative affirmations reward correct choices. Instrumentation makes it RL‑ready for future personalization.

🧩 Architecture (at a glance)

[Glasses / Phone Capture]
      │
      ├─(MediaRecorder/Camera + Mic)─► [Local FastAPI Ingest]
      │                                   └─ WebSocket stream (frames + audio)
      │
      ├─► [On‑device / Local LLM (Ollama) / RealTimeSTT] ──► entity extraction, captioning, PII filtering
      │
      ├─► [Cloud Agents (Google ADK + A2A, Gemini)]
      │       ├─ MemoryAgent     (summarize & index moments)
      │       ├─ RoutineAgent    (reminders, day scaffolding)
      │       ├─ SafetyAgent     (geofence rules, escalation)
      │       └─ FamilyAgent     (care circle briefs)
      │
      └─► [Firestore]  moments / relationships / highlights / safezones
                └─ feeds ► Mobile App (Expo RN) + Unity WebGL “Memory Walk” (via WebView)

Key flows

Browser/Messenger capture: Chrome Extension Offscreen + MediaRecorder → WebSockets to Local FastAPI for low‑latency chunking; resilient reconnects.
Mobile: Expo React Native app handles capture, geofencing, auth, and game embed (Unity WebGL via WebView).
Content graph: Firestore holds relationships (people ↔ roles), highlights (food/car/home/landmark/photo), moments, and safe-zones. The same data drives galleries and the 3D game.

🛠️ Technologies We Used — And Why

Capture & Ingest

Chrome Extension (Offscreen + MediaRecorder) for stable screen/audio capture in Chromium surfaces like Messenger; streams via WebSockets to our local ingest service.
Why not headless puppeteer right away? Browser permissions & UX are better handled by an extension during a hackathon; fewer cross‑origin surprises and simpler distribution.
Local FastAPI for streaming ingest and preprocessing.
Why FastAPI over Node here? Typed Pydantic models, great async IO, and quick CPU/GPU bindings for image/audio transforms.

AI & Agents (Decentralized + Cloud Hybrid)

Local LLMs via Ollama (e.g., Gemma/Qwen families) for decentralized inference on laptops/desktops:
- Privacy: sensitive frames and transcripts can be parsed locally.
- Cost: reduces token burn; local runs handle commodity captioning/entity work.
- Latency: sub‑second entity extraction on M‑series hardware keeps the UI snappy.
- Resilience: continues to function offline; queues to sync when online.
Google ADK + A2A with Gemini for higher‑level reasoning and collaboration:
- ADK loops: four agents (Memory, Routine, Safety, Family) run continuously.
- A2A handoffs: parallelize where safe (e.g., Memory + Routine) and sequence when needed to avoid cross‑talk; retries and state passing baked‑in.
- Why not roll our own agent framework? ADK/A2A gave us robust run‑loops, tool patterns, and inter‑agent messaging we could trust in a weekend.
Design decision — decentralized first: we treat local models as first‑class citizens for PII‑heavy extraction and reserve cloud calls for synthesis, narrative, and complex planning. This split‑brain keeps costs predictable and respects user privacy while still delivering top‑tier language quality where it matters (summaries, caregiver briefs).

Data & Backend

Firebase Firestore (with Firebase Admin via Express/TypeScript):
- Why Firestore over Supabase (Postgres)?
- Offline‑first: Firestore’s cache & sync are excellent for mobile wearers who may have spotty connectivity.
- Security Rules map naturally to our per‑wearer, per‑care‑circle access model.
- Multi‑client SDKs (web, RN, server) are mature and quick to wire.
- Realtime streams feed both the mobile UI and the Unity game without extra infra.
- When would Supabase be better? Heavier analytics/joins, row‑level SQL logic, and long‑lived transactional workflows. We scoped those out of the weekend to minimize schema/devops risk.
Express (TypeScript) API: thin endpoints for relationships/highlights/moments and Unity content feeds; helmet/cors hardening; health checks for judging stability.

Mobile App

Expo React Native for rapid builds, native APIs (geofencing, camera, background tasks), and smooth bundling.
Auth0 for OAuth/OIDC (PKCE), token storage, and quick caregiver sign‑ins.
Unity WebGL embedded via React Native WebView to ship the same 3D build on iOS/Android without wrestling a native Unity export during the hackathon.

Game & RL‑Readiness

Unity (URP) → WebGL build focused on clarity and cozy aesthetics; waypoint scene pulls People, Places, Highlights as interactive nodes.
Instrumentation for personalization/RL: we log per‑node dwell time, choice accuracy, time‑of‑day, and recent familiarity to a compact stream:
- Enables contextual bandits / Thompson sampling later (start with high‑familiarity items, gently introduce variety).
- Reward shaping favors confidence and short, successful loops over difficulty spikes.
Why WebGL over native Unity export? Consistent cross‑platform delivery via WebView, faster iteration, tiny install friction.

🔒 Privacy, Safety, and Cost Posture

PII Minimization Pipeline: local LLMs redact or tag sensitive spans before any cloud call. Images can be blur‑boxed on device (faces, addresses) with a reversible mask held locally.
Consent & Control: explicit toggles for sharing with caregivers, export/delete‑all, and a “quiet mode.”
Geofencing Rules: all safe‑zone boundaries and transitions are processed locally first; outbound alerts carry only the minimum context (time, coarse location category).
Cost Controls: tiered routing (local → cloud as needed), batched summaries, and short contexts. We sized Gemini calls to keep per‑day spend predictable.

🧪 How We Built It (End‑to‑End)

Capture loop: Chrome Extension Offscreen → MediaRecorder 2–4s chunks → WebSocket to FastAPI.
Local parsing: Ollama pipeline runs captioning, face/object tags, time/place normalization.
Agents wake: ADK loop picks up new “moments”; MemoryAgent summarizes; RoutineAgent updates reminders; SafetyAgent evaluates geofence; FamilyAgent drafts a calm daily brief.
Persist: Express service writes normalized records to Firestore; Storage holds thumbnails/blobs.
Game feed: Unity scene requests relationships + highlights → renders walkable graph; telemetry streams back.
Mobile UX: Expo RN surfaces galleries, safe‑zones, reminders, and the WebView game; caregiver settings and notifications live here.

🧗 Challenges & How We Solved Them

Low‑latency multimodal streaming: chunked WebSocket ingest with back‑pressure and resumable segments; defensive reconnections.
Agent orchestration: race conditions between Memory/Routine fixed by adding “moment locks” and A2A sequencing for certain tools.
Color‑shifted frames on some GPU paths: normalized pixel formats and disabled browser color‑space conversion; verified with test cards.
Cross‑platform Unity delivery: WebGL + WebView kept the build universal and the iteration loop fast.
Privacy boundaries: kept redaction local and asserted “minimum necessary” context on all notifications.

🧩 Why Our Stack Was the Right Choice

Local LLM + Cloud Agents = the best of both worlds: privacy & cost control without sacrificing narrative quality.
Firestore = offline‑first and realtime; exactly what a memory companion needs in the field.
Expo RN + Unity WebGL = “one build everywhere” for both app and game under hackathon constraints.
ADK + A2A = production‑grade agent loops and inter‑agent collaboration out of the box.
Express/FastAPI = thin, maintainable seams; each piece can be swapped later without a rewrite.

We evaluated Supabase carefully and would choose it for analytics‑heavy, SQL‑forward features in a follow‑on. For the weekend, Firestore’s sync and rules removed entire categories of offline/sync bugs.

🏆 Sponsor Challenges — How We Fit

Google Cloud A2A/ADK (Featured): four collaborating agents with loops, tools, retries, and A2A handoffs; cloud where it adds value, local where it preserves privacy.
Netflix – Beyond the Binge: interactive, personal narrative via Memory Walk; media‑driven branching stories with reinforcement of familiar people/places.
Waymo – Mobility: safe‑zones, gentle orientation cues, and context logging supporting safer community movement.
Microsoft / Google – AI for Good: accessibility, caregiver relief, respectful prompts.
NVIDIA – Hack for the Future: AI‑centric, edge‑aware pipeline ready for acceleration.
(We completed all sponsor challenges we targeted **except Wolfram and Capital One.)

👥 Team

Kovidh Gandreti — Backend, AI, reverse‑engineering integrations, frontend polish & game embed
Kai McFarlane — AI agents (A2A/ADK), frontend; prior eyewear industry experience
Zijian Gong — Unity / 3D scene & telemetry
Aarav Bejjinki — Frontend

📈 What’s Next

Richer Ray‑Ban | Meta integration, unified caregiver portal, HIPAA‑minded posture.
Lightweight bandit policy for the game using existing telemetry; extend to ML‑Agents curriculum later.
On‑device wake‑word and safer background capture controls.
Better “moments to story” pipeline with personal music/photo cues.

🔧 Built With (Stack)

Python, FastAPI, WebSockets, Ollama (local LLMs), Google ADK + A2A, Gemini, Express/TypeScript, Firebase Firestore/Storage, Expo React Native, Unity (URP → WebGL), Chrome Extension (Offscreen + MediaRecorder), Auth0, Docker/Nginx.