Arnold AI

Your fitness & Physical Rehabilitation Companion

Comment

🧠 Inspiration

1 in 4 Canadians wait over a month for physiotherapy. Meanwhile, 60%+ of adults don’t meet basic fitness guidelines, and most fitness apps are either static, generic, or unengaging. We wanted to build something smarter: a real-time AI-powered assistant that adapts to your body, coaches your form, and keeps you accountable... just like a personal trainer, but fully digital.

💪 What it does

Arnold AI is your intelligent fitness and rehab assistant, combining 3D anatomy, AI-generated workouts, pose analysis, and audio coaching into one sleek web app.

You can:

  • 🎯 Select muscles via an interactive 3D anatomical model
  • 📋 Get muscle-specific workout tips, Gemini-generated in real time
  • 🧠 View muscle insights, summarized by AI.
  • 🎥 Receive real-time form feedback from your webcam using MediaPipe + Gemini
  • 🔊 Listen to coaching tips, auto-synthesized with Google Cloud Text-to-Speech
  • 📆 Sync workouts with Google Calendar
  • 📄 Export a PDF plan for offline access

POWERED BY GOOGLE: gemini-2.5-pro, gemini 2.5, gemini-2.5-flash, gemini-1.5-pro, Google Calendar API, Google Cloud Text-to-Speech API.

It blends visual, audio, and motion feedback into one seamless coaching experience.

🛠️ How we built it

  • Frontend: Next.js (App Router), TailwindCSS, React Three Fiber for 3D rendering
  • 3D Models: Sculpted in Blender and rendered via Three.js
  • AI Backend: Google Gemini via the Generative AI API:
    • /api/rehab-feedback – Real-time pose analysis + TTS feedback
    • /api/muscle-info – Muscle function summaries
    • /api/muscle-workout-tts – Coaching tips synthesized into MP3s
    • /api/streamed-workout-tips – Fast, low-latency streamed cues
  • Media: Google Cloud Text-to-Speech for voice coaching
  • Integration: Google Calendar API to auto-schedule your workouts
  • Infrastructure: Deployed on Vercel with serverless streaming APIs

🚧 Challenges we ran into

  • Streaming pose data + Gemini output while staying performant
  • Coordinating real-time feedback across webcam, text, and audio
  • Rendering responsive 3D models on mobile without frame drops
  • Building natural prompts that ensure Gemini gives coach-like, safe feedback

🏆 Accomplishments that we're proud of

  • Live pose-to-feedback pipeline using Gemini + MediaPipe
  • Audio + visual + textual feedback combined in one UX
  • Full-stack app with 3D rendering, AI, TTS, and real-time APIs
  • Clean mobile UX that handles AI latency gracefully

📚 What we learned

  • How to integrate LLMs with real-time pose tracking and multimedia output
  • Optimizing 3D rendering and animation pipelines for mobile
  • How to use Gemini to generate consistent, context-aware coaching prompts
  • Streaming TTS and LLM output together for fluid user experience

🚀 What's next for Arnold AI

  • Add pose comparison against real trainers or reference videos
  • Build a mobile-first PWA with offline workouts
  • Add RAG-enhanced explanations for exercises and rehab plans
  • Use personalized injury history to adapt workouts over time

Built With

Share this project:

Updates