Inspiration
We wanted to build a simple and accessible fitness tool that helps people exercise with better form. Many AI coaches focus on complex routines, but we wanted something anyone can use with just a camera. The goal was to combine fast pose tracking with real time feedback to make training feel more interactive and personal.
What it does
Exercise Coach uses your webcam to detect your body position and count pushup reps automatically. It draws your skeleton on the screen, checks whether you are going up or down, and increases your rep counter when a full pushup is completed. It also provides feedback and a form score during the session, and uses Gemini to summarize your movement.
How we built it
We built the front end in React and used the TensorFlow MoveNet model for pose detection. The camera feed is streamed through a video element, and a canvas on top renders the detected joints and skeleton. Our rep counting logic looks at changes in shoulder height to determine movement phases. We track top and bottom positions and update the counter when a full cycle completes. We also connected Gemini through the Google Generative AI API. Keypoints are sent once per second and Gemini gives a short description of what the user is doing. Everything runs in a clean dark mode interface that shows the camera, skeleton, rep count, score, and feedback.
Challenges we ran into
Setting up MoveNet inside a React component took time because we had to carefully manage async loading and make sure the camera initialized before the detector. Tuning the rep counter was also challenging. Small variations in movement or shaky camera angles caused false positives at first, so we had to tune thresholds and phases. We also ran into issues limiting Gemini calls so the model would not be overloaded.
Accomplishments that we are proud of
We built a working real time pose tracker that runs directly in the browser with no external software. We created a rep counter system that is consistent and accurate while staying lightweight. We integrated AI feedback using Gemini and built a clean and modern UI that makes the experience feel like a real workout tool.
What we learned
We learned how to work with pose detection models, how to process keypoints, and how to structure motion into phases for better accuracy. We learned how to manage asynchronous tasks in React, how to use the WebGL backend for TensorFlow, and how to optimize performance in a continuous animation loop. We also gained experience connecting a Generative AI model to live sensor data.
What is next for Exercise Coach
We want to expand beyond pushups and support squats, planks, and other exercises. We plan to add form correction so the AI can point out mistakes like low hips or uneven shoulders. We would also like to build a history system that tracks your stats over time, introduces structured workouts, and adds voice feedback for a more personal training experience.
Log in or sign up for Devpost to join the conversation.