FORMWATCH | Devpost

FORMWATCH

Inspiration

Many people worldwide go to the gym for various reasons, whether it be to stay in good shape, weight loss, or for daily exercise. There's the saying, "everyone starts somewhere", and for many, that starts with not knowing the proper form. We want to make the hurdle of starting to actually go to the gym easier by taking away the feeling of not knowing how to do exercises and thinking that you look silly. By adding a game aspect, we want to motivate people to keep going to the gym because consistency is also key.

What it does

FORMWATCH uses your camera to analyze your form while exercising and gives you: an overall score, time it took to complete repetitions, and tips/feedback. We also implemented a multiplayer feature where two players race to complete more reps before the timer ends.

How we built it

Our project, FORMWATCH, is a smart fitness application that utilizes the computer tracking framework, MediaPipe Pose, in React Native for recording exercise poses. All the motions are analyzed with skeleton nodes on human joints for motion tracking which determines angles and positions between joints as well as completion of the exercise, such as squatting.

Furthermore, we have an automatic grading system implemented with Gemini API. It determines the accuracy of the pose in exercises to a professional standard with online resources. The steps of determination is done in Express, where the video is divided into frames, which Gemini AI then processes for validation on the overall performance of the exercise (position of the knee to the ground, straightness of the back) and gives a score out of 100 and feedback.

Challenges we ran into

We originally wanted to use Presage to measure exertion/effort by measuring the increase in heartrate and breathing over time. This could be put into an exercise as another metric to improve scoring. However, Presage is only supported natively in android studio and C++, something that did not fit into our workflow.

Additionally, to save compute time, we implemented rep checking using Google's mediapipe library via Javascript. However, much of the documentation was outdated or missing, leaving very little information for how to properly integrate mediapipe into React Native. This was temporarily solved by using a WebView to render it in pure HTML/CSS/JS, but future solutions involve a custom React Native library for quick performance and offline functionality.

Accomplishments that we're proud of

We were able to implement the multiplayer feature, which we had struggled with for some time, as well as having a clear UI. This project was a collaboration of people with varying skill levels and skill sets to create a functioning app that will motivate people to go to the gym.

What we learned

In a technical sense, we learned that AI has became a powerful assistant in programming with precise given AI prompts. This means that programs can be written and run properly without the user needing to understand everything about the program. However, it still implies that the user must know some basic knowledge of the program they are using in order to spot errors and provide valuable feedback to the AI.

What's next for FORMWATCH

Implementing leaderboards
- Form guide sections
- Challenges and rewards
- Improving the AI coaching: real time voice feedback, personalized workouts
- Ability to create groups with friends

See GitHub for project demo!