Healthcare Track, Education Track
Inspiration
We got the idea for making a Jarvis-like user interface and that led to us developing with mediapipe. From there, we thought of sign languages and translation but evenentually decided to settle on exercises.
What it does
FormFix takes in a video upload, or input through the webcam and aims to help fix up the form of the exercise being performed by the user.
How we built it
We used cv2 and mediapipe to segment the inputted footage into a singular rep of an exercise. From there, we compared the angles between the relevant nodes of a person doing an exercise with "golden" standard angles of the exercise obtained from a professional performing the exercise in a video. We give this data, along with data about the timing of the rep to Gemini to generate a summary of what to improve on. If the video is uploaded, a text transcript will be displayed on the screen. If the user is performing the exercise live, pyttsx3 will convert the transcript into audio and output it that way.
Challenges we ran into
We ran into interesting problems when it came to designing an algorithm for isolating a singular rep in the video. Another big challange was deciding on how to actually compare the user's video to the "golden" standard. We decided to go with angle as that is less dependent on a person's genetics, gender, distance from camera, etc. than something like distance the nodes move.
Accomplishments that we're proud of
We are proud of developing the cool algorithm for isolating a singular rep in a video or live recording. We went though a few theoretical algorithms but the one we stuck with has good performance and is reatively simple.
What we learned
We learned a lot about using mediapipe and cv2. These two libraries were crucial for dealing with videos and motion while doing exercises. It was also an experimental but educational experience to integrate the backend with the fronent.
What's next for FormFix
The concept of FormFix can be tweeked to be implemented in many areas, such as dance practice or sports. This project can also be improved to encompass many more exercises than it currently does.
Log in or sign up for Devpost to join the conversation.