Form Focused

CODE: 0BD4D4E62925C7EE

Inspiration

While image and video processing software/AI has developed greatly in the past few years, the concrete day-to-day applications for a usual person haven’t taken the spotlight as much. We decided to take this developing field in a new direction and design a product that is useful to a broad, diverse audience: a digital workout assistant. Most people who workout, from beginners to experts, are interested in having correct form during various exercises to optimize strength and muscle gains, but it’s difficult to find reliable, centralized information on how to do so. Using AI, CV, and biometric data, we give live user feedback while the user is exercising, based on data from workout videos across the internet, allowing users to progress in the gym at a faster rate. Millions of people can benefit from this technology, making the gym more accessible, personalized, and effective.

What it does

Form Focused is a digital work out assistant. Using CV and biometric data (heart rate and body temp, collected from a watch-like device we constructed), it analyzes performance and exertion while the user is exercising. It then delivers feedback to the user in a variety of forms. A percentage score is computed based on the user’s form for a specific exercise, and the system uses math such as the Karvonen formula to compare their exertion (measured with HR and body temp) to a baseline state determined in a calibration period. Gemini takes all of the data feedback (HR, temp, form feedback) and creates a summary displayed to the user. Then, Elevenlabs text-to-speech reads the text feedback out loud, allowing for the user to react in real time without having to watch the laptop screen while exercising. Additionally, the biometric data is graphed over the workout period for the user to look back at. Alongside being graphed, heart rate provides a good metric for intensity of workout, and temperature serves as a warning signal for if the user is in danger of overexerting themself. User data is stored across sessions with Snowflake, allowing the coach to store user preferences and information, along with adjusting feedback in the context of a longer fitness journey.

How we built it

The hardware for this project, a wearable data collection device, was built around an Arduino Lilypad microcontroller. The holder for the microcontroller was 3D modeled and then printed out, with additional features for attaching the wristband and organizing wiring. We wired two sensors to the Lilypad: a heart rate sensor and a temperature sensor (thermo resistor in a voltage divider circuit). Components were wired via conductive thread sewn into the wristband of the wearable.

We built most of the software for the project using python. We used mediapipe to extract features from the webcam and pyserial to interface with arduino.

Challenges we ran into

A main challenge we encountered was integrating all of the different components of our project with each other in a final interface that is user-friendly and compact. We had multiple different data streams feeding through different channels into one file (the Streamlit program). For example, to read data from the wearable device we first needed to configure the Lilypad Arduino and connect the circuits appropriately to output data, read this data from the arduino into python, then display it with a live-updating graph on the site. We used this and the CV metrics to feed into Elevenlabs and Gemini inputs for customized and useful feedback to the user, and on top of all of that we implemented Snowflake in the same file for a registration system. We ran into many issues trying to ensure all of the different parts of this project came together smoothly into the final project and that all of the code worked together.

Accomplishments that we're proud of

One accomplishment we’re proud of is building a functioning wearable that communicated effectively through Arduino and filtered/displayed with python. This task required many parts to work together at once, so we are proud that we saw it through to the end and created a working physical to software product.

What we learned

Lee: in designing the wearable, I learned a lot about reframing hardware priorities for different projects. Much of my prior experience has been with more solid structures, like robotics and rocketry. Designing a flexible system that has to be small, aesthetic, comfortable, efficient, and adjustable forced me to get really creative with the layout of the electronics and the method of connecting each component to the microcontroller without diminishing the comfort or appearance of the device.

Ethan Sharma: I gained experience with electrical components like the arduino and heart rate sensor, which allowed me to use my previous experience in designing and building circuits for this project. The software components were more novel to me, specifically I learned how to implement API keys for models like Gemini and Elevenlabs, as well as working with the frontend using Streamlit.

Leah: A major learning outcome was understanding the end-to-end pipeline of a computer vision system which involved capturing live video input, processing frames efficiently, extracting pose data, and translating that data into verbal feedback. I gained hands-on experience working with coordinate systems, joint angles, and temporal consistency, which helped me appreciate how small inaccuracies in detection can significantly affect downstream logic like form correction.

Ethan Hu: I learned a lot about project infrastructure setting up and integrating the Snowflake database along with working on project compatibility for collaboration and website hosting. I learned what different parts of the data pipeline were, along with how to query and store actual data.

What's next for Form Focused

For scaling the product, a good next step would be to integrate the software with already-existing wearables tracking data about the body, like the Apple Watch for example. This way, more users could take advantage of the software with the hardware systems they already own.

Another update with the product would be seamlessly integrating the UI with all of the different parts of our product. This could include a troubleshooting mechanism for the user if the hardware or any software bugs occur, as well as displaying the relevant data and past feedback in a clear way. Also, we could add additional features using Snowflake API to enhance user experience by allowing them to friend other users while tracking their workouts, similar to Strava.

Built With

• elevenlabs
• mediapipe
• python
• snowflake