pOSU!

pOSU! is a fun and interactive physiotherapy web app game that uses motion-tracking to guide users through rehabilitation exercises. Make physiotherapy fun!

Comment

Inspiration

Existing physiotherapy exercises that rely on visual simulations often require expensive VR headsets, making them out of reach for many people. That's why we created pOSU!—a web-based, motion-tracking web app that can run on any device with a simple webcam. We also wanted to make the experience feel fun and rewarding for users by introducing a point-based game system.

What it does

Using live motion-tracking through a webcam, pOSU!! guides users through a series of arm exercises by prompting them to move their arms in a specific sequence. Each movement is visually prompted with on-screen cues, and a point is awarded each time the user successfully completes an arm pose. The point system allows users to do physical rehabilitation exercises whilst engaging in a fun mini-game.

How we built it

  • The front end was implemented using Vite with React, and the back end was configured using Python and FastAPI. We also used Google’s MediaPipe pose landmark detection model to detect landmarks of human bodies in an image or video.
  • The model takes pose data (of body points such as wrist and elbow) and passes them through the FastAPI framework to a Python back-end.
  • Target points to prompt the pose that the user need to do are displayed as big pink points overlay over the camera feed. We made sure to take notes of their specific coordinates for accuracy calculations.
  • The back end then calculates the Euclidean Distance between the current position of the body landmark and the target point that the user needs to hit, accounting for a margin of error. This margin of error is set differently for every pose, depending on their difficulty.
  • When the player successfully hits the pose, the backend server returns a value of 1 to indicate the pose has been successfully executed. Then, the frontend has a local state variable counter that increments the user score and the next pose is introduced to them.

Challenges we ran into

  • Displaying the landmark mapping of various joints and body parts live through webcam video footage: solved by working with frames and sizing of landmark mapping points and lines.
  • Parsing coordinate data for the landmarks in the motion-tracking so that they can be retrieved by media pipe and passed into the backend for calculations
  • Collision detection calculations and integration with frontend feedback to check for user arm poses
  • Null data: When there are no pose/human figures detected by media pose, it ends up sending empty objects into the backend, which is trying to do calculations. This created mysterious errors at first, and we tested some edge cases to narrow down the issue. We were able to do some error catching to prevent empty objects from being passed into the backend functions.
  • Accuracy of coordinates and poses: We want the game to be challenging but also accessible and doable for users, so the collision detection calculations have to be as accurate as possible while still providing margins of error for the user.

Accomplishments that we're proud of

  • We managed to configure a Python-based backend with FastAPI that is able to parse and receive coordinate data from the webcam motion-tracking in the React frontend. The web app is able to check for a sequence of 3 different arm poses and differentiate between the left arm and the right arm. We also successfully incorporated the webcam video input into the webpage and provided live prompting as to where to position their arm through overlaid images.
  • We were able to figure out how to extract and analyze the data produced by mediapipe pose, and do accurate calculations for collision detection. Specific data points were extracted from the elbow and wrist points to create poses, which can be extended to other points on the body for more poses Successfully implemented live visual prompts and feedback, making the experience interactive and engaging.
  • Multiple rounds of the game can be played for different pre-determined poses and we can add many more!
  • Overall, we made the game and the code structure very modular, making it easier to add any features and gives a lot of room for expansion

What we learned

  • Integration of Frontend and Backend: We gained valuable experience in integrating a React-based frontend with a Python (FastAPI) backend, troubleshooting challenging communication bugs between the two
  • We learned how to use Google’s MediaPipe for real-time pose detection in front-end and how to process and interpret coordinate data for collision detection
  • We improved our debugging skills by identifying and resolving issues like null data from MediaPipe and ensuring the system gracefully handles edge cases
  • We learned to leverage Github for collaboration, resolving merge conflicts and streamlining the management of branches to avoid them.
  • We learned how to balance challenge and accessibility in game design, ensuring the exercises are both engaging and achievable for users
  • We discovered the importance of modularity in code, making it easier to extend features and maintain the project in the future.

What's next for pOSU!

So many things!

  • Do more research into the different exercises of physiotherapy and incorporate more poses into the game that help target specific muscles and joints
  • Make user profiles and enable friendly competition features so everyone can connect and create a network of fun physiotherapy!
  • Mediapipe pose can detect multiple people, we can change this number in the code to allow more players to play together on one screen!
  • Players can play against each other on different devices live!
  • Create a leaderboard to rank all public profiles, if they opt in for ranked
  • Player data can be collected in a database for storage and processing, same with in game scoring data
  • Implement a model that learns the physio conditions of each player/patient and personalizes the exercises to their needs
  • Implement a model that learns from the actions of the player (how they do the poses, how they’re getting them wrong, the time it takes to hit the poses) to access and monitor the recovery period, while giving the right encouragements to the patient!
  • Inspired by the rhythm game osu!, we can add in background music and have the poses (points) hit to the rhythm of the songs.

Built With

Share this project:

Updates