JampJax | Devpost

JampJax Example Screen

Inspiration

JampJaxs was inspired by both Cookie Clicker and step tracker applications. We wanted to create a game with similar progression/gameplay as Cookie Clicker, except it involves exercise tracking similar to step tracker applications.

What it does

JampJaxs is a video game that accesses your laptop's webcam and detects whether you are doing an exercise (jumping jack, squats, or high knees). If you are, you will gain a certain number of "Jacks" in the game (which act as both score and currency). You can also buy upgrades in the shop that cost Jacks which increase the number of Jacks you make.

How we built it

JampJaxs was built using Google's Media Pipeline machine learning library to do 2D motion tracking through the user's laptop webcam. The angles between the landmarks provided by Media Pipeline are used to detect whether the user is doing a jumping jack, a squat, high knees, or no exercise. The game interface was built using Godot Engine 4.0, and communication between the game interface and Python script backend was done using UDP packets.

Challenges we ran into

Out of screen land marks: When landmarks are located outside of the webcam's view (e.g. the webcam can't see your hand) the model attempts to predict the location of the landmark. There was no explicit differentiation between the landmarks visible on the webcam and landmarks that were predicted, which led to highly inaccurate exercise predictions when required landmarks were out of webcam visibility. To solve this problem, when a landmark is outside of the webcam's view, all exercises that require that landmark are automatically not considered.

Accomplishments that we're proud of

We are proud of:

Learning the Media Pipe framework well enough to use it for exercise detection, despite not having any previous experience with 2D motion tracking.
Implementing an algorithm from scratch to use for exercise detection, based on the landmarks provided by the Media Pipe framework.
Connecting a Python ML backend with a game interface built on Godot using UDP packets.

What we learned

2D motion tracking by combining webcam feed with the MediaPipe framework.
How to detect exercises using landmark coordinates provided by MediaPipe framework.
How UDP packets could be used to connect a Godot UI and Python backend.