Telekinesis | Devpost

Pac-Man Gameplay
Gestures for Pac-Man Gameplay
Gestures for Flappy Bird
Leaderboard System

Inspiration

Pac-Man and Flappy Bird are two of the most famous retro games of the past 100 years. We decided to recreate these games, but with a twist. These games were integral parts of our childhood, and it was great to put our technical skills to use in a project that was both challenging and fun.

What it does

Our games are a recreation of Pac-Man and Flappy Bird but without the use of a keyboard. The player uses his/her hand to direct the Pac-Man either top, down, left, or right. The player then opens and closes his/her hands into a fist to represent the jumping of the bird. The user scores are displayed on the screen and a frontend application displays the score leaderboards for the respective games.

How we built it

Our project connects various facets of programming and math. We used primarily Python as our programming language. We utilized Django as a backend that includes CRUD functionality for user data. Taipy was used for the frontend, which provided incredibly easy to use and beautiful designs. PyGame was used for the general game logic using OOP and computer vision libraries like OpenCV/Tensorflow/Mediapipe were used together to handle hand gesture recognition.

Challenges we ran into 😈

We faced many challenges both large and small. One large and expected challenge was training a Convolutional Neural Network that would accurately detect hand signs. We initially implemented a ML solution that trained on thousands of images of our own hand using Google Teachable Machines but we were disappointed by the speed and accuracy of the model when using it to run our games. Fortunately, we were able to implement a completely new technique using hand landmark detection and linear algebra techniques to optimize our network solution. This let us find the direction of the finger for player direction and the distance of landmark points from the mean point on the hand to detect whether the hand was open or closed. To handle the issue where we could not get consistent results depending on how far the hand was from the camera, we divided our distance between the total distance between the top of our finger and our wrist to ensure consistent accuracy in our inference. The other major challenge was about optimizing for efficiency, mainly for the Pac-Man game. In order to move the Pac-Man, we had to translate the predicted hand signal from the Neural Network to PyGame. We tried many approaches to do so. One was using Django REST Framework to make a POST request for every hand sign recognized by the ML model. Another was for the ML model to write the predicted hand signal to another file, which would be read by PyGame. However, these approaches were slow and resulted in high latency and thus slow results on the GUI. To solve these issues, we utilized concurrent programming by implementing multithreading. Thanks to this, we were able to simultaneously run the game and ML model with great efficiency.

Accomplishments that we're proud of

We were all able to learn from each other and expand upon our interests. For example, we often learn complex math in school, but we were able to find real life use cases for linear algebra to determine the direction of our finger which we are quite happy about.

What we learned

We each learned varying things, whether it is learning about the intuition behind a neural network, the way to make API requests, and learning the intricacies of version control using Git.