GestureArt | Devpost

Drawing using the pointer gesture
With the swipe of your left palm, you can erase part of your art
After holding up your right palm for 2 seconds, the canvas will clear

Inspiration

Our group wanted to create a project that would take us out of our comfort zones and make us learn about technologies new to us. We also wanted something that could have a future impact on society. We felt that gesture art could open up a whole new world of creative expression if developed enough.

What it does

Using computer vision and neural networks, our program is able to recognize certain hand gestures and finger positions. We use these gestures and positions to allow the user to point to where they want to draw, as if their finger were a brush. We later added more features such as erasing, clearing, and color palettes.

How we built it

For most of the computer vision and neural network aspects, our project is built off of an already trained model by Kazuhito Takahashi (Repo translated by Nikita Kiselov : https://github.com/kinivi/hand-gesture-recognition-mediapipe). Using this repository with slight modifications, we mapped hand landmarks to coordinates and functions on a Pygame engine.

Challenges we ran into

Much of the complicated computer vision and neural network code was tough to understand for our level of experience. After experimenting enough, we eventually got the gist of what the programs inputs and outputs were and what we could possibly use them for. Also, working with the python engine to create a drawing game for users was challenging. Pygame is an old python game engine with a somewhat unorganized documentation, so a lot of the programming we did was through trial and error.

Accomplishments that we're proud of

We're very proud that we managed to get our project up and running, completing the main goal of the project which was to draw using only hand gestures. Computer vision and neural networks were definitely out of current level but we managed to understand enough to pull out a completed project. We had little to no experience with almost everything we used in this project, so the learning experience was tough but highly rewarding.

What we learned

From this project, we learned about the capabilities of computer vision and how to utilize their outputs in a python game engine. We learned a lot about communicating with neural networks and how to make use of their outputs. Before GestureArt, we had no idea what we were getting into when it came to computer vision or neural networks. Now, we feel like we've taken our very first small steps into a career in AI technologies.

What's next for GestureArt

There are many different paths for gesture recognition and art. We want to focus on new ways for artists to represent their creativity. Expanding on GestureArt and adding many more features that artists use in their workflow will create new mediums for creative expression. Allowing artists to incorporate their own body movements into their pieces will make their works intuitive and fluid.