Pianable | Devpost

Inspiration

As aspiring and experienced piano players, we found our main limiting factor in improvement was access to a piano. Pianos are relatively expensive and take up a decent amount of space. With our interest in computer vision, we thought creating an alternative piano for this hackathon was the best idea.

What it does

Pianable works as a portable, budget friendly alternative to electric pianos with a built-in soundfont. It takes the front view and top-down view of a paper piano using two cameras to simulate a real life piano. Several soundfonts are also available for the enjoyment of the user.

How we built it

We used Python, OpenCV to detect the hand key points, pygame to render the interface, and fluidsynth to play the sounds. The physical piano was built using nandeck and the soundfonts were found on musical-artifacts.com and edited using Polyphone.

Challenges we ran into

The accuracy of the piano is relatively low due to the fact that we need a detection for when the key is "pressed". This was improved by increasing the length of the keys. Pygame while easily accessible, lacks the complex features that were required to simply the workflow of the project. Playing Pianable also had its troubles. We had to test on many occasions the limits of our finger and key detection systems. For example, the fingers should be spread and not close together nor scrunched. These cause Pianable to play multiple notes when you are only playing one and the knuckles being treated as fingers respectively.

Accomplishments that we're proud of

Pianable is able to (mostly) recognize when a key is being pressed and play the correct corresponding note. By implementing multiple soundfont presets, we were able to not only create a working piano, but also one that plays funny sounds. There was also an extensive amount of math required to compute the location of the keys and the "press" detection.

What we learned

We learned about soundfonts, how to manage multiple camera inputs, and how to use pygame to make an app interface. We also gained more experience with real time computer vision models. Working as a team, we learned how to problem solve together, and parallelize tasks.

What's next for Pianable

We wish to make Pianable significantly more accurate by training a model to recognize when the finger is "pressing" the key. Another way to improve Pianable is to possibly create a single camera version which is placed at a diagonal view which can detect both the finger press and what key is supposed to be played. Finally, more soundfonts could be added to Pianable.