AirJam | Devpost

AirJam
Live Demo

🎵 Inspiration

After working with practical applications for Computer Vision, our team decided to do something fun and interactive for this Hackathon. AirJam is a culmination of all of our technical expertise, with each of us contributing something unique and highly technical to this project to make something truly special. Jam out with us during judging.

🎸 What is AirJam?

AirJam is a web application that allows you to play over 30 instruments, ranging from niche instruments like the Ocarina or Tinkle Bell to the entire string family. AirJam allows you to compose a catchy tune, and not just play the music, but feel the music with your own two hands.

🎼 How we composed it

AirJam was composed with three core components, which we worked on separately for the first half of hacking and merged together.

The Music handling was done by the WebAudioFont, a robust music framework built on top of MIDI encoding. The framework allowed us to set important interval timings, and use high quality fonts to make the instruments sound as realistic as possible.
Our frontend was built with React & Next.js to handle information conveyed to it by the backend webhook. Using a state management framework like React allowed us to handle timings even more accurately, and leverage modern state management paradigms to make the virtual space much more interactive. We styled our interface with Tailwind CSS for a sleek and clean design.
Our backend was a Python FastAPI Webhook, which received webcam data frame by frame and altered it using Google Mediapipe & Ultralytics YOLO Pose Detection. Using these models allowed us to process key components of the body, like your hands, and detect colissions with key components of the virtual jam space.

♩Out of Tune

Working with less intensive hardware like our laptops posed a few issues for the image processing pipeline. Working with a native resolution for the webcam was super slow, even with nano models. We had to sacrifice some frame rate responsiveness and resolution for a working demo, but making a stronger back end in the future would be great.

🎻 Our Symphony

Not only does AirJam work solo, but playing with your friends is a fun feature we are very proud of implementing. Step back and watch 3-5 people AirJam out together, and you'll see why we're excited by it.

🎹 Takeaways

We learned a LOT about real time image processing, from how to store the images and assessing which computer vision models to use. The pros and cons between Mediapipe and YOLO were fun to explore and we learned a lot.

🔊 What's our Next Album?

After HackUSF we hope to refine our backend a bit so we can formally publish AirJam. We think this project was a lot of fun to make, and even more fun to use, so we would love to put it up for others to use anywhere!