JazzCasters

Jazz Casters is an interactive music-driven spellcasting tower-defense game where players use hand gestures to compose music and combat the enemies in real time.

Comment

Inspiration

JazzCasters was inspired by the idea that music and movement can be just as expressive as traditional game controls. We were drawn to how performers communicate intention through gesture, and we wanted to translate that expressiveness into VR combat. Rhythm games and spellcasting mechanics inspired us to explore a hybrid space where sound is not just feedback, but the primary system driving gameplay.

What it does

We built a VR prototype where the player stands at the center of a stage surrounded by four instrument towers. Using hand tracking, the player performs simple, intuitive gestures to cast spell orbs, which can be dragged onto instruments to modify their musical output and attack behavior. Once enhanced, instruments automatically attack incoming enemies using sound- and rhythm-driven effects, turning music into both a weapon.

How we built it

The project was built in Unity using Meta Quest hand tracking for gesture recognition and real-time interaction via Arduino. We focused on a small but readable gesture set to ensure reliability within a hackathon scope, mapping each gesture to a spell and musical effect. Audio systems were tightly coupled with gameplay logic so that changes in sound directly reflected changes in combat behavior, reinforcing music as a core mechanic.

Challenges we ran into

One of our biggest challenges was balancing expressive input with technical reliability—designing gestures that felt natural while still being consistently recognized by the hardware. We also learned the importance of scope control: by prioritizing hand-gesture tracking and music activation, we were able to polish the core interaction instead of overextending into secondary features. Overall, the project taught us how powerful focused prototyping can be when validating a single, strong gameplay idea.

Accomplishments that we're proud of

We were able to integrate each team member's passion and their skills into one project. We enjoyed seeing each other's progress and achievements, then came together and discussed the best way to intertwine the smaller parts together.

What we learned

We learned that individuals with different skills and backgrounds can come together and create awesome projects that can bring people joy.

What's next for JazzCasters

We took accessibility into account from the beginning, but we immediately knew that it should be on our "nice-to-have" list. So for our next step, we would love to have a voice command feature that would help out users who have difficulties using their hands.

Built With

Share this project:

Updates