Mindful Melody VR

• 

  Galaxia from MLH tries an early version of the app

Inspiration

The best hacks are unexpected combinations of technology that are just plain fun, so we got our inspiration from that vibe. The original idea: A VR app. Then someone on the team suggested adding a piano keyboard and the rest followed from there.

What it does

In the VR scene, a digital piano is mapped on top of the physical one (tracked via Quest controller). You can then play the instrument physically and see it animate in the digital world, with the world changing based on what's being played. Try different melodies and combinations of notes to see what happens!

How we built it

Much love and pain. Mapping the digital piano on top of the physical one took a lot of careful measurement and alignment in order to create a piano based on the position of the Oculus Touch controller. Hands are tracked using the Quest's built-in hand tracking. A proprietary heuristic music analysis algorithm (if statements) decides how to change the scene based on what's played.

Challenges we ran into

Hardware hacks are notorious for "integration hell," where getting everything to work together fluidly is very difficult and painful when it really doesn't seem like it should be. Getting the digital world to cooperate with the physical world isn't any different and honestly the fact that the digital piano even exists is incredibly impressive.

Accomplishments that we're proud of

Created a beautiful app that combines the Metaverse with the... Realityverse? Lifeverse? Oh, it's just called the Universe? Anyways combining those was an impressive technical feat that we pulled off with 0 budget unlike those hacks at Valve with their fancy VR greenscreen cameras and controller finger tracking and...

What we learned

The most recent Unity or C# experience any of us has was back in 2016. We now kinda know how to use Unity. Maybe.

What's next for Mindful Melody VR

We were going to use ML to do sentiment analysis on the music but decided on a heuristic solution instead to simplify the scope. It would be cool to revisit the idea with more time.

Built With

• c#
• oculus
• unity