PearPiano | Devpost

A musician's best friend.
Record your spontaneous compositions.
Ask Pear any music-related questions!
Interact with the AR piano.
Drag-and-drop your notes for instant edits.

Bringing your music to life, not just to your ears but to your eyes 🎶

Inspiration 🍐

Composing music through scribbling notes or drag-and-dropping from MuseScore couldn't be more tedious. As pianists ourselves, we know the struggle of trying to bring our impromptu improvisation sessions to life without forgetting what we just played or having to record ourselves and write out the notes one by one.

What it does 🎹

Introducing PearPiano, a cute little pear that helps you pair the notes to your thoughts. As a musician's best friend, Pear guides pianists through an augmented simulation of a piano where played notes are directly translated into a recording and stored for future use. Pear can read both single notes and chords played on the virtual piano, allowing playback of your music with cascading tiles for full immersion. Seek musical guidance from Pear by asking, "What is the key signature of C-major?" or "Tell me the notes of the E-major diminished 7th chord." To fine tune your compositions, use "Edit mode," where musicians can rewind the clip and drag-and-drop notes for instant changes.

How we built it 🔧

Using Unity Game Engine and the Oculus Quest, musicians can airplay their music on an augmented piano for real-time music composition. We used OpenAI's Whisper for voice dictation and C# for all game-development scripts. The AR environment is entirely designed and generated using the Unity UI Toolkit, allowing our engineers to realize an immersive yet functional musical corner.

Challenges we ran into 🏁

Calibrating and configuring hand tracking on the Oculus Quest
Reducing positional offset when making contact with the virtual piano keys
Building the piano in Unity: setting the pitch of the notes and being able to play multiple at once

Accomplishments that we're proud of 🌟

Bringing a scaled AR piano to life with close-to-perfect functionalities
Working with OpenAI to synthesize text from speech to provide guidance for users
Designing an interactive and aesthetic UI/UX with cascading tiles upon recording playback

What we learned 📖

Designing and implementing our character/piano/interface in 3D
Emily had 5 cups of coffee in half a day and is somehow alive

What's next for PearPiano 📈

VR overlay feature to attach the augmented piano to a real one, enriching each practice or composition session
A rhythm checker to support an aspiring pianist to stay on-beat and in-tune
A smart chord suggester to streamline harmonization and enhance the composition process
Depth detection for each note-press to provide feedback on the pianist's musical dynamics
With the up-coming release of Apple Vision Pro and Meta Quest 3, full colour AR pass-through will be more accessible than ever — Pear piano will "pair" great with all those headsets!

Built With

Submitted to

Hack the North 2023
- Winner Hack the North 2023 Finalist

Created by

I worked in Unity/C# for VR features like hand-tracking, collision detection for our air piano, recording notes and recreating those blocks, and more!

Justin Lin
UW SE
I worked in Unity and C# to integrate OpenAI's Whisper package, used to build our Voice Assistant Pear. Keywords were extracted to return the correct answer to your music questions.

Sophia Sun
I worked in Unity and C# on the AI assistant Pear, integrating OpenAI's Whisper package for speech-to-text recognition, answering all your music-related questions, and more on the VR side :)

Sarah Wilson
Computer Science @ UW | Schulich Leader
I primarily worked in Unity and C#, implementing features including the VR piano keyboard itself, as well as note block generation and editing.

Emily Wang