Museify | Devpost

Architecture

Inspiration

Our team of four was inspired to create Museify after noticing a frustrating gap in existing solutions for audio-to-sheet music conversion. While tools exist, they often fail to produce sheet music that’s truly playable. As music lovers and tech enthusiasts, we knew this problem needed a better solution. With Museify, we set out to design a product that combines the power of AI with a focus on usability, creating accurate, piano-ready sheet music that brings melodies to life. Our mission is simple: to empower creators with a tool they can trust. Check the video here: https://youtu.be/G_0oaxd5r18

What it does

Museify uses advanced AI to convert audio, including uploaded files and YouTube links, into accurate, piano-ready sheet music. It streamlines the transcription process, delivering clean, playable sheets that capture the essence of the original sound.

How we built it

On the frontend, we used React.js with Three.js to create an engaging and user-friendly experience. To add an extra layer of creativity, we incorporated a free 3D piano model sourced from Sketchfab—special thanks to the original creator for making this possible. You can find the 3D piano model here.

For processing MIDI files, we relied on powerful libraries like mido and pretty_midi to prepare the data for generating piano sheets. Our backend, built with FastAPI, handles the core task of converting audio and YouTube links into accurate, piano-ready sheet music. Together, we worked to make Museify an intuitive and reliable tool for music lovers like us.

Challenges we ran into

One of our biggest challenges was ensuring the AI-generated sheet music was not only accurate but also playable for pianists of all skill levels. Working with MIDI files required extensive fine-tuning to balance musical accuracy and usability. Integrating YouTube link processing added complexity, as we had to handle varying audio qualities and formats. On the frontend, incorporating the 3D piano model with Three.js while maintaining performance and responsiveness was a steep learning curve. Despite these obstacles, we worked together, learned new tools, and found creative solutions to bring our vision to life.

Accomplishments that we're proud of

We’re incredibly proud of the milestones we achieved while building Museify. One of our biggest accomplishments was creating a Generative Adversarial Network (GAN) using PyTorch to tackle the challenge of generating accurate and playable sheet music—a complex but rewarding feat. On the frontend, we successfully imported and integrated a visually stunning 3D piano model using Three.js, adding a polished and interactive touch to our user interface. Another highlight was establishing seamless communication with MuseScore through their API, allowing us to provide high-quality, piano-ready sheet music directly to our users. Lastly, we overcame significant technical hurdles by ensuring all Python dependencies worked together without version conflicts, a crucial step in maintaining the stability and functionality of our backend. These accomplishments reflect our teamwork, problem-solving, and dedication to delivering a product we’re proud of.

What we learned

Building Museify was a journey full of learning and growth. We deepened our understanding of how Generative Adversarial Networks (GANs) work, exploring their structure and training process to generate playable sheet music. We also familiarized ourselves with PyTorch, gaining insights into its syntax and how it differs from TensorFlow. On the frontend, we tackled the intricacies of Three.js, creating an interactive 3D experience, while enhancing our proficiency with TypeScript to build more robust and type-safe code. Additionally, we learned about the structure and processing of MIDI files, which was crucial for turning audio into accurate sheet music. These lessons not only helped us bring Museify to life but also expanded our skill sets for future projects.

What's next for Museify

The next step for Museify is to implement a fully functional database to store MIDI files and PDFs uploaded by users. This will allow users to track the type of music they enjoy and easily access sheet music they’ve previously downloaded. To support this, we’ll introduce authentication, enabling users to create accounts and have a personalized experience. Additionally, we plan to add a piano tile visual component that animates the playback of the sheet music. After a user uploads audio, this feature will read the generated sheet and visually play it out, creating an engaging and interactive way to experience their music. These improvements will make Museify even more intuitive, personalized, and fun for music lovers.