TuneScriber

Convert your favourite songs into sheet music

Comment

Inspiration ✨

For many people, playing music serves as a language where they can communicate complex feelings and relive their most cherished memories. It has remained a therapeutic medium for centuries, where people can express their vulnerabilities and seek comfort among others. However, accessing sheet music is challenging for several musicians due to the high cost of music libraries and the difficulty of transcribing music. Thus, we created TuneScriber, a web app that allows users to convert their favourite songs into sheet music! It separates the audio into distinct tracks based on instrumentation, which provides musical education more accessible to musical groups and solo musicians alike.

What it does 🚀

Simply upload your audio file and watch the magic happen! Our product will split the audio into distinct tracks, enabling you to play each part separately, preview the sheet music, and download it as a PDF file.

How we built it 🔧

  • We used React, TailwindCSS, and Framer Motion to create a scalable project that provides an interactive and responsive experience for everyone!
  • Flask was used for server-side integration by sending the audio file to the backend for track separation, audio-to-MIDI conversion and sheet music generation.
  • We used Music.ai to split the song into individual tracks based on instrumentation, including vocal, bass, string, and more!
  • Next, the generated track files are fed into Basic Pitch for MIDI conversion, which enables easy editing and integration with other music production software (i.e. MuseScore).
  • Finally, these MIDI files were converted into sheet music using Mido, which provides musicians with a visual representation of music with traditional notation. We also used Whisper.ai for reliable lyric generation, where arrays of verses are routed to the frontend so musicians can enjoy their favourite songs.

Challenges we ran into 🥲

  • Lots of typos! While they are silly mistakes, we spent hours trying to fix bugs that ended up being spelling and grammatical errors.
  • Whether it's CORS, merge conflict, or API integration, we ran into several challenges when integrating our front-end and back-end. A big contributor was the system differences between MacOS and Windows, where Pip package, directory, and dependency errors extended our hacking process by 10+ hours :(

Accomplishments that we're proud of 🌟

  • Successfully creating a responsive and intuitive UI to enhance the user experience by providing scroll features, engaging transitions, and drag-and-drop functionality
  • Integrating uploaded files onto our backend server for audio processing, which enables our product to convert audio files into MIDI and download them as sheet music PDFs
  • Despite how frustrated we were, we stayed resilient throughout our Hack the 6ix journey and overcame every obstacle we encountered this weekend!

What we learned 🍎

  • How to create scalable React components to simplify the front-end process! We also explored various UI libraries like Aceternity UI to create smooth, responsive elements for enhanced user experience (i.e. moving border buttons).
  • How to resolve merge conflicts and collaborate effectively using Git. Each team member created their branch to work on their tasks, and we merged these changes to ensure punctual project completion.
  • The integration of several packages, dependencies and APIS, including Basic Pitch and Whisper.ai. for server-side routing. It allowed our product to create individual MIDI files and deliver lyric transcriptions for enhanced musical experience.

What's next for TuneScriber 👀

  • Adding an in-app MIDI editor to enable user personalization! Whether it’s changing the key, the tempo, or the difficulty, they should have the opportunity to tailor results to fit their needs!
  • Since the musical timbre of drum sets functions differently compared to other instrument counterparts, optimizing our product to generate percussion sheet music can greatly improve user experience and make music more accessible for large ensembles.
  • Utilize database integration to store previous conversions so users can access their sheet music whenever and wherever!
  • Adding lyrics to the sheet music (rather than as a separate file/pop-up) so musicians can play their instruments and sing their hearts out 🎤
  • Improve conversion efficiency by reducing the number of API calls to turn our multi-step process into one-step

Built With

Share this project:

Updates