Gen.re | Devpost

The first 10 epochs of training - it took over 90 minutes!
Our homepage
Our virtual piano, which can both create and display MIDI.
The tedious process of data scraping...

Inspiration

As a group of students that enjoy listening and performing music, we understand that crafting harmonies that seamlessly resonates with a melody is no small feat. It's a painstaking process that demands technical expertise and hours of hours sometimes struggling to find that perfect harmony. This roadblock is not confined to any level of instrument, expertise nor exerptise. Having attended various variety shows or music shows where many of our peers had the oppournity to perform the music at shows, yet backed down because they were not confident with their peices and harmonies, we realized that this is an issue that is restring people from enjoying music to its fullest.

Being musicians ourselves, we wanted our project to solve this problem, and bridge the gap between what we imagine and what we can create, allowing our creativity to soar without constraints.

What it does

At it's core, Gen.re is a harmonization engine that takes your right hand melodies and weaves intricate left hand accompaniments that elevate your musical creations to new heights.

Gen.re accepts a user's input melody - either as a MIDI file or played on the virtual piano - and also a genre that the user would like to harmonize with. To process the melody to generate an appropriate harmoney, we created and trained over 12 models, scraping the internet for thousands upon thousands of tracks. Using the corresponding model, Gen.re generates a suitable harmony track, taking into account both the inputted melody and the genre This program is not just a tool, but it's a musical partner that understands your artistic intent and translates it into a complete song.

How we built it

When we started the project, we had some experience with ML, but we were very unfamiliar with musicology libraries, such as MIDI and other MIDI libraries in JavaScript and Python.

So, our approach as a team was to individually explore different music libraries. Eventually, we converged on 12 similar but outstanding models, which we gradually optimized and trained using hours and hours of MIDI files.

Given our limited time, we trained the model for about 8 hours overnight, using pairs of {Melody, Accompaniment} as training data.

Challenges we ran into

The two main challenges that we ran into was being overwhelmed by the abundance of ML models and not knowing where to find suitable MIDI datasets.

Since many ML models were fundamentally different, it was very hard to choose exactly which ones to use. Additionally, MIDI datasets come in a variety of formats, so we struggled to choose datasets that matched with certain models.

A few other minor challenges that we ran into were:

Learning the Seq2Seq ML model, which is quite different from traditional neural networks (which is what we have experience with)
Not knowing how to parse MIDI into a format that ML models could understand
Trouble integrating the frontend (the virtual piano) with the backend (the produced MIDI file)

Accomplishments that we're proud of

We are very proud of the variety of modern technologies that we used to build our projects. Many of the music ML models that we chose were very recent - some of which are only a few months old - and were developed by companies like Google and Facebook.

Additionally, in retrospect, we're very proud of our teamwork - we worked well as a team to divide up the workload, which allowed us to divide and conquer the two main challenges mentioned earlier.

As a team, we split ourselves into two pairs to handle ML and MIDI respectively. In total, we must have explored at least 20 different models and 30 MIDI datasets, which gave us a lot inspiration of our project's direction.

Lastly, we are proud of our frontend, which made use of 3D geometry to display the virtual piano (which we also built ourselves!). Parsing MIDI is one thing, but displaying it on the piano was a completely separate challenge that we handled well.

What we learned

This experience taught us a lot about music softwares and ML models. By the end of the project, we were all very familiar with MIDI, Jupyter Notebook, Google Colab, and other libraries that we used.

Additionally, we learned many strategies for data collection. We used data scraping software such as Selenium, processed the MIDI files using many encoding / decoding methods (16th note hashing, one-hot encoding), and tweaked training parameters to achieve the highest model accuracy.

What's next for Gen.re

Given that we had limited resources to train the AI during the hackathon, one of our main points of improvement would be expanding the dataset, variety of genres, and training higher model accuracy.