Diffusion Music Generator

Inspiration

We love music and want to make it easy for everyone to create their own. Creative quality should not be limited to those with access to expensive equipment.

What it does

It is an Android, iOS, and web app that generates never-heard-before songs using a custom diffusion machine learning model trained on thousands of hours of songs. Seeing how diffusion models have made great advances in AI image generation, we applied the same technology to music.

How we built it

The front-end is written in React Native (using the Native Base UI framework) and the back-end primarily uses Flask and PyTorch. The machine learning model was trained on thousands of hours of popular songs and is using diffusion to generate audio.

Challenges we ran into

We swapped roles for this project: people familiar with the back-end worked on the front-end and people familiar with the front-end worked on the back-end. We ran into some complicated React state issues and found there to be a lack of existing components for React Native that were also compatible with React Native web. Thus, for example, we had to custom write our audio player. Additionally, our in-painting function was more difficult to create than expected and involved complex audio encoding logic.

Accomplishments that we're proud of

We're proud of quickly learning how to use React Native to make a basic application. We're also excited about our machine learning inference code that we developed into an API for use by our front-end. Thirdly, we wrote a custom distribution generation script that allows us to tune the model's output. All thrilling!

What we learned

We learned about modern web development with React and React Native. Additionally, we learned about state-of-the-art AI architectures (diffusion) and how to connect large machine learning models to a front-end interface.