IMGIC

Have you ever looked at a picture and wanted to generate a song that goes with it? IMGIC is a web app that allows you to upload a picture and generate a completely original song relevant to the image!

Comment

Inspiration

In the heart of every moment lies a unique melody: we wanted to meld the visual and auditory experiences into one. According to research, music has a positive effect across many areas of early child development including motor, language, social, cognitive, and academic abilities. Moreover, for adults, music boosts creativity, reduces stress, and improves sleep quality.

What if there was a way to generate music that matches with an image or a real-life scene? What if there was a way for the visually impaired to perceive the beauty of a real-life scene through the language of music? Now there is with IMGIC.

What it does

IMGIC allows you to upload or take an image and generate completely original music that is relevant to the image.

How we built it

We decided to go with Flask for our backend. We used two different pre-trained models. First, we used the blip image captioning model to convert the uploaded image into a description using conditional captioning. We then do some modifications to the caption and feed it into the musicgen pre-trained model as a prompt to generate completely original music.

Challenges we ran into

During our development, we definitely encountered some challenges. Firstly, throughout the project, we encountered issues making sure our program was platform and environment-independent. Secondly, we also had to change our image-to-text model from one that generated keywords from an image to one that gave a description of an image so that we could give the music generation model better data. One of our team members had to sacrifice their Window install for this project. R.I.P Barry's Windows Install 2022-2023.

Accomplishments that we're proud of

None of us are proficient in frontend development so we are proud that we were able to get a working and nice-looking frontend. We are also proud that we came up with an idea and were able to execute it with success (unlike some previous hackathons for us lol).

What we learned

We all gained more experience with Flask as well as frontend development. Additionally, we got to brush up on our machine-learning skills.

What's next for IMGIC

We would like to deploy our app on the web so that users can more easily access our project. Additionally, we are interested in making a mobile app using React Native.

Built With

Share this project:

Updates