Mixart | Devpost

Inspiration 💡:

The idea behind Mixart was to explore the unique interactions between two separate mediums of music and visual art and to bridge them closer together. Our goal was to develop a completely new platform that enables users to experience their artwork in an entirely fresh manner.

What it does ❓:

Mixart is an AI-powered platform that takes user-created doodles and a song as inputs. The doodles are processed by Bard, a now multimodal LLM, capable of interpreting the visual elements and emotions behind the artwork. These interpretations are then converted to music, based on the song that the user inputted. This effectively creates a new remix, all following the sentiment of the doodle.

How we built it 🛠️:

Mixart's foundation lies in three main technologies: Google Bard, HuggingFace MusicGen, and OpenAI Clip. We utilized Bard's multimodal capabilities to extract meaning from the doodles and translate them into musical elements. We then used our MusicGen endpoint to combine a song and Bard's description into a song remix.

To enhance user engagement, we also introduced a user feedback system that enables users to share their thoughts on the generated remixes. This feedback and the encoded images from Clip are stored in a LanceDB vector database. When a new user submits a doodle, Mixart searches in the LanceDB database to find the most similar pictures and feedback. This context is then passed to Google Bard, adding to the prompt to generate more refined and enriched musical outputs.

The frontend is built with React, and connected to the backend using Python Flask.

Challenges we ran into 💪:

Because Bard's multimodal capabilities are so new, it was pretty difficult to quickly learn and understand how to utilize the image-to-text services. Additionally, creating our MusicGen endpoint was quite difficult, as our whole team lacked experience with HuggingFace. Their inference endpoint caused a lot of issues that we spent the large majority of time debugging.

What's next for Mixart 🚀:

Mixart's adventure is far from done. In order to create a healthy community of music and art lovers, we intend to increase the platform's functionality in the future by introducing project galleries, allowing users to discover other creators' musical remixes and styles. Other collaborative elements, such as enabling artists to remix each other's works, would contribute much to this community.

Mixart is positioned to grow into a groundbreaking platform that, through ongoing innovation, completely alters the way we see and interact with art and music.