MelodAI | Devpost

Inspiration

Our inspiration for creating this app stemmed from our deep appreciation for music and the desire to explore the hidden layers of emotion and meaning within song lyrics. We wanted to offer music lovers a tool that could provide deeper insights into the songs they enjoy, uncovering the stories that often go unnoticed.

What it does

Our app allows users to search for songs, displaying their lyrics and a short summary of the track. It uses cutting-edge generative AI technology to decipher the meaning and emotions embedded within lyrical ranges of your choosing. Users can simply select lyrics from any song, and the app provides a nuanced interpretation, offering insights into the song's themes, moods, and even the artist's intentions. It transforms lyrics into vivid narratives, providing a richer understanding of the music.

How we built it

We started by using Genius for indexing songs and scraping their lyrics. We then moved on to generation with Cohere's LLM, using a custom prompt to summarize both whole songs, and specific lyrics within a track. To provide a category for viewed music, we trained a custom classifier, again using Cohere, using diverse examples from the music world. To offer more personalization, we integrated the Spotify API, providing easy access to quick analysis of your favorite tracks.

Challenges we ran into

Fine-tuning the AI model prompt and temperature to provide the most useful responses with the least hallucination.
Working around language model restrictions surrounding sensitive content that may be present in song lyrics.
Gathering data to train the category classifier.
Creating aesthetically pleasing UI in a limited timeframe.

Accomplishments that we're proud of

Creating a cromulent React application with nearly no frontend experience.
Harnessing the power of generative AI for non-traditional uses.
Deploying the frontend and backend in a serverless environment.

What we learned

The challenges that come with communicating in an application with a completely separate yet dependent frontend and backend.
How to interact with AI models over an API.
Further insight into how popular JavaScript/Python frameworks like React, Flask, etc. interact with each other.

What's next for MelodAI

Further expanding the rudimentary Spotify integration and adding support for more music platforms.
Expanding on the idea of music classification through non-traditional categories.
Improving lyric analysis by exploring prompt context and more specialized LLMs.
Venture into other subindustries such as recommendations based on taste.