Inspiration
As engineers who must suffer through countless signal processing and controls classes, we decided to explore the inane notion that what we learn in classes could actually be tangibly useful. Despite strong setbacks from the existence of the C# programming language, we set out to explore how we could use signal processing and acoustics to revolutionize the high-brow lyric video market.
What it does
Frisson is an analysis pipeline that first, takes lyrics and uses audio spikes in the time domain to match the words to music. Next it takes those words and analyzes the frequency spectrum at each moment to understand what's going on in the music when the word is being said. Finally, it applies animations, colors, and fonts based on it's frequency analysis and further displays the lyrics in a unique manner.
How I built it
We split up the heavy lifting in analysis with Cheyenne handling Lyric matching and Rahul handling frequency response analysis, both in Python. Then, we worked together to turn our analysis into visual effects on text, and properly overlay the text over a music video in Unity.
Challenges we ran into
C# is an awful programming language. Unity is terrible for viewing anything with quality.
Accomplishments that we're proud of
We were pleasantly surprised at how artistically inclined our system is. We also learned quite a bit about how to program for the unity framework in an efficient manner, because we had ensure our program could quickly display the next piece of text with appropriate formatting without slowing the whole thing down. This also meant creating more filters for the types of words displayed and the length needed so that the word would be picked up in updating the text.
What I learned
We both learned how to work with Python to do audio analysis, as well as learned to use C# and unity.
What's next for Frisson
We intend to apply our algorithm to analyze Humanity's greatest acoustic works including "my longest yeah boy ever (10 hours)" and "Umbrella" by Rihanna.
Log in or sign up for Devpost to join the conversation.