About re:wind

Do you have fond memories of watching corny TV shows with unrelenting laugh tracks and other silly sound effects? Although the laughing and clapping seemed never-ending, they brought an irreplaceable sense of childhood. re:wind allows you to re-experience those carefree moments with scenes from your very own life! Simply upload a video taken by you, and our algorithm will insert appropriate sound effects throughout the recording. In addition to providing a fresh new perspective to your own experiences, re:wind saves everything it makes for you to revisit on a rainy day.

Inspirations

When we thought of nostalgia, we thought of television shows from a bygone era of warm and bubbly programming. This, combined with the fact that nostalgia is derived from the fondness with which we view our past experiences, served as the main inspiration for our project. We wanted to create something that could leverage both forms of nostalgia in order to soothe our feelings in a hectic world, and so re:wind was born.

How we built our project

Once you click the DVD drive and upload a video, our React.js frontend sends its audio to our Flask backend, which uses multiple third-party resources and our custom algorithm to transcribe the audio and cut the text into meaningful sentences.

To identify the perfect timestamp for specific soundtracks and understand the tone of each sentence, we leveraged Cohere’s LLMs and Google’s Speech-to-Text APIs. More specifically, we used Google to produce a transcript of the audio with timestamp data. Furthermore, we used Cohere's Generate functionality for transcript refinement and Cohere’s Classify functionality for label extraction and fine-tuned their language model with our text-to-mood dataset.

Challenges we ran into

Pinpointing exactly what information we need and how to retrieve it from our numerous APIs
Passing the audio file from frontend to backend and converting audio from mp4 to WAV as Google favors WAV audio files
The processing speed of video editing