15s | Devpost

GIF
Our logo.
The main page. Here, the user an upload an mp3 file.
On this page, users can upload many images at once.
Here, users can reorder, remove and add single images.
This page gives the users a preview of the final video. They can download the video from this page.

Inspiration

First with Vine, then with Musical.ly which turned into TikTok, and then with Instagram Reels, it is undeniable how these seemingly innocent short videos have taken the world by storm. You love the content that's shared on these platforms but you want to be unique in your storytelling.

What it does

15s takes any piece of music and compiles an aesthetically crafted short video. As the name implies, it produces a video with a length of 15 seconds to capture the moments you want to highlight in your life into something that can be shared with family and friends online. Whether that be a grad trip, a vacation, a honeymoon, or just your daily life, 15s can capture these moments and share them with the world..

The web application takes a song of your choice, extracts a 15s segment of the chorus using musical analysis, and divides the segment according to the beat. The user supplies an image for each subdivision, and optionally chooses a filter for the final video. Upon submission, 15s combines the music and the images into a polished, TikTok-style video.

How we built it

We use a Python Flask backend and a Bootstrap Vue frontend for our web application.

On the front-end, we upload and order the files before sending them to the back-end. When completed, the front-end shows a preview of the video and lets the user download the final result. Using HTTP requests, the front-end calls two endpoints in the Flask application.

The first endpoint allows the user to upload audio to the backend. Upon receiving the audio, the application uses pychorus, pydub, and librosa to extract the chorus and BPM of the song. First, it applies a Fourier transform to convert the audio into a vector representation of the frequencies present in the song. Then, using the norm of the difference between vectors as a similarity measure, we isolate the most similar-sounding segments of the song, which typically correspond to the choruses. Finally, the beat track is separated from the rest of the chorus and used to determine the BPM. Using this analysis, the back-end tells the front-end how many images should be uploaded and how long each image will be shown for.

The second endpoint allows the user to upload images to go along with the audio they uploaded previously. In the backend, moviepy is used to combine the images and the song according to the earlier analysis. A unique cookie included in the first endpoint’s response is used in the second request to match images to audio. moviepy also applies filters to the video as specified by the user. The resulting video is sent back to the front-end.

Challenges we ran into

Our biggest challenge was passing large files (both audio and video files) between the front end and back end. To do so, we learned to implement file upload and download functionalities on the front end. Additionally, we learned to encode and decode files using base64 encodings in order to send them in HTTP requests. It was also challenging to use the musical analysis libraries, as they only accept/produce specific file formats and are not very well documented.

Accomplishments that we're proud of

By designing an intuitive user interface, we managed to make video editing a more simple and fun task. Additionally, we are happy that our tool works well on multiple platforms, including mobile devices.

We are proud of our musical analysis results, which we fine-tuned through experimentation to effectively isolate the chorus and beat of songs. We also optimized the length of time each image is shown while ensuring that the transitions remain in beat to the music to obtain the best final result.

What we learned

We learned how to send files of different formats between the front-end and back-end of a web application. Additionally, we explored several different libraries that allow the analysis of audio files. Some of us were previously unfamiliar with using git, Vue and Flask.

What's next for 15s

Currently, 15s is a web application. In the future, we would like to create native applications for Android and iOS to make 15s more accessible to users on mobile devices. Additionally, we would like to make the tool more flexible by adding the following features: