subsync

Automagically synchronize subtitles with video, aligning them to the correct starting point.

This is my submission for HackIllinois 2019.

Turn this:	Into this:

Install

First, make sure ffmpeg is installed. On MacOS, this looks like:

brew install ffmpeg

Next, grab the script. It should work with both Python2 and Python3:

pip install git+https://github.com/smacke/subsync

Usage

subsync video.mp4 -i unsynchronized.srt > synchronized.srt

subsync video.mp4 -i unsynchronized.srt -o synchronized.srt

Although it can usually work if all you have is the video file, it will be faster (and potentially more accurate) if you have a correctly synchronized "reference" srt file, in which case you can do the following:

subsync reference.srt -i unsynchronized.srt -o synchronized.srt

Whether to perform voice activity detection on the audio or to directly extract speech from an srt file is determined from the file extension.

VLC Integration

To demonstrate how one might use subsync seamlessly with real video software, we developed a prototype integration into the popular VLC media player, which was demoed during the Hackillinois 2019 project expo. The resulting patch can be found in the file subsync-vlc.patch. Here are instructions for how to use it.

First clone the 3.0 maintenance branch of VLC and checkout 3.0.6:

git clone git://git.videolan.org/vlc/vlc-3.0.git
cd vlc-3.0
git checkout 3.0.6

Next, apply the patch:

wget https://github.com/smacke/subsync/raw/master/subsync-vlc.patch
git apply subsync-vlc.patch

Follow the normal instructions on the VideoLAN wiki for building VLC from source. Warning: this is not easy.

You should now be able to autosynchronize subtitles using the hotkey Ctrl+Shift+S (only enabled while subtitles are present).

Future Work

The prototype VLC patch is very experimental -- it was developed under pressure and just barely works. The clear next step for this project is a more robust integration with VLC, either directly in the VLC core, or as a plugin.

Credits

This project would not be possible without the following libraries:

ffmpeg and the ffmpeg-python wrapper, for extracting raw audio from video
VAD from webrtc and the py-webrtcvad wrapper, for speech detection
auditok, for backup audio detection if webrtcvad misbehaves
srt for operating on SRT files
numpy and, indirectly, FFTPACK, which powers the FFT-based algorithm for fast scoring of alignments between subtitles (or subtitles and video).
Other excellent Python libraries like argparse and tqdm, not related to the core functionality, but which enable much better experiences for developers and users.