BS - FaceOver | Devpost

GIF
Comparison
GAN Training with Loss Function
Style-mixing of generated faces with selected seeds

Inspiration

With the rise of concerns over personal data and privacy protection, videography and photography lovers have suffered greatly due to regulations and the need to seek permission to film in public locations.

The classic solution of manual tracking and blurring faces is both cumbersome and distracting in a video.

What it does

FaceOver uses AI generated faces to replace faces of those whose privacy is to be protected. The main subject(s) in the video can be left unedited.

How we built it

Fake faces were generated using StyleGAN and face detection was done using DeepFaceLab. With StyleGAN’s own pretrained algorithm, we were able to generate many faces based on different seeds. Using this, we categorised the faces and selected the ones best suited to match the faces to be replaced in the video based on aspects such as lighting, skin tone and facial features.

Each fake face was then used as a source to train a DeepFaceLab model to replace a destination face in the video while one face was selected as the main subject and not replaced.

Challenges we ran into

Generating the faces that were a good fit for the destination for the face swap was essential since it would greatly increase the success of the face swap. Currently, with the shortage of time, it was difficult to face swap faces that were not of the ideal condition (i.e. facing forward directly at the camera). Furthermore, an archive of categorised faces to match the features of the destination faces could not be generated in the limited time.

When training using DeepFaceLab, we were only able to use 1 training frame instead of a range of data for a single face, which is not ideal since this means we would not have different angles of the face to make the training model much more robust.

Accomplishments that we're proud of

We managed to train our model using a single source of a fake face to replace the destination face in the video smoothly. We successfully separated each tracked face and the face of the main subject.

What we learned

Machine learning algorithms are becoming advanced enough to smoothly replace faces from less training data. AI-generated faces are becoming indistinguishable from real faces and can be used ethically in practical applications.

What's next for FaceOver

With a wider selection of AI-generated faces to choose from, the fit can be made better for each face. In the future, with enough computing power FaceOver may be able to replace faces in real-time, similar to a snapchat filter but with more photorealism and for a larger number of faces.