Closed-Form Factorization of
Latent Semantics in GANs
Yujun Shen,  Bolei Zhou
The Chinese University of Hong Kong
[Paper]  [Code]  [Colab]
Image
* The interface is powered by StreamLit.
Overview
In this work, we propose a closed-form algorithm, called SeFa, for unsupervised latent Semantics Factorization in GANs. More concretely, we investigate the very first fully-connected layer used in the GAN generator. We argue that this layer actually filters out some negligible directions in the latent space and highlights the directions that are critical for image synthesis. By finding these important directions, we are able to identify versatile semantics across various types of GAN models with an extremely fast implementation (i.e., less than 1 second).

Image
Fun Animations
The following animations are created by manipulating the versatile semantics unsupervisedly found by SeFa from GAN models trained on various datasets.

Anime Faces

Pose Mouth Eye
Image Image Image

Cats

Posture (Left & Right) Posture (Up & Down) Zoom
Image Image Image

Cars

Orientation Vertical Position Shape
Image Image Image

Below shows the full demo video of our manipulation interface using SeFa.
BibTeX
@inproceedings{shen2021closedform,
  title     = {Closed-Form Factorization of Latent Semantics in GANs},
  author    = {Shen, Yujun and Zhou, Bolei},
  booktitle = {CVPR},
  year      = {2021}
}
Related Work
Image
Y. Shen, J. Gu, X. Tang, B. Zhou. Interpreting the Latent Space of GANs for Semantic Face Editing. CVPR 2020.
Comment: Interprets the face semantics emerging in the latent space of GANs with the help of off-the-shelf classifiers.
Image
L. Goetschalckx, A. Andonian, A. Oliva, P. Isola. GANalyze: Toward Visual Definitions of Cognitive Image Properties. ICCV 2019.
Comment: Controls the latent space of GANs to increase the memorability of synthesized images.
Image
A. Jahanian, L. Chai, P. Isola. On the "Steerability" of Generative Adversarial Networks. ICLR 2020.
Comment: Shifts the data distribution by steering the latent code to fit camera movements and color changes.
Image
C. Yang, Y. Shen, B. Zhou. Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis. arXiv preprint arXiv:1911.09267.
Comment: Explores the emergent semantic hierarchy in scene synthesis models.
Image
A. Voynov and A. Babenko. Unsupervised Discovery of Interpretable Directions in the GAN Latent Space. ICML 2020.
Comment: Interprets meaningful directions in GAN latent space by unsupervisedly training a direction reconstructor.
Image
E. Härkönen, A. Hertzmann, J. Lehtinen, S. Paris. GANSpace: Discovering Interpretable GAN Controls. arXiv preprint arXiv:2004.02546.
Comment: Unsupervisedly discovers the latent semantics learned by GANs using PCA.