StyleAlign: Analysis and Applications of Aligned StyleGAN Models

Check our video here:

StyleAlign: Analysis and Applications of Aligned StyleGAN Models
Zongze Wu, Yotam Nitzan, Eli Shechtman, Dani Lischinski
https://arxiv.org/abs/2110.11323

Abstract: In this paper, we perform an in-depth study of the properties and applications of aligned generative models.
We refer to two models as aligned if they share the same architecture, and one of them (the child) is obtained from the other (the parent) via fine-tuning to another domain, a common practice in transfer learning.
Several works already utilize some basic properties of aligned StyleGAN models to perform image-to-image translation.
Here, we perform the first detailed exploration of model alignment, also focusing on StyleGAN. First, we empirically analyze aligned models and provide answers to important questions regarding their nature. In particular, we find that the child model’s latent spaces are semantically aligned with those of the parent, inheriting incredibly rich semantics, even for distant data domains such as human faces and churches.
Second, equipped with this better understanding, we leverage aligned models to solve a diverse set of tasks.
In addition to image translation, we demonstrate fully automatic cross-domain image morphing.
We further show that zero-shot vision tasks may be performed in the child domain, while relying exclusively on supervision in the parent domain.
We demonstrate qualitatively and quantitatively that our approach yields state-of-the-art results, while requiring only simple fine-tuning and inversion.

pretrained checkpoint

The pretrained checkpoints could be downloaded from here. The FFHQ model is from StyleGAN2 repo. The FFHQ512, FFHQ512_dog, FFHQ512_cat, FFHQ512_wild models are from StyleGAN2-ada repo. Other models are trained or fine tuning by ourselves.

To download all checkpoints:

gdown --fuzzy 'https://drive.google.com/drive/folders/1MqCHQ6Yx-eon-3fu1g_AGjpyAUmzH6Jy?usp=sharing' -O /checkpoint --folder

Image-to-Image Translation

source_img_path='./example/dog/'   
source_path='./img_invert/ffhq512_dog/z/'  # path for saving inverted latent codes and images
target_path='./img_invert/ffhq512_dog/translate/cat/' #path for saving translation images 

source_pkl='./checkpoint/ffhq512_dog.pkl'
target_pkl='./checkpoint/ffhq512_dog_cat.pkl'

compare_html='./img_invert/ffhq512_dog/translate/cat.html'

python projector_z.py --outdir=$source_path  \
 		      --target=$source_img_path \
		      --network=$source_pkl


python I2I.py --network $target_pkl \
  	      --source_path $source_path \
  	      --target_path $target_path	


python Compare.py --source_img_path $source_img_path \
  	      --source_path $source_path \
  	      --target_path $target_path \
  	      --save_path $compare_html 	

Shared Semantic Controls Between Parent and Child Models

Image Translation

Cross-domain Image Morphing

Knowledge Transfer from Parent to Child Domain

GitHub

View Github