HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping

IJCAI, 2021.

Yuhan Wang^1,2* Xu Chen^1,3* Junwei Zhu¹ Wenqing Chu¹ Ying Tai^1†
Chengjie Wang¹ Jilin Li¹ Yongjian Wu¹ Feiyue Huang¹ Rongrong Ji^3,4

¹Youtu Lab, Tencent ²Zhejiang University
³Media Analytics and Computing Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University
⁴Institute of Artificial Intelligence, Xiamen University
hififace.youtu@gmail.com

The face in the target image is replaced by the face in the source image. All results are generated by our end-to-end model HifiFace-512. If you want to see more celebrity demos, please refer to our supplementary material.

Introduction

In this work, we propose a high fidelity face swapping method, called HifiFace, which can well preserve the face shape of the source face and generate photo-realistic results. Unlike other existing face swapping works that only use face recognition model to keep the identity similarity, we propose 3D shape-aware identity to control the face shape with the geometric supervision from 3DMM and 3D face reconstruction method. Meanwhile, we introduce the Semantic Facial Fusion module to optimize the combination of encoder and decoder features and make adaptive blending, which makes the results more photo-realistic. Extensive experiments on wild faces demonstrate that our method can preserve better identity, especially on the face shape, and can generate more photo-realistic results than previous state-of-the-art methods.

Videos

1-min Presentation Video

Selected Video from FF++

Publication

Paper - ArXiv - pdf (abs)
If you find our work useful, please consider citing it:

@inproceedings{ijcai2021-157,
    title     = {HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping},
    author    = {Wang, Yuhan and Chen, Xu and Zhu, Junwei and Chu, Wenqing and Tai, Ying and Wang, Chengjie and Li, Jilin and Wu, Yongjian and Huang, Feiyue and Ji, Rongrong},
    booktitle = {Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, {IJCAI-21}},
    publisher = {International Joint Conferences on Artificial Intelligence Organization},
    editor    = {Zhi-Hua Zhou},
    pages     = {1136--1142},
    year      = {2021},
    month     = {8},
    note      = {Main Track}
    doi       = {10.24963/ijcai.2021/157},
    url       = {https://doi.org/10.24963/ijcai.2021/157},
}

FaceForensics++

We have prepared 1000 fake videos of well-known forgery detection dataset FaceForensics++. We strictly follow the source and target pair settings of FF++. Besides, we also generated 10k frames of FF++ videos for quantitative test, which is widely adopted by recent face swapping research.

If you would like to access our FF++ videos, you can download them from either Google Drive or Baidu Netdisk.

Baidu Netdisk

Download all six splices from this link with extraction code nszb

Concat these splices to form a compressed file and unzip it:

cat FF++_HifiFace_* >FF++_HifiFace.tar.gz
tar -xzvf FF++_HifiFace.tar.gz

Google Drive

Download the compressed file from this link

Unzip this file:

tar -xzvf FF++_HifiFace.tar.gz

The md5sum of the compressed file is 5b596ac8025c25f69f24fd7783ec8133. It contains 1k manipulated videos and a config file. The config file indicates which frame of the original video is used as the identity's source image. The frame is numbered from zero.
If you encounter any problems or have anything else to ask, please contact us directly by email: hififace.youtu@gmail.com