HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping


IJCAI, 2021.

Yuhan Wang1,2*      Xu Chen1,3*      Junwei Zhu1      Wenqing Chu1      Ying Tai1†     
Chengjie Wang1      Jilin Li1      Yongjian Wu1      Feiyue Huang1      Rongrong Ji3,4     

1Youtu Lab, Tencent       2Zhejiang University
3Media Analytics and Computing Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University
4Institute of Artificial Intelligence, Xiamen University
hififace.youtu@gmail.com


The face in the target image is replaced by the face in the source image. All results are generated by our end-to-end model HifiFace-512. If you want to see more celebrity demos, please refer to our supplementary material.


Introduction


In this work, we propose a high fidelity face swapping method, called HifiFace, which can well preserve the face shape of the source face and generate photo-realistic results. Unlike other existing face swapping works that only use face recognition model to keep the identity similarity, we propose 3D shape-aware identity to control the face shape with the geometric supervision from 3DMM and 3D face reconstruction method. Meanwhile, we introduce the Semantic Facial Fusion module to optimize the combination of encoder and decoder features and make adaptive blending, which makes the results more photo-realistic. Extensive experiments on wild faces demonstrate that our method can preserve better identity, especially on the face shape, and can generate more photo-realistic results than previous state-of-the-art methods.


Videos


1-min Presentation Video

Selected Video from FF++


Publication


Paper - ArXiv - pdf (abs)
If you find our work useful, please consider citing it:
@inproceedings{ijcai2021-157,
    title     = {HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping},
    author    = {Wang, Yuhan and Chen, Xu and Zhu, Junwei and Chu, Wenqing and Tai, Ying and Wang, Chengjie and Li, Jilin and Wu, Yongjian and Huang, Feiyue and Ji, Rongrong},
    booktitle = {Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, {IJCAI-21}},
    publisher = {International Joint Conferences on Artificial Intelligence Organization},
    editor    = {Zhi-Hua Zhou},
    pages     = {1136--1142},
    year      = {2021},
    month     = {8},
    note      = {Main Track}
    doi       = {10.24963/ijcai.2021/157},
    url       = {https://doi.org/10.24963/ijcai.2021/157},
}
                    

FaceForensics++


We have prepared 1000 fake videos of well-known forgery detection dataset FaceForensics++. We strictly follow the source and target pair settings of FF++. Besides, we also generated 10k frames of FF++ videos for quantitative test, which is widely adopted by recent face swapping research.

If you would like to access our FF++ videos, you can download them from either Google Drive or Baidu Netdisk.

Baidu Netdisk
Google Drive
The md5sum of the compressed file is 5b596ac8025c25f69f24fd7783ec8133. It contains 1k manipulated videos and a config file. The config file indicates which frame of the original video is used as the identity's source image. The frame is numbered from zero.
If you encounter any problems or have anything else to ask, please contact us directly by email: hififace.youtu@gmail.com