Kuaishou sumber terbuka LivePortrait, GitHub 6.6K Star, untuk mencapai penghijrahan ekspresi dan postur yang sangat pantas

• Code address: https://github.com/KwaiVGI/LivePortrait
• Paper link: https://arxiv.org/abs /2407.03168
• Project homepage: https://liveportrait.github.io/
• HuggingFace Space one-click online experience: https://huggingface.co/spaces/KwaiVGI/LivePortrait

Same identity driver: From the above same identity driver comparison results, it can be seen that compared with the existing non-diffusion model method and the diffusion model-based method, LivePortrait With better generation quality and driving accuracy, it can capture the subtle expressions of the eyes and mouth driving frames while preserving the texture and identity of the reference image. Even in larger head postures, LivePortrait has a more stable performance.

Cross-identity driver: As can be seen from the above cross-identity driver comparison results, compared with existing methods, LivePortrait can accurately inherit the subtle eye and mouth movements in the driver video, and is also relatively stable when the posture is large. . LivePortrait is slightly weaker than the diffusion model-based method AniPortrait [11] in terms of generation quality, but compared with the latter, LivePortrait has extremely fast inference efficiency and requires fewer FLOPs.

Expansion

Multi-person driver: Thanks to LivePortrait’s fitting module, for group photos, LivePortrait can drive designated faces with designated driver videos, thus achieving Multi-person photo driver broadens the practical application of LivePortrait.

Animal driven: LivePortrait not only has good generalization for portraits, but also can be accurately driven for animal portraits after fine-tuning on animal data sets.

Portrait video editing: In addition to portrait photos, given a portrait video, such as a dance video, LivePortrait can use the driving video to perform motion editing on the head area. Thanks to the fitting module, LivePortrait can accurately edit movements in the head area, such as expressions, postures, etc., without affecting the images in non-head areas.

Implementation and prospects

LivePortrait’s related technical points have been implemented in many of Kuaishou’s businesses, including Kuaishou magic watch, Kuaishou private messaging, Kuaiying’s AI emoticon gameplay, and Kuaishou live broadcast , and the Puchi APP for young people incubated by Kuaishou, etc., and will explore new implementation methods to continue to create value for users. In addition, LivePortrait will further explore multi-modal driven portrait video generation based on the Keling basic model, pursuing higher-quality effects.

^References

^{[1] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In CVPR, 2021.}

^{[2] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: a large-scale speaker identification dataset. In Interspeech, 2017.}

^{[3] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-face generation. In ECCV, 2020.}

^{[4] Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. In PloS one, 2018}

^{[5] Mingcong Liu, Qiang Li, Zekui Qin, Guoxin Zhang, Pengfei Wan, and Wen Zheng. Blendgan: Implicitly gan blending for arbitrary stylized face generation. In NeurIPS, 2021.}

^{[ 6] Haotian Yang, Mingwu Zheng, Wanquan Feng, Haibin Huang, Yu-Kun Lai, Pengfei Wan, Zhongyuan Wang, and Chongyang Ma. Towards practical capture of high-fidelity relightable avatars. In SIGGRAPH Asia, 2023.}

^{[7] Kai Zhao, Kun Yuan, Ming Sun, Mading Li, and Xing Wen. Quality-aware pre-trained models for blind image quality}

^{assessment. In CVPR, 2023.}

^{[8] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con-}

^{vnext v2: Co-designing and scaling convnets with masked autoencoders. In CVPR, 2023.}

^{[9] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2019.}

^{[10] Wenzhe Shi, Jose Caballero, Ferenc Husz ´ar, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network . In CVPR, 2016.}

^{[11] Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint:2403.17694, 2024.}