Generate a Marvel 3D digital human in five minutes! The American Spider-Man and the Joker can all do it, and the facial details are restored in high definition.

With the development of computer graphics, 3D generation technology is gradually becoming a research hotspot. However, there are still many challenges in generating 3D models from text or images.

Recently, companies such as Google, NVIDIA, and Microsoft have launched 3D generation methods based on neural radiation fields (NeRF), but these methods are compatible with traditional 3D rendering software (such as Unity, Unreal Engine, Maya, etc.) Sexual issues limit its wide application in practical applications.

To this end, the R&D team of Yingmo Technology and ShanghaiTech University proposed a text-guided progressive 3D generation framework designed to solve these problems.

Generate 3D assets based on text descriptions

The text-guided progressive 3D generation framework (DreamFace for short) proposed by the research team combines visual-language models, implicit diffusion models and physics-based Material diffusion technology generates 3D assets that comply with computer graphics production standards.

The innovation of this framework lies in its three modules: geometry generation, physics-based material diffusion generation and animation capability generation.

This work has been accepted by the top journal Transactions on Graphics and will be presented at SIGGRAPH 2023, the top international computer graphics conference.

Project website: https://sites.google.com/view/dreamface
Preprint paper: https://arxiv.org/abs/2304.03117
Web Demo: https://hyperhuman.top
HuggingFace Space: https ://huggingface.co/spaces/DEEMOSTECH/ChatAvatar

How to implement the three major functions of DreamFace

DreamFace mainly includes three modules, geometry generation and physics-based materials Diffusion and animation capabilities are generated. Compared with previous 3D generation work, the main contributions of this work include:

1. proposes DreamFace, a novel generation scheme that combines recent visual-language models with animatable and physically materialable faces Assets are combined to separate geometry, appearance and animation capabilities through progressive learning.
2. Introduces the design of dual-channel appearance generation, combining a novel material diffusion model with a pre-trained model, while performing two-stage optimization in latent space and image space.
3. Facial assets using BlendShapes or generated Personalized BlendShapes are animated and further demonstrate the use of DreamFace for natural character design.

Geometry generation: This module generates a geometric model based on text prompts through the CLIP (Contrastive Language-Image Pre-Training) selection framework.

First randomly sample candidates from the face geometric parameter space, and then select the rough geometric model with the highest matching score based on text prompts.

Next, implicit diffusion model (LDM) and Scored Distillation Sampling (SDS) processing are used to add facial details and detailed normal maps to the coarse geometry model to generate high-precision geometry.

Physically Based Material Diffusion Generation: This module targets predicted geometry and text Tips for generating facial textures. First, the LDM is fine-tuned to obtain two diffusion models.

The two models are then coordinated through a joint training scheme, one for directly denoising U-texture maps and the other for supervised rendering of images. Additionally, a hint learning strategy and non-face area masking are employed to ensure the quality of the generated diffuse maps.

Finally, apply the super-resolution module to generate 4K physically-based textures for high-quality rendering.

Animation capability generation: The model generated by DreamFace has animation capability. Different from traditional BlendShapes-based methods, this framework animates the Neutral model by predicting unique deformations to generate personalized animations.

The geometric generator is first trained to learn the expression latent space, and then the expression encoder is trained to extract expression features from RGB images. Finally, personalized animations are generated by using monocular RGB images.

Generate specified 3D assets in 5 minutes

The DreamFace framework has achieved good results in tasks such as celebrity generation and description generation characters, and has achieved results exceeding previous work in user evaluations.

At the same time, compared with existing methods, it has obvious advantages in running time.

In addition, DreamFace supports texture editing using tips and sketches to achieve global editing effects (such as aging, makeup) and local editing effects (such as tattoos) , beard, birthmark).

Can be used in film, television, games and other industries

As a text-guided progressive 3D generation framework, DreamFace combines visual -Language model, implicit diffusion model and physically based material diffusion technology achieve 3D generation with high precision, high efficiency and good compatibility.

This framework provides an effective solution for solving complex 3D generation tasks and is expected to promote more similar research and technology development.

In addition, physically based material diffusion generation and animation capability generation will promote the application of 3D generation technology in film and television production, game development and other related industries.

The above is the detailed content of Generate a Marvel 3D digital human in five minutes! The American Spider-Man and the Joker can all do it, and the facial details are restored in high definition.. For more information, please follow other related articles on the PHP Chinese website!