Let Lu Xun speak tongue twisters and Hepburn play hip-hop. Another video model went viral and was founded by a Chinese doctor from Stanford.

Team members are all from Stanford University, and the CTO is also a die-hard fan of Taylor Swift.

The AI ​​video field is going crazy.

The carnival caused by Luma is not over yet, there is another challenger in the AI ​​video circle——

Stanford Proteus produced by the university team.

According to reports, Proteus is a low-latency basic model that can generate highly realistic and expressive characters.

For example, let the protagonist in a world-famous painting - Mona Lisa or the girl with a pearl earring - laugh unbridled and have a natural and smooth facial expression: Let Audrey Hepburn change her past lady image and start hip-hop rap:

Also let Si in "Harry Potter" Professor Nepp sings "Despacito":

Proteus has just been released, and a number of big guys sent "congratulatory letters":

AI scientist Jia Yangqing praised that the quality of real-time artificial intelligence avatars is surprisingly good.

NVIDIA scientist Jim Fan said that this project is impressive.

Early investor Brian Zhan wrote that the biggest problem with existing AI video tools, such as Runway and Pika, is that they can produce hallucinations, especially when generating videos containing humans. hour. Apparate Labs takes AI video generation to the next stage by solving problems such as temporal coherence and object constancy.

Outrageous! Lu Xun talked about tongue twisters

Proteus is a new generation of basic model for real-time human expression generation.

#You must know that even the most advanced and powerful generation models currently cannot fully realize the real-time generation of human expressions.

Existing models are slow and fail to provide intuitive control over the complex facial expressions and body movements of generated characters, and they still suffer from realism and expressiveness What is lacking.

Proteus adopts the most advanced latent diffusion model of transformer architecture. Its innovative latent space design ensures high real-time efficiency, and with the development of architecture and algorithm With continuous optimization, Proteus is able to achieve video streaming of more than 100 frames per second (100+ FPS).

In other words, with just a simple photo, Proteus is not only able to imitate human laughter, rapping, singing, blinking, smiling and conversation, but it can also perform much more So many vivid expressions and movements.

For example, the always serious Lu Xun talked about tongue twisters:

Or let Madame Curie sing a cappella "Le Festin》: Or hold a roundtable meeting for scientists:

According to the Proteus R&D team, they It is expected that Proteus can become a voice-controllable visual avatar, provide an intuitive interactive interface for artificial intelligence dialogue entities, and be seamlessly compatible with many multi-modal large language models to provide customized services for various application scenarios.

In response to this, many netizens have opened their minds -

"Just use Einstein's By fine-tuning the large language model with data, coupled with his vivid facial expressions, the great Einstein can become a teaching assistant and teach physics classes in person, so teenagers no longer have to worry about failing to learn science."

Some netizens said, I love it so much. This year is definitely the year of AI video.

The team behind Qidian

##This model is praised by big guys, small and Beautiful model, what kind of team is behind it?

According to the official website, this was developed by Apparate Labs at Stanford University.

Currently there are only 6 people in the team. Judging from the names and photos, 3 of them are Chinese.

CEO and co-founder Dr. William Shen studied in the Department of Computer Science at Stanford University, co-supervised by well-known professors Silvio Savarese and Leonidas J. Guibas .

His research covers multiple fields of artificial intelligence, including computer vision, robotics, graphics, generative models and embodied intelligence. His papers have won many awards, such as winning the Best Paper Award at IEEE-CVPR and being a finalist for the Best Student Paper Award on RSS.

Previously, he also received a bachelor’s degree in computer science from Stanford University with an excellent GPA of 4.0.

Chief Technology Officer and Co-Founder Connor Lin is also a top student.

He studied at Carnegie Mellon University for his bachelor's and master's degrees, studying under Professor Keenan Crane. In 2020, he will go to Stanford University to pursue a PhD in computer science. He is currently a fourth-year doctoral student, co-supervised by professors Leonidas Guibas and Gordon Wetzstein.

Connor Lin’s research focuses on 3D prior knowledge and neural representation for 3D reconstruction, generation and editing. He was supported by the David Cheriton Stanford Graduate Scholarship.

During his PhD studies, he interned at Google Research, NVIDIA Research and Adobe Research. Previously, he worked as a software engineer at Google, responsible for the development of portrait mode for Pixel phones.

In addition, this guy has a wide range of interests. He likes travel and sports, cooking, badminton, swimming, board games and music. He is also a die-hard fan of Taylor Swift

Like Connor Lin, chief scientist Linqi (Alex) Zhou is also a doctoral student at Stanford University, supervised by Professor Stefano Ermon.

Previously, Linqi Zhou received a bachelor's degree in computer science and applied mathematics from the University of California, Los Angeles, under the supervision of Professors Song-Chun Zhu and Ying-Nian Wu.

He conducts research mainly in the fields of computer vision and machine learning, and is committed to building models that can understand the world in a structured and probabilistic way.

Reference link:

##https://apparate.ai/ stream.html

The above is the detailed content of Let Lu Xun speak tongue twisters and Hepburn play hip-hop. Another video model went viral and was founded by a Chinese doctor from Stanford.. For more information, please follow other related articles on the PHP Chinese website!