Single-lens 16-second 720p high-definition video is generated with one click, and the open source version of Sora has new surprises

The Luchen Open-Sora team has achieved breakthrough progress in 720p high-definition Vincent video quality and generation time, supporting Seamlessly produce high-quality short films of any style. Surprisingly, they have chosen to shock the open source community and continue to open source them all. Open source address: https://github.com/hpcaitech/Open-Sora

Through their model weights, various cool short films can be generated, such as The intimate contact between waves and conches, as well as the unfathomable secrets of the forest.

The rendering of character portraits is also quite realistic.

can also accurately render the cyberpunk style, making the short film instantly full of a strong sense of the future. and a sense of technology.

The image is compressed.

can also generate interesting and vivid animation shots, bringing a highly expressive visual experience.

##                                                                                                                                                                                                         to

^{Even film-level lens production can be easily handled.} For example, achieve smooth zoom effects and add professional-grade visual effects to videos.

The image is compressed.

^{also helps filmmakers create realistic movie shots.}

The image is compressed.

^{Luchen’s Open-Sora model reveals the broad prospects in the field of video generation with its excellent performance, and their model weights and training code have been fully open source. Interested friends can Visit their GitHub address:} https://github.com/hpcaitech/Open-Sora

Open-Sora Warriors in the Vincent Video Industry

LambdaLabs , a unicorn in the American technology industry, has created a digital Lego universe based on the weight of the Open-Sora model previously open sourced by the Luchen team, where Lego fans have found the ultimate creative experience.

The Luchen team is well aware of the acceleration of open source for Vincent Video’s technological breakthroughs. They not only continue to open source the model weights, but also publish the technical route on Github, allowing Every player can become the controller of Vincent Video's large model, and is no longer just a spectator. As soon as the report came out, we quickly conducted in-depth digging and analysis, trying to capture the latest trends in Vincent video technology. (Report address: https://github.com/hpcaitech/Open-Sora/blob/main/docs/report_03.md)

Technical depth decoding

In this technical report released by the Luchen OpenSora team, they deeply analyzed the core and key points of this model training. Based on the previous version, it introduced the Video Compression Network, a better diffusion model algorithm, more controllability, and used more data to train a 1.1B diffusion generation model.

In this era of "computing power is king", we are well aware of the two major pain points of video model training: the huge consumption of computing resources and the high standard of model output quality. The Luchen Open-Sora team successfully found a balance between cost and quality with a minimalist and effective solution.

The OpenSora team proposed an innovative video compression network (VAE) that performs compression in both space and time dimensions. Specifically, they first performed an 8x8x compression in the spatial dimension, followed by a further 4x compression in the temporal dimension. Through this innovative compression strategy, it not only avoids the disadvantage of sacrificing video smoothness due to frame extraction, but also significantly reduces training costs, achieving dual optimization of cost and quality.

##                                           Video compression network structure

Stable Diffusion 3, the latest diffusion model, uses rectified flow technology to replace DDPM, significantly Improved the quality of image and video generation. Although the rectified flow training code of SD3 has not yet been made public, the Luchen Open-Sora team has provided a complete training solution based on the research results of SD3, including:

• Simple and easy Rectified flow training

• Logit-norm time step sampling for training acceleration

• Based on resolution and video length Time step sampling

Through the integration of these technologies, we can not only speed up the training of the model, but also significantly reduce the waiting time in the inference stage and ensure the smoothness of the user experience. In addition, this training solution also supports the output of multiple video aspect ratios during the inference process, meeting the needs of video materials in diverse scenarios and providing video content creators with richer creative tools.

They also revealed more core details about model training in the report, including practical techniques for data cleaning and model tuning, as well as the construction of a more complete model evaluation system to ensure the robustness and versatility of the model. ization ability. They also provide a Gradio application that can be deployed with one click, and supports adjusting parameters such as output motion scores, aesthetic scores, and lens movement methods. It can even automatically modify instructions through GPT-4o with one click and supports Chinese input. If you can’t help but get your hands dirty, please click https://github.com/hpcaitech/Open-Sora/blob/main/docs/report_03.md

to break the closed loop and empower open source

Since the release of OpenAI Sora, the industry's expectations for Sora's openness have exploded, but the reality is that it is a continuous waiting game. The open source of Luchen Open-Sora has injected strong vitality into the innovation and development of Vincent Video. "Teaching a man to fish is worse than teaching him to fish." By visiting their github address, you can get the model weights and a full set of training codes for free with zero threshold, which enables users to transform from passive content consumers to active content creators. This transformation has unlocked new skills for enterprise users to independently develop Vincent Video applications. Whether it is creating immersive games, creative advertisements or producing film and television blockbusters, the application scenarios of Vincent Video technology have been expanded exponentially.

We hope that this spark can ignite the passion for innovation in the entire Vincent video field and achieve a prairie fire from point to surface.

Finally, attach the Luchen OpenSora open source link:

https://github.com/hpcaitech/Open-Sora

Reference link:

^{https://wandb.ai/lambdalabs/lego/reports/Text2Bricks-Fine-tuning-Open-Sora-in-1-000-GPU-Hours--Vmlldzo4MDE3MTky}

https://hpc-ai.com/blog/open-sora-from-hpc-ai-tech-team-continues-open-source-generate-any -16-second-720p-hd-video-with-one-click-model-weights-ready-to-use

The above is the detailed content of Single-lens 16-second 720p high-definition video is generated with one click, and the open source version of Sora has new surprises. For more information, please follow other related articles on the PHP Chinese website!