Huawei releases two new AI storage products

The conference site. (Photo provided by Huawei)

[Shenzhen Business News] (Reporter Chen Shu) Artificial intelligence is gradually moving from large models to very large models, from single modality to multi-modality, and data storage has become a key element. The era of large models centered on storage and computing power has arrived. On July 14, Huawei released a new AI storage product for the large model era, providing storage "optimal solutions" for basic model training, industry model training, and segmented scenario model training and reasoning.

Zhou Yuefeng, President of Huawei's data storage product line, said that enterprises face four major challenges in the process of developing and implementing large model applications: First, data preparation time is long, data sources are scattered, and collection is slow. It takes 10 years to preprocess 100 TB of data. About days; second, multi-modal large models use massive texts and pictures as training sets. The current loading speed of massive small files is less than 100MB/s, and the training set loading efficiency is low; third, large model parameters are frequently tuned, and the training platform Unstable. Training is interrupted once every 2 days on average. The Checkpoint mechanism is required to resume training. Failure recovery takes more than a day. Finally, the implementation threshold for large models is high, the system is complex to set up, and resource scheduling is difficult. The GPU resource utilization rate is usually less than 40%. .

Huawei has launched OceanStor A310 deep learning data lake storage and FusionCube A3000 training/promotion hyper-converged all-in-one machine for large model applications in different industries and scenarios. Among them, OceanStor A310 deep learning data lake storage is oriented to basic/industry large model data lake scenarios, realizing full-process massive data management of AI from data collection and preprocessing to model training and inference application. FusionCube A3000 training/push hyper-converged all-in-one machine is designed for industry large model training/inference scenarios and for tens of billions of model applications. It integrates OceanStor A300 high-performance storage nodes, training/push nodes, switching equipment, AI platform software, and management and operation software. , providing large model partners with a turn-key deployment experience and achieving one-stop delivery.

In an exclusive interview with the media, Ni Guangnan, an academician of the Chinese Academy of Engineering, said that data has become the country’s basic strategic resource. Data storage capacity (referred to as "storage capacity"), information computing capacity (referred to as "computing power"), and network transport capacity (referred to as "transport capacity") are the core and foundation of the development of my country's information industry, and are the strategic support for building a technologically powerful country. He believes that energy storage will become a national strategic and basic industry and a new international competitive advantage.

"In the era of large models, data determines the height of AI intelligence. As a carrier of data, data storage has become the key infrastructure of AI large models." Zhou Yuefeng said in an interview after the meeting that China's artificial intelligence industry must develop rapidly. We must pay attention to digitization and the digital recording of data and information. Data preparation is the biggest challenge encountered when implementing large AI models that have caused a stir recently. According to him, the cost of large AI models is mainly accounted for 25% by computing power, while the cost of purchasing servers, data cleaning and pre-processing accounts for 22%. It can be seen that data and data storage and processing are becoming more and more important. This sentence is rewritten as follows: This important point is not only that the amount of data has increased, but more importantly, the data processing process has become more complex. Han Zhenxing, vice president of Huawei's distributed storage field, pointed out that China will usher in the large-scale development of storage centers and predicted that higher-performance storage products will emerge in the future.

The above is the detailed content of Huawei releases two new AI storage products. For more information, please follow other related articles on the PHP Chinese website!