China's Yongle Palace murals are as large as 1,000 square meters. Can they be restored by AI?

Today, artificial intelligence (AI) is not only increasingly used in the fields of science and business, but also begins to emerge in the field of art.

While people are amazed by the endless AI technologies, a new AI mural restoration technology has emerged.

Recently, in order to solve the problem of "difficulty in repairing" the Yongle Palace murals, a masterpiece of ancient Chinese murals, A research team from Shanxi Datong University, University of Science of Malaysia, and Dali University proposed an AI model that can repair giant murals—— 3M-Hybrid.

According to reports, this model improved the structural similarity index (SSIM) by 14.61% compared with the best model among four representative convolutional neural network (CNN) models in regular-sized mural restoration. and a peak signal-to-noise ratio (PSNR) of 4.73%. In addition, it also showed good results in the final restoration of large murals

A related research paper titled "3M Hybrid Model for the Restoration of Unique Giant Murals of Yongle Palace Murals: A Case Study" has been published on the preprint website arXiv

What is the repair method for large-scale murals?

The Yongle Palace murals are located in the Yongle Palace (also known as the Dachunyang Wanshou Palace) in Ruicheng, Shanxi Province. The most artistic value of the murals is the exquisite large-scale murals. The entire mural area exceeds 1,000 square meters and is painted in Wuji Hall, Sanqing Hall, Chunyang Hall and Chongyang Hall

As a precious cultural heritage, the Yongle Palace murals represent artistic masterpieces in the history of Chinese painting. However, due to a long-term lack of protection, these unique murals have suffered a lot of damage, making their restoration an urgent task

Compared with manual repair techniques, digital repair methods are more efficient and reversible. In particular, image repair technology based on deep learning has achieved remarkable results. However, the literature on mural restoration based on deep learning mainly focuses on Dunhuang murals or other regular-sized murals, and there is a lack of research specifically on the restoration of Yongle Palace murals and similar huge murals.

Compared with other studies on mural restoration, the restoration work of the huge Yongle Palace murals faces two main challenges: 1) The scarcity and unique style of the Yongle Palace murals; 2) The size of the murals Huge, models have limited proficiency in repairing defects of different types and sizes.

Picture | The types and scales of the incomplete murals in Yongle Palace vary, and the actual manifestations of the incompleteness are also more diverse.

According to the paper, the 3M-Hybrid model proposed in this study can effectively repair the Yongle Palace murals.

Among them, "3M" refers to the three key strategies: multi-frequency, multi-angle and multi-scale, while "Hybrid" refers to the hybrid CNN-VIT network.

First, the research team divided the huge mural into regular-sized parts for restoration, and then reassembled the restored parts back to their original size. In order to enable the regular-size mural repair model to effectively handle various types and sizes of defects with a limited amount of image data, the research considered two aspects: optimizing training data and improving the model structure.

In order to obtain better repair results, this research adopts an independent network specifically designed to learn and extract high-frequency and low-frequency signals to enhance feature learning and repair capabilities in these specific frequency ranges. Through a frequency-based training method, the model is able to effectively handle defects of different scales and types

In terms of model structure, this research integrates a convolutional neural network (CNN) with a pre-trained visual transformer (VIT) to enhance the feature extraction capability of the model.

While restoring huge murals, we discovered that basic cutting methods resulted in seam gaps and structural distortion when repairing oversized defects. To solve this problem, our research team adopted a multi-angle strategy to reduce the gap and a multi-scale repair method that combined cutting and shrinking methods. In this way, we can ensure the accuracy of the repair, and at the same time extract the overall structure of the mural, solving the problem of multi-scale defects

From the perspective of visual performance, the model shows considerable results in the repair results of free dust-like, free gel-like and free linear masks. Additionally, repair results on free-form block masks demonstrate preserved structural integrity and believable texture. The 3M-Hybrid model proved to be a viable method of restoring these unique and monumental murals.

However, this study is not without shortcomings

First, the method proposed in the study relies on multiple experiments to select the best value of the three scale fusion weights. However, this approach may not be precise enough given that the weight settings cover countless possibilities and the number of experiments is limited. Therefore, weight values ​​determined based on experimental results can only guarantee a relatively favorable final result.

Secondly, the rewriting that needs to be made is: Secondly, the evaluation indicators used in the study are not objective enough. The four currently used evaluation indicators fail to comprehensively evaluate the structure of the image and usually cannot accurately reflect human perception and evaluation of the image

However, we cannot deny the importance of this study, which explores the application of deep learning in the restoration of huge murals, with a special focus on the use of deep learning technology to restore the Yongle Palace murals. This study represents the first attempt to explore deep learning restoration methods for large-scale artworks

In terms of improving the conventional size image repair model, this research comprehensively improved it from two perspectives: data and structure. This provides future research with new insights into repairing unique small data sets

Helping mankind protect and inherit the value of cultural relics

In the past few years, people have witnessed the amazing integration of artificial intelligence technology and cultural relics and history

In 2020, a Weibo netizen named "大谷Spitzer" used AI technology to restore the 1920 black and white image data of Beijing released by the People's Daily 4 years ago, completing coloring, frame rate restoration, Expand the resolution and so on.

In 2021, Tencent Multimedia Laboratory also cooperated with the Dunhuang Research Institute to use deep learning methods to analyze Dunhuang mural disease data, and developed an efficient AI mural disease identification tool. It also provided immersive remote consultation technology, using 4K ultra- The high-definition 360-degree picture shows the scene inside the cave and the details of cultural relics, enabling barrier-free remote cultural relic consultation.

In June this year, at the Heritage Day home city event held in Chengdu, the theme forum of "Cultural Relics Protection and Utilization and Cultural Confidence and Self-Strengthening", Tencent demonstrated the Sanxingdui human-machine collaborative simulation splicing effect using AI technology.

The many applications of AI technology in the field of cultural relic restoration are exciting. In the future, we expect AI technology to go further in the protection and restoration of cultural relics, helping humans protect and inherit the value of cultural relics.

The above is the detailed content of China's Yongle Palace murals are as large as 1,000 square meters. Can they be restored by AI?. For more information, please follow other related articles on the PHP Chinese website!