Automatically convert images into text, and image descriptions are of higher quality and more accurate.

The AIxiv column is a column where this site publishes academic and technical content. In the past few years, the AIxiv column of this site has received more than 2,000 reports, covering top laboratories from major universities and companies around the world, effectively promoting academic exchanges and dissemination. If you have excellent work that you want to share, please feel free to contribute or contact us for reporting. Submission email: liyazhou@jiqizhixin.com; zhaoyunfeng@jiqizhixin.com

Pi Renjie: a third-year PhD student at Hong Kong University of Science and Technology, studying under Professor Zhang Tong and Professor Zhou Xiaofang. Received Apple Scholarship in 2024. The current main research directions are multi-modal large language models and data-centered AI.

Zhang Jianshu: A third-year undergraduate student at Wuhan University. Currently working as a research intern under the guidance of Professor Zhang Tong. His main research directions are large language models, multi-modal large language models and continuous learning. Currently looking for PhD admission opportunities for 2025 fall.

In the development of today's multi-modal large models, the performance of the model is closely related to the quality of the training data. It can be said that "the data gives the model most of its capabilities."

In this, image-text datasets play a vital role in many fields such as image understanding, text generation and image retrieval.

However, existing image description data sets are mainly derived from network crawling and manual annotation, and there are problems such as uneven quality, lack of details, and high description noise. Although humans can provide detailed descriptions for images, the high annotation cost limits its scale and feasibility. Therefore, there is an urgent need for an efficient and scalable method to generate accurate and detailed image descriptions.

In order to address the above challenges, researchers from Hong Kong University of Science and Technology, Wuhan University, Zhejiang University, and UIUC jointly proposed an innovative automation framework - Image-Textualization (IT), which integrates multi-modal large language models (MLLMs) and a variety of visual expert models collaborate to textualize image information, and finally use a pure text large language model with powerful reasoning capabilities to transform this textualized information into high-quality image descriptions.

• Paper: Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions

• Paper address: https://arxiv.org/pdf/2406.07502v1

• Project address: https: //github.com/sterzhang/image-textualization/

• uses a picture made by multi-modal models to describe vs using IT pictures to describe. Generally speaking, the contribution of this article includes:

• Innovative framework: We propose an image textualization framework, which uses the coarse-grained image understanding capabilities of multi-modal large models, the fine-grained perception capabilities of visual expert models, and the reasoning capabilities of plain text large language models to automatically generate Image descriptions that are rich in detail and clearly articulated.

• Evaluation benchmarks and experiments: Multiple benchmarks for evaluating detailed image descriptions are proposed, and the effectiveness of the framework is verified through extensive experiments.

Dataset and code release: Leveraging our image textualization framework, we generated a large-scale, high-quality image description dataset (IT-170K). To facilitate future research, we have made all source code and generated datasets publicly available.

1. Coarse-grained picture textualization (Holistic Textualization): First, use a multi-modal large language model to generate reference descriptions for pictures. Although these descriptions may have missing details and illusions, they represent the visual information and language expression of the image. The basic structure is provided. The visual structure here is mainly reflected in the fact that reference descriptions often contain some large, core objects, which can provide an "anchor" effect for subsequent details, making the final textualized recaptioning better. of added details. In addition, the structure of language expression is mainly reflected in the large plain text language model included in the multi-modal large model, which makes it have strong language capabilities. This allows the reference description generated here to be well organized in language. For example, it will be first Tell what the picture roughly describes, then expand on the details, and finally summarize. This description style is more biased towards human preference. This also enables the final textualized recaptioning to be processed on a template with better language capabilities.

2. Visual Detail Textualization: At this stage, we extract details from the picture side and the text side at the same time.

The first is the text side. Since the reference description we generated using the multi-modal large model in the previous stage may contain hallucinations, the first thing we do here is "hallucination detection". We first use LLM to capture the entity contained in the reference description, and then use an open set detector to match the entity in the picture. If it is not detected, the entity is judged as an illusion. Here we also textualize the detected hallucinations and delete them in the final textualized recaptioning.

On the visual side, we use visual expert models on various tasks trained on high-resolution images to extract detailed information in the image. If you want to express the detailed information of an object in a picture using text, it is not enough to just use the object's caption. We first use the bounding box of these objects to extract the left-right relationship of these objects in the form of text. But the objects in the picture not only have left and right information, but also front and back information. In this regard, we first use the segmentation model to extract the masks of these objects, then convert the original pictures into depth maps, and reflect the depth information in the text by calculating the depth scores corresponding to the masks of specific objects in the depth map. At this point, we can use text to restore detailed information such as the size, left and right positions, and context of each object in the picture.

3. Textualized Recaptioning: Combining the textualization results of the image information in the first two stages, plus our carefully designed rewritten prompt, the large language model of plain text can be very good It restores image information through pure text and generates detailed and accurate image descriptions through powerful understanding and reasoning capabilities.

Comprehensive evaluation and experimental verification

To verify the effectiveness of our framework, we constructed three evaluation benchmarks, namely DID-Bench (Detailed Image Description Benchmark), D2I-Bench (Description-to-Image Benchmark) and LIN -Bench (Language Benchmark). We conduct extensive experiments and show that the image descriptions generated by the IT framework significantly outperform existing methods in terms of detail richness and accuracy. In particular, MLLMs trained on datasets generated by our IT framework, such as LLaVA-7B, exhibit stronger image description capabilities and reduced hallucination phenomena.

DID-Bench (Detailed Image Description Benchmark): used to evaluate the similarity between image descriptions and human manually labeled detailed image descriptions. It can be seen that our modified IT-{LLaVA} and IT-{GPT4-V} image descriptions are more detailed and accurate than before the modification, and are more consistent with the descriptions marked by humans.

D2I-Bench (Description to Image Benchmark): Use the Vincentian graph model to convert the generated description into a picture, and compare the similarity with the original image. Here we selected CLIP-score and DINO-score for evaluation. can achieve higher scores.

In addition, we also verified on POPE and LIN-Bench that LLaVA-7B, which is trained using data generated by our framework, can generate more detailed and complex descriptions (LIN-Bench on the right side of the table) , and can also reduce hallucinations (POPE benchmark on the left side of the table).

Finally, we statistically compared the generated data, and we can see that the number of each part of speech in our modified description has been greatly improved.

Future Outlook

Our work not only addresses the limitations of existing image description datasets, but also provides inspiration for designing more efficient and scalable methods. We look forward to the IT framework demonstrating its potential in more application areas and promoting the further development of image understanding and generation technology.

The above is the detailed content of Automatically convert images into text, and image descriptions are of higher quality and more accurate.. For more information, please follow other related articles on the PHP Chinese website!