Without downstream training, Tip-Adapter greatly improves CLIP image classification accuracy

• Paper link: https://arxiv.org/pdf/2207.09519.pdf
• Code link: https://github.com/gaopengcuhk/Tip-Adapter

1. Research Background

The Contrastive Image Language Pre-training Model (CLIP) has recently demonstrated strong visual domain transfer capabilities and can perform zero-shot image recognition on a new downstream data set. In order to further improve the migration performance of CLIP, existing methods use few-shot settings, such as CoOp and CLIP-Adapter, which provide a small amount of training data for downstream data sets, allowing CLIP to better make decisions for different visual scenarios. Adjustment. However, this additional training step will bring considerable time and space resource overhead, which to a certain extent affects the inherent rapid knowledge transfer capability of CLIP. Therefore, we propose Tip-Adapter, a few-shot image classification method that does not require additional downstream training and can greatly improve the accuracy of CLIP. Based on this, we have proposed a solution that can achieve state-of-the-art performance with only a small amount of fine-tuning: Tip-Adapter-F, achieving the best compromise between efficiency and performance. As shown in Table 1 below, Tip-Adapter does not require any training time, that is, it can improve CLIP's accuracy in the ImageNet data set by 1.7% (Accuracy), while Tip-Adapter-F only requires one-tenth of the training time of the previous solution ( Epochs, Time), you can achieve the best existing classification performance.

Table 1: Comparison of 16-shot image classification accuracy and training time of different schemes on the ImageNet dataset

two. Research method

1.Tip-Adapter

The overall network structure of Tip-Adapter is shown in Figure 1 below. For the given few-shot training data set and labels, we use CLIP uses a non-training solution to build a cache model (Cache Model), which stores classification knowledge from downstream training data; during testing, Tip-Adapter performs a linear summation of the predictions of the Cache Model and the predictions of the original CLIP , to obtain stronger final classification results.

In detail, we use CLIP pre-trained visual encoder (Visual Encoder) to extract the features of all images in the few-shot training set as the Keys of the Cache Model; and Convert the corresponding image tags into one-hot encoding form as the Values ​​of the Cache Model. This Key-Value Cache Model construction method does not require any training overhead because it uses the pre-trained Visual Encoder; and considering that the few-shot training set contains only a small number of images for each category (1 to 16 shots), the Cache Model also takes up almost no additional graphics memory overhead. Refer to the GPU Mem. indicators in Table 1.

For a test image, we will first use CLIP's Visual Encoder to get its features, and then treat the features as Query to Cache Model for downstream few-shot data Knowledge retrieval. Since Keys are also extracted by CLIP's Visual Encoder, they have the same origin as the test image feature Query. We can directly calculate the cosine similarity between them to obtain a Key-Query adjacency matrix. This matrix can be regarded as each The weight corresponding to Value. Therefore, we can calculate the weighted sum of Values ​​to get the classification prediction for this test image obtained by retrieving the Cache Model. In addition, we can also get CLIP's zero-shot prediction by matching the test image features with CLIP's Textual Encoder text features. By linearly weighting the sum of the two, we obtain the final classification prediction, which contains both the image language contrastive knowledge pre-trained by CLIP and the few-shot knowledge of the new downstream data set, so it can achieve more accurate predictions. Strong image classification accuracy.

Based on the network structure of Tip-Adapter, we can further change the Keys part in the Cache Model into learning parameters, that is, they can be updated through training. This solution is Tip-Adapter- F. With the help of the already built Cache Model, Tip-Adapter-F only requires one-tenth of the training rounds and time of the existing CLIP-Adapter to achieve higher performance, as shown in Table 1.

Figure 1: Network flow chart of Tip-Adapter and Tip-Adapter-F

2.Differences and connections between Tip-Adapter and existing solutions​

Compare CLIP-Adapter, as shown in Figure 2, Tip-Adapter stores Keys and Values In fact, they can respectively correspond to the two linear layers of the adapter structure in CLIP-Adapter, except that the former does not require training to build, while the latter is randomly initialized and then requires training to learn the best parameters.

Figure 2: Tip-Adapter compared to CLIP-Adapter

Compared with other existing solutions for building Cache Model, as shown in Figure 3, the Cache Model of Tip-Adapter can be regarded as a multi-modal visual language Cache. Because the features output by CLIP's Textual Encoder can be regarded as the Key-Value of the text, which is equivalent to testing the image features as Query, retrieving knowledge in the visual and text Cache respectively. Compared with the existing solution that only contains the visual Cache, Tip-Adapter can utilize multi-modal knowledge to obtain stronger recognition performance.

Figure 3: Tip-Adapter compared to other solutions for building Cache Model

three. Experimental results

1. Classification accuracy in ImageNet

Figure 4 and Table 2 compare Tip-Adapter, Tip-Adapter-F and existing solutions in 1 and 2 , 4, 8, and 16 shots of few-shot image classification accuracy; Table 3 compares the accuracy of Visual Encoder using different CLIP on the 16-shot ImageNet dataset. It can be seen that both of our solutions achieve excellent performance with very little resource overhead.

Figure 4 and Table 2: 1~16- of different methods on the ImageNet dataset Shot image classification accuracy comparison

##Table 5: 16-shot ImageNet Comparison of image classification accuracy of Visual Encoder with different CLIP

As shown in Figure 5, we provide The accuracy comparison results of another 10 image classification data sets are obtained, namely StandfordCars, UCF101, Caltech101, Flowers102, SUN397, DTD, EuroSAT, FGVCAircraft, OxfordPets and Food101. As shown in the figure, our Tip-Adapter-F all achieved the highest recognition accuracy.

Figure 5: 1~16-shot of different methods on another 10 data sets Image classification accuracy comparison

3. Evaluation of Domain Generalization Ability

We also tested the performance of Tip-Adapter and Tip-Adapter-F in Domain Generalization. As shown in Table 6, both of our schemes exhibit strong robustness and feature transfer capabilities.

Four. Conclusion

This paper proposes Tip-Adapter, a training-free solution for using CLIP for downstream few-shot image classification. Tip-Adapter builds a Key-Value Cache Model as a knowledge retrieval database for the test image Query, and obtains stronger recognition performance by fusing the predictions of the Cache Model and the zero-shot predictions of CLIP. We hope that Tip-Adapter can inspire more follow-up work on efficient migration of pre-trained models.

The above is the detailed content of Without downstream training, Tip-Adapter greatly improves CLIP image classification accuracy. For more information, please follow other related articles on the PHP Chinese website!