The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.-AI-php.cn

The AIxiv column is a column where this site publishes academic and technical content. In the past few years, the AIxiv column of this site has received more than 2,000 reports, covering top laboratories from major universities and companies around the world, effectively promoting academic exchanges and dissemination. If you have excellent work that you want to share, please feel free to contribute or contact us for reporting. Submission email: liyazhou@jiqizhixin.com; zhaoyunfeng@jiqizhixin.com

##The author of this article is the original team of VMamba, among whom the first The author Wang Zhaozhi is a 2022 doctoral student jointly trained by the University of Chinese Academy of Sciences and Pengcheng Laboratory, and the co-author Liu Yue is a 2021 direct doctoral student of the University of Chinese Academy of Sciences. Their main research directions are visual model design and self-supervised learning.

How to break through the Attention mechanism of Transformer? The University of Chinese Academy of Sciences and Pengcheng National Laboratory proposed a visual representation model vHeat

based on heat conduction. Treat image feature blocks as heat sources, and extract image features by predicting thermal conductivity and using physical heat conductionprinciples. Compared with visual models based on the Attention mechanism, vHeat takes into account: computational complexity (1.5th power), global receptive field, and physical interpretability. When using vHeat-base+%E6%A8%A1%E5%9E%8B for high-resolution image input, the put, GPU memory usage, and flops are Swin-base+%E6%A8%A1 respectively. 3 times, 1/4, 3/4 of %E5%9E%8B. It has achieved advanced performance on basic downstream tasks such as image classification, target detection, and semantic/instance segmentation.

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

Overview

The two most mainstream basic visual models currently are CNN and Visual Transformer (ViT). However, the performance of CNN is limited by local receptive fields and fixed convolution kernel operators. ViT has the ability to represent global dependencies, but at the cost of high quadratic norm computational complexity. We believe that the convolution operators and self-attention operators of CNN and ViT are both pixel transfer processes within features, which are respectively a form of information transfer, which also reminds us of heat conduction in the physical field. So based on the heat conduction equation, we connected the spatial propagation of visual semantics with physical heat conduction, proposed a visual conduction operator (Heat Conduction Operator, HCO) with 1.5 power computational complexity, and then designed a heat conduction operator with low Visual representation model vHeat for complexity, global receptive field, and physical interpretability. The calculation form and complexity comparison between HCO and self-attention are shown in the figure below. Experiments have proven that vHeat performs well in various visual tasks. For example, vHeat-T achieves 82.2% classification accuracy on ImageNet-1K, which is 0.9% higher than Swin-T and 1.7% higher than ViM-S. In addition to performance, vHeat also has the advantages of high inference speed, low GPU memory usage and low FLOPs. When the input image resolution is high, the base-scale vHeat model only has 1/3 more throughput, 1/4 GPU memory usage, and 3/4 FLOPs compared to Swin.

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

Method introduction

Use

to represent the temperature of point

at time t, and the physical heat conduction equation is The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field. , where k>0, represents thermal diffusivity. Given the initial conditions at time t=0, the heat conduction equation can be solved by using Fourier transform, which is expressed as follows: The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

where The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field. and represent Fourier transform and inverse Fourier transform respectively, represents frequency domain spatial coordinates.

We use HCO to implement heat conduction in visual semantics. First, we extend the The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field. in the physical heat conduction equation to a multi-channel feature , and treat as input and as output. , HCO simulates the general solution of heat conduction in the discretized form, as shown in the following formula:

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

where The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field. and represent the two-dimensional discrete cosine transform and the inverse transform respectively, The structure of HCO is shown in Figure (a) below.

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

In addition, we believe that different image contents should correspond to different thermal diffusivities. Considering that the output of The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field. is in the frequency domain, we determine the thermal diffusion based on the frequency value Rate,. Since different positions in the frequency domain represent different frequency values, we propose Frequency Value Embeddings (FVEs) to represent frequency value information, which is similar to the implementation and function of absolute position encoding in ViT, and use FVEs to control heat diffusion. The rate k is predicted so that HCO can perform non-uniform and adaptive conduction, as shown in the figure below.

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

vHeat is implemented using a multi-level structure, as shown in the figure below. The overall framework is similar to the mainstream visual model, and the HCO layer is shown in Figure 2 (b).

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

Experimental results

ImageNet classification

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

By comparing the experimental results, it is easy to see that under similar parameter amounts and FLOPs:

vHeat-T achieved 82.2 The performance of % exceeds DeiT-S by 2.4%, Vim-S by 1.7%, and Swin-T by 0.9%.
vHeat-S achieved 83.6% performance, exceeding Swin-S by 0.6% and ConvNeXt-S by 0.5%.
vHeat-B achieved 83.9% performance, exceeding DeiT-B by 2.1% and Swin-B by 0.4%.

At the same time, due to vHeat's O (N^1.5) low complexity and parallel computation, the inference throughput is compared to ViTs and SSM The model has obvious advantages. For example, the inference throughput of vHeat-T is 1514 img/s, which is 22% higher than Swin-T and 87% higher than Vim-S. , it is also 26% higher than ConvNeXt-T, and has better performance.

Downstream tasks

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

On the COCO data set, vHeat also has a performance advantage: in the case of fine-tune 12 epochs , vHeat-T/S/B reaches 45.1/46.8/47.7 mAP respectively, exceeding Swin-T/S/B by 2.4/2.0/0.8 mAP, and exceeding ConvNeXt-T/S/B by 0.9/1.4/0.7 mAP. On the ADE20K data set, vHeat-T/S/B reached 46.9/49.0/49.6 mIoU respectively, which still has better performance than Swin and ConvNeXt. These results verify that vHeat fully works in visual downstream experiments, demonstrating the potential to replace mainstream basic visual models.

Analysis Experiment

Effective Feeling Field

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

vHeat has the global effective feeling However, among the mainstream models for visual comparison, only DeiT and HiViT also have this feature. However, it is worth noting that the cost of DeiT and HiViT is square level complexity, while vHeat is 1.5 power level complexity.

Calculate the cost

The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.

The above picture from left to right is vHeat-B and Comparison of inference throughput/GPU memory usage/computation FLOPs of ViT-based models at other base scales. It can be clearly seen that due to the computational complexity of O (N^1.5), vHeat has faster inference speed, lower memory usage and fewer FLOPs than the contrasting models, and when the image resolution is larger, , the advantages will be more obvious. When the input image is 768*768 resolution, the inference throughput of vHeat-B is about 3 times that of Swin-B, and the GPU memory usage is 74% lower than that of Swin-B. FLOPs are 28% lower than Swin-B. Comparison of the computational cost of vHeat and ViT-based models demonstrates its excellent potential in processing high-resolution images.

The above is the detailed content of The visual representation model vHeat inspired by physical heat transfer is here. It attempts to break through the attention mechanism and has both low complexity and global receptive field.. For more information, please follow other related articles on the PHP Chinese website!