New quantization method based on reordering RPTQ: realizing 3-bit quantization of large language models

Large language models (LLMs) perform well on a variety of tasks, but present deployment challenges due to their sheer model size.

In this paper, researchers from Houmo Intelligence, Tencent AI Lab, Huazhong University of Science and Technology, Peking University, and Illinois Institute of Technology found that the main challenge in quantifying large language models comes from Different activation ranges between channels, not just an outlier issue.

The author proposes a novel reranking-based quantization method RPTQ, which solves the problem of quantifying large language model activations. RPTQ reduces the impact of channel-wide differences by rearranging the channels in activation and then clustering them for quantification.

Additionally, the authors reduce storage and computational overhead by avoiding explicit reordering. This work is the first to push activation quantization of LLM models to 3 bits. The 3-bit quantization of activation values can also be used together with the 3-bit quantization of weights, which greatly reduces the overhead of large speech models.

##Paper address: https://arxiv.org/abs/2304.01089

Open source address: https://github.com/hahnyuan/RPTQ4LLM

Large-scale language models (LLMs) are widely used in various have demonstrated excellent performance in this task, but their deployment faces challenges due to their huge model size. For example, a 66B model contains 66 billion parameters and requires a lot of memory to store. In addition, as the sequence length and batch size increase, the memory consumption problem becomes more serious, because activation also consumes a large amount of memory, such as the key and value cache (Key/Value Cache) in Self-attention. However, a single GPU or server does not have enough memory capacity to store such a large number of model weights and intermediate activations. Therefore, LLM needs to be split into multiple chunks and stored in different devices. Since weights and activations are stored on different devices, data needs to be transferred between these devices during computation, resulting in significant bandwidth and energy consumption.

To overcome the model scaling challenges of LLMs, model quantification has emerged as a promising approach. This technique involves quantizing the weights and activations of LLMs using low-bit integers, thereby significantly reducing the cost of storing and running these models. Specifically, quantization helps reduce the memory requirements for holding tensors and accelerates computationally intensive operations such as matrix multiplications and linear layers. By quantizing weights and activations, storage and communication overhead are reduced, and efficiency and inference speed are improved.

Figure 1 Numerical distribution of different channels. Each point represents a (maximum, minimum) value in an activation channel.

Recently, GPTQ successfully quantized the weights of LLMs into 4 bit or 3 bit using PTQ. However, quantifying activation in LLMs remains a challenging task.

Two observations can be made from Figure 1:

1) Some channels have significant outliers with their maximum or minimum The value is hundreds of times larger than other channels. Previous studies have also discovered this problem and proposed special treatments for outliers.

2) Different channels have significant differences in numerical ranges. Using the same quantization parameters to quantize different channels can lead to significant quantization errors. Even if two channels have the same absolute value of outliers, they can have large variations in numerical range.

Per-tensor quantization techniques that quantize the entire tensor using the same quantization parameters may be less efficient. The reason is that setting the quantization range to cover a large range of values may result in large quantization errors for channels with smaller values, while setting it to cover a small range of values may result in significant truncation of outliers and result in Significant quantization error. For example, one channel might have a value range of -100 to -50, while another channel might have a value range of 80 to 100. Trying to cover their range by quantizing the range from - 100 to 100 will result in significant quantization errors for both channels. This is a challenge that has not been effectively addressed in previous research.

Figure 2 Comparison of different methods to solve the activation quantization problem.

To solve the problem of channel differences in activation, the authors propose a novel reordering-based post-training quantization method RPTQ.

As shown in Figure 2, this method clusters channels with similar numerical ranges together and then quantizes the values in each cluster using the same quantization parameters. To implement the reranking-based quantification method, the authors first used a calibration dataset as inference input, from which the maximum and minimum values of each activation channel were derived.

Next, they used the KMeans algorithm to divide different channels into g clusters, based on the points formed by the maximum and minimum values of each channel. Once clusters are established, they perform channel reordering to place channels from the same cluster in close proximity. This way channels with similar maxima and minima are grouped together and share a common set of quantization parameters. After the reordering process, the authors quantified activation within each cluster. This method calculates the quantization parameters (scaling factor s and zero point z) separately for each cluster, ensuring that these parameters are specific to the corresponding channel.

Ultimately, this method ensures that the quantization of each cluster is optimized, reducing quantization errors.

Figure 3: Failure diagram of the inference process of a quantized Transformer layer with rearranged weights and activations. Rearrangement indices are represented by the symbols R1 to R5.

Explicit rearrangement is a run-time operation of rearranging active channels, which requires physically moving data from different channels from one memory location to another, so For large models with a large number of channels, the rearrangement process can be very time-consuming. Additionally, storing source and target activation tensors increases memory overhead. Therefore, the authors propose strategies to avoid explicit reordering, thereby reducing computational overhead and improving inference efficiency.

As shown in Figure 3, the authors reorder the weights of the linear layers so that they produce activations directly in the sorted order. Additionally, the authors modified LayerNorm's method to directly produce reordered activations, eliminating the need for explicit channel adjustments during inference.

The authors evaluated the performance of OPT under three different bit-width configurations: W4A16, W4A8, and W4A4. In addition, the author also developed a new quantification scheme, W4A4KV, W4A3KV and W3A3KV, which only quantifies the main memory consumption - key cache and value cache (Key/Value Cache). Based on the table, the following observations can be made: In general, as the activation quantization bitwidth decreases, the performance of the model decreases. But in most cases, this degradation is not significant. For some tasks, performance dropped by less than 1% or even improved.

The memory overhead of LLM comes from three aspects: First, the storage of weights in device memory can be significantly reduced through quantization. Second, temporary activations generated during network execution require device memory to be allocated. These temporary activations can be released after use and therefore do not significantly impact overall memory usage. Finally, the Key/Value Cache is necessary to predict subsequent words. When the batch size and sequence length are large, the key and value cache takes up most of the memory.

The table shows the memory usage under different settings. It can be found that lower bit activation can significantly reduce memory usage, especially when the batch size and sequence length are large. situation. Specifically, when the batch size is 64 and the sequence length is 8192, the W4A4 configuration under OPT-66b saves more than 900GB of memory compared to the W4A16 configuration. Therefore, employing RPTQ to quantify activations can greatly reduce memory pressure in long text tasks or high-volume scenarios.

The above is the detailed content of New quantization method based on reordering RPTQ: realizing 3-bit quantization of large language models. For more information, please follow other related articles on the PHP Chinese website!