The rephrased title is: ByteDance's cooperation with East China Normal University: Exploring the contextual learning capabilities of small models

It is well known that large language models (LLM) can learn from a small number of examples through contextual learning without the need for model fine-tuning. Currently, this contextual learning phenomenon can only be observed in large models. For example, large models such as GPT-4 and Llama have shown excellent performance in many fields, but due to resource constraints or high real-time requirements, large models cannot be used in many scenarios

So, do regular-sized models have this capability? In order to explore the contextual learning capabilities of small models, research teams from Byte and East China Normal University conducted research on scene text recognition tasks.

Currently, in practical application scenarios, scene text recognition faces a variety of challenges: different scenes, text layout, deformation, lighting changes, blurred writing, font diversity, etc. Therefore, It is difficult to train a unified text recognition model that can handle all scenarios.

A direct way to solve this problem is to collect corresponding data and fine-tune the model in specific scenarios. However, this process requires retraining the model, which is computationally intensive, and requires saving multiple model weights to adapt to different scenarios. If the text recognition model can have context learning capabilities, when faced with new scenarios, it only needs a small amount of annotated data as prompts to improve its performance on new scenarios, thus solving the above problems. However, scene text recognition is a resource-sensitive task, and using a large model as a text recognizer will consume a lot of resources. Through preliminary experimental observations, researchers found that traditional large model training methods are not suitable for scene text recognition tasks

In order to solve this problem, research from ByteDance and East China Normal University The team proposed a self-evolving text recognizer, E2STR (Ego-Evolving Scene Text Recognizer). This is a regular-sized text recognizer that incorporates contextual learning capabilities and can quickly adapt to different text recognition scenarios without the need for fine-tuning

paper Link: https://arxiv.org/pdf/2311.13120.pdf

E2STR is equipped with a contextual training and contextual reasoning mode, which not only reaches the SOTA level on conventional data sets , and a single model can be used to improve the recognition performance in various scenarios and achieve rapid adaptation to new scenarios, even exceeding the recognition performance of a dedicated model after fine-tuning. E2STR demonstrates that regular-sized models are sufficient to achieve effective context learning capabilities in text recognition tasks.

Method

In Figure 1, the training and inference process of E2STR is shown

1. Basic text recognition training

#The basic text recognition training phase uses an autoregressive framework to train the visual encoder and language decoder , the purpose is to obtain text recognition capabilities:

2. Context training

Context training phase E2STR will be further trained according to the context training paradigm proposed in the article. At this stage, E2STR will learn to understand the connections between different samples to gain reasoning capabilities from contextual cues.

As shown in Figure 2, this article proposes the ST strategy to randomly segment and transform the scene text data to generate a set of "subsample". The subsamples are intrinsically linked both visually and linguistically. These intrinsically related samples are spliced into a sequence, and the model learns contextual knowledge from these semantically rich sequences, thereby gaining the ability to learn context. This stage also uses the autoregressive framework for training:

The content that needs to be rewritten is: 3. Contextual reasoning Rewritten content: 3. Reasoning based on context

#For a test sample, the framework will select N samples from the contextual cue pool, which are in the visual latent space Has the highest similarity with the test sample. Specifically, this article calculates image embedding I by averaging pooling on the visual token sequence. Then, the top N samples with the highest cosine similarity between image embeddings and I are selected from the context pool, thus forming contextual cues.

After the contextual cues and test samples are spliced together and fed into the model, E2STR will learn new knowledge from the contextual cues without training. , thereby improving the recognition accuracy of test samples. It is important to note that the contextual cue pool only retains tokens output by the visual encoder, making the contextual cue selection process very efficient. Furthermore, since the context hint pool is small and E2STR requires no training for inference, additional computational overhead is also minimized

Experiment

The experiment is mainly conducted from three aspects: traditional text recognition set, cross-domain scene recognition and difficult sample correction

1. Traditional data set

Randomly select a few samples (1000, 0.025% of the number of samples in the training set) from the training set to form a context prompt pool, and test it in 12 common scene text recognition test sets , the results are as follows:

##It can be found that E2STR still improves on traditional data sets whose recognition performance is almost saturated, surpassing the performance of the SOTA model .

The content that needs to be rewritten is: 2. Cross-domain scenario

Each test set in the cross-domain scenario Only 100 in-domain training samples are provided. The comparison results between no training and fine-tuning are as follows. E2STR even exceeds the fine-tuning results of the SOTA method.

The content that needs to be rewritten is: 3. Modify difficult samples

The researchers collected a batch of difficult samples and provided 10% to 20% annotations for these samples. They compared E2STR's context learning method without training and the SOTA method's fine-tuning learning method. The results are as follows:

Compared with fine-tuning methods, E2STR-ICL significantly reduces the error rate of difficult samples

Future Outlook

E2STR proves that using appropriate training and inference strategies, small models can also have in-context learning capabilities similar to LLM. In some tasks with strong real-time requirements, small models can also be used to quickly adapt to new scenarios. More importantly, this method of using a single model to achieve rapid adaptation to new scenarios brings one step closer to building a unified and efficient small model.

The above is the detailed content of The rephrased title is: ByteDance's cooperation with East China Normal University: Exploring the contextual learning capabilities of small models. For more information, please follow other related articles on the PHP Chinese website!