Modularly reconstruct LLaVA. To replace components, just add 1-2 files. The open source TinyLLaVA Factory is here.

The TinyLLaVA+ project is jointly created by Professor Wu Ji’s team from the Multimedia Signal and Intelligent Information Processing Laboratory (MSIIP) of the Department of Electronics of Tsinghua University and Professor Huang Lei’s team from the School of Artificial Intelligence of Beihang University. The MSIIP Laboratory of Tsinghua University has long been committed to research fields such as intelligent medical care, natural language processing and knowledge discovery, and multi-modality. The Beijing Airlines team has long been committed to research fields such as deep learning, multi-modality, and computer vision. The goal of the TinyLLaVA+ project is to develop a small cross-language intelligent assistant with multi-modal capabilities such as language understanding, question and answer, and dialogue. The project team will give full play to their respective advantages, jointly overcome technical problems, and realize the design and development of intelligent assistants. This will bring new breakthroughs in research fields such as intelligent medical care, natural language processing and knowledge discovery, and multimodality. At the same time, Tsinghua University’s MSIIP laboratory has long been committed to intelligence.

Recently, Tsinghua University and Beihang University jointly launched TinyLLaVA Factory, a multi-model software that supports customization, training, and evaluation. The code base of the dynamic large model, the code and models are all open source. This code base uses the factory model of software engineering as the design concept, modularly reconstructs the LLaVA code base, focusing on the readability of the code, the scalability of functions, and the reproducibility of experimental results. It makes it easier for researchers and practitioners to explore the training and design space of multi-modal large models.

• ##Github project: https://github.com/TinyLLaVA/TinyLLaVA_Factory
• Paper address: https://arxiv.org/abs/2405.11788
• ##Hugging Face model address: https:// huggingface.co/tinyllava/TinyLLaVA-Phi-2-SigLIP-3.1B or https://huggingface.co/bczhou/TinyLLaVA-3.1B-SigLIP
• 心心SOTA model address: https://sota.jiqizhixin.com/project/tinyllava

LLaVA is a high-quality open source project of the multi-modal community , favored by researchers and developers. Beginners who are new to multi-modal large models are also accustomed to using the LLaVA project as a starting point to learn and train multi-modal large models. However, the code of the LLaVA project is relatively obscure and difficult to understand. Once errors are accidentally changed, the training effect may be affected. For novices, they often dare not easily modify the details, which hinders understanding and exploring the essential details of multi-modal large models. encountered certain difficulties.

Recently, Tsinghua University and Beihang University jointly launched TinyLLaVA Factory, which modularizes and reconstructs the original LLaVA code, focusing on simple code implementation, scalability of new functions, and training results The reproducibility allows you to customize and train your own multi-modal large model with the smallest amount of code, while reducing the code error rate! Under the same model configuration, training data and training strategy, using TinyLLaVA Factory can train a model with slightly better performance than using LLaVA code. In order to make it easier for users to understand the code and use the model, the TinyLLaVA Factory project is also equipped with code documentation and a Demo website. Its overall architecture is shown in the figure below: [Architecture Diagram].

In the data preprocessing part, TinyLLaVA Factory abandons the brain-burning image processing and prompt processing processes in the LLaVA code, and provides a standard and reliable Extended image and text preprocessing for clarity. Among them, image preprocessing can be customized with a Processor, or you can use the Processor of some official visual encoders, such as the Image Processor that comes with CLIP ViT and SigCLIP ViT. For text preprocessing, the base class Template is defined, providing basic and common functions, such as adding System Message (Prompt), Tokenize, and generating tags Ground Truth. Users can easily extend to different LLMs by inheriting the base class. Chat Template.

In the model part, TinyLLaVA Factory naturally modularizes the large multi-modal model into three components - the large language model component, the visual encoder component, and the intermediate connector component. Each component is controlled by a factory object, which is responsible for the registration and replacement of new models. Users can more easily replace any one of the components without involving other parts.

TinyLLaVA Factory provides the current mainstream models for each component, as shown in the table below.

The trainer is still modeled after LLaVA, using the Trainer that comes with Hugging Face, integrating Gradient Accumulation, Wandb logging and other features, and also supports DeepSpeed ZeRO2/ZeRO3 parallel training. For the evaluation part, TinyLLaVA Factory provides evaluation of 8 Benchmarks of SQA/GQA/TextVQA/VQAv2/POPE/MME/MM-Vet/MMMU.

Next, highlight the key points! The TinyLLaVA Factory Github project also teaches you how to customize your own multi-modal large model. LLM components, vision encoder components, connector components can be easily replaced by simply adding 1-2 files.

Take the replacement LLM model as an example. According to feedback from students who have used the LLaVA code base, LLaVA code is prone to errors when trying to replace non-Llama series language models. TinyLLaVA Factory can easily replace the language model by adding 2 py files, one is the Chat Template file and the other is the model file. When replacing the visual encoder, you only need to add 1 py file and inherit the base class of the visual encoder.

TinyLLaVA Factory also supports customization of training strategies. Users only need to modify the configuration file to pretrain and The finetuning stage implements any combination of freezing/full fine-tuning/partial fine-tuning/lora fine-tuning for the three module components (LLM/visual encoder/connector). It’s an easy-to-use tutorial for beginners!

As early as February this year, the TinyLLaVA project keenly captured the potential of LLM below 3B in multi-modal large models, taking advantage of the mainstream in the market The small-scale LLM trained a series of multi-modal large models with parameter sizes ranging from 0.89B to 3.1B. Experimental results show that through high-quality data selection and more detailed training strategies, small-scale LLM can also achieve similar or even superior task performance to large models. (See technical report https://arxiv.org/abs/2402.14289 for details)