New SOTA for target detection, real-time recognition on the device and side, Shen Xiangyang rarely forwards and likes

The field of target detection has ushered in new progress-

Grounding DINO 1.5, produced by the IDEA Research Institute team, can achieve real-time recognition on the device side.

This progress was forwarded by AI tycoon Shen Xiangyang, who usually makes changes every year.

There are two main versions of this release: Pro and Edge. The Pro version is stronger and the Edge version is faster.

It still retains the previous versionGrounding DINODual encoder-single decoder structure, based on which the model size is expanded by combining a larger visual backbone and uses more than 20 million The grounding data has obtained rich corpus, which greatly improves the detection accuracy and speed, and is optimized for different application scenarios through the Pro and Edge versions.

In large-scale data set construction and high-precision demand scenarios, the Pro version performs excellently, while the Edge version demonstrates its unique advantages in end-side deployment.

Let’s take a look at them separately.

Pro version target detection new SOTA

Grounding+DINO+1.5 Pro version achieves the current SOTA level of open set target detection, performs well in semantic understanding of images and text, and can be fast and accurate Detect and identify target objects in images based on language cues.

△Zero-shot migration performance comparison in COCO, LVIS, ODinW35 and ODinW13 benchmarks

Object-level understanding is the perception of the interaction between the machine and the physical world The foundation is also the basic problem that cannot be bypassed to solve the hallucination problem of multi-modal large models (VLM).

As the current best-performing open set detection model, Grounding DINO 1.5 Pro can help construct massive multi-modal data with object-level semantic information, thereby effectively assisting the training of multi-modal large models.

It can accurately match phrases in long text descriptions with specific objects or scenes in images to enhance AI’s understanding of the relationship between visual content and text

In addition, Grounding DINO 1.5 Pro also has strong application value in other fields that require processing large amounts of complex data, such as e-commerce, social media, and autonomous driving.

For example, in the field of e-commerce, this model can help quickly label product images and optimize search and recommendation systems. In social media, this model can automatically label images uploaded by users, improving the efficiency of content review and classification.

Supports industry data fine-tuning

In addition, the Pro version also supports fine-tuning through industry data to meet the specific needs of various industries, thereby achieving more accurate identification results. .

In order to verify the improvement brought by fine-tuning, the CVR team conducted comparative experiments on public data sets such as LVIS that are common in the visual field.

As can be seen from the last two lines, Grounding DINO 1.5 Pro has been fine-tuned and has shown substantial performance improvements on multiple data sets.

And it is also very suitable for many practical scenarios.

Like in the medical field, the fine-tuned Grounding DINO 1.5 Pro can more accurately identify lesions in medical images, assist doctors in diagnosis, and improve diagnosis and treatment efficiency.

In the retail industry, fine-tuned models can more accurately identify and classify goods, helping with inventory management and sales analysis.

Edge version can be deployed on the client side

In terms of client-side deployment, Grounding DINO 1.5 Edge version has been successfully deployed on the NVIDIA Orin NX card through model structure optimization, and achieved an inference speed of 10FPS. .

Furthermore, it allows robots to interact with open environments.

In the field of autonomous driving, Grounding DINO 1.5 Edge can run in real-time on vehicles in the future to achieve efficient target detection and environment perception, improving driving safety. In smart security, this model can quickly process video surveillance data, detect abnormal behaviors in real time, and improve the response speed of security monitoring.

In the future, the running speed of Grounding DINO 1.5 Edge is expected to increase to 20 to 30FPS, further expanding its application scope in the field of edge computing.

Paper link:
https://arxiv.org/abs/2405.10300
Project trial link:
https://deepdataspace.com/playground/grounding_dino