New progress in Li Feifei's 'Spatial Intelligence' series, Wu Jiajun's team's new 'BVS' suite evaluates computer vision models

Application scenarios

The researchers demonstrated the role of data generated by BVS in three application scenarios, including:

• Parameters controllably evaluate the robustness of the visual model when environmental parameters continuously change: Explore the model's performance under different environmental parameters (occlusion degree, environment (brightness, shooting angle, object joint movement, field of view) continuously changes, for example, to evaluate the accuracy of the object detection model when the refrigerator door is fully closed to fully open. The application can cope with various environmental changes. The capability boundaries of the model under different limit parameter conditions can also be explored.
• Evaluate scene understanding models: Using fully annotated images, the performance of various scene understanding models is systematically evaluated.
• Train a new vision task model: Train a new vision task model for object state and relationship prediction on synthetic data, and evaluate its performance from simulation to real applications Transfer capabilities to ensure model effectiveness in real-world environments.

Parameters controllably evaluate the robustness of the vision model when environmental parameters continuously change

By generating data that continuously changes in a certain dimension, the researchers systematically evaluate the robustness of the visual model under this change. For example, data with gradually increasing degrees of object occlusion in the same scene are generated to evaluate the performance of the visual model under partially occluded objects.

By evaluating different SOTA models, researchers found that the performance of existing models on data outside common distributions is still insufficient. Since these data are difficult to obtain or label in the real world, these conclusions are difficult to draw directly from real image data sets. Therefore, BVS can help researchers evaluate the robustness of the model under the conditions of interest to them to better develop and improve the model.

The existing SOTA model still has room for improvement in robustness under changing conditions (such as camera elevation)

Performance of different detection models when five environmental parameters change continuously

Evaluating scene understanding models

Another major feature of the dataset generated by BVS is that it contains multi-modal real labels, such as Depth, semantic segmentation, target bounding box, etc. This allows researchers to use data generated by BVS to evaluate prediction models for different tasks on the same image.

The research team evaluated the SOTA model for four tasks: open word detection and segmentation, depth estimation, and point cloud reconstruction, and found that the model's performance on the BVS data set was in the same order as in the corresponding tasks. The performance on real data benchmarks is consistent. This shows that the high-quality data generated by BVS truly reflects and represents real-world data, and researchers hope that such data sets can promote the development of multi-task prediction models.

In the open source code, the research team also provides a script to facilitate users to sample trajectories in the scene.

The researchers collected many scene browsing videos to evaluate the scene understanding model

##Overall scene understanding data set. The researchers generated a large number of traversal videos in representative scenes, each containing more than 10 camera trajectories. For each image, BVS generates various labels (e.g., scene map, segmentation mask, depth map)

The relative performance order of the SOTA model on the BVS data is consistent with the real task benchmark

Train the new vision task model

#BVS data generation is not limited to model evaluation. For tasks that are difficult to collect or label data in real-life scenarios, BVS data can also be used for model training.

The author used BVS to generate 12.5k pictures, and only used it to train an object spatial relationship and state prediction model. This model achieved an F1 score of 0.839 in real scenarios without using real data for training, reflecting excellent simulation-to-real transfer capabilities.

Simulation generated training data set and real test data set illustration

Object spatial relationship and state prediction model trained using data generated by BVS

Summary

BVS provides a set of powerful A large set of tools and resources provides computer vision researchers with new ways to generate customized synthetic data sets.

By systematically controlling and adjusting various parameters in the data generation process, researchers can more comprehensively evaluate and improve the performance of computer vision models, laying the foundation for future research and development. Application lays a solid foundation.