CVPR 24｜ETH Zurich and other teams: Redefining the small sample 3D segmentation task, the new benchmark opens up broad potential for improvement!

The AIxiv column is a column where this site publishes academic and technical content. In the past few years, the AIxiv column of this site has received more than 2,000 reports, covering top laboratories from major universities and companies around the world, effectively promoting academic exchanges and dissemination. If you have excellent work that you want to share, please feel free to contribute or contact us for reporting. Submission email: liyazhou@jiqizhixin.com; zhaoyunfeng@jiqizhixin.com

The first author of this article, An Zhaochong, is currently studying for a PhD at the University of Copenhagen, under the supervision of Serge Belongie. He graduated with a master's degree from ETH Zurich. During his master's degree, he participated in multiple research projects in the laboratory of his mentor Luc Van Gool. His main research interests include scene understanding, few-shot learning, and multi-modal learning.

3D scene understanding allows humanoid robots to "see" the surrounding scene, allowing the car's autonomous driving function to perceive possible situations during driving in real time, thereby making more intelligent behaviors and reactions. All of this requires detailed annotation of a large number of 3D scenes, which sharply increases time cost and resource investment.

Recently, ETH Zurich and other teams proposed a few-shot learning method, which greatly improved this limitation, re-examined the current FS-PCS task, and introduced a new benchmark in the field of 3D scene perception to provide a good foundation for the future. Model design and development breaks new ground.

• Paper link: https://arxiv.org/abs/2403.00592
• Code link: https://github.com/ZhaochongAn/COSeg

^{3D Few-shot segmentation result example}

Technical background

3D scene understanding plays a vital role in fields such as autonomous driving and intelligent robots. It enables devices to perceive and understand their surroundings three-dimensional world. Although traditional fully supervised learning models perform well at recognition of specific categories, these models are often limited to identifying these predefined categories. This means that whenever a new object category needs to be recognized, a large amount of 3D scene data must be collected and annotated in detail. This process is not only time-consuming and labor-intensive, but also greatly limits the application of fully supervised models in the real world. Application breadth and flexibility.

However, with the help of the few-shot learning method, this situation has been significantly improved. Few-shot learning is a technique that requires very few labeled samples to quickly adapt to new categories. This means that the model can quickly learn and adapt to new environments with a small number of examples, greatly reducing the cost of data collection and processing. This fast and flexible learning method makes 3D scene understanding technology more adaptable to the rapidly changing real world, opening up new possibilities for various application scenarios such as autonomous driving and advanced robotic systems. Therefore, studying few-shot 3D models can effectively promote the practical application of many important tasks in the wider world.

Specially, for the Few-shot 3D point cloud semantic segmentation (FS-PCS) task, the input of the model includes the support point cloud and the annotation of the new category (support mask) and query point cloud. The model needs to obtain knowledge about new categories by utilizing the support point cloud and support mask and apply it to the segmentation query point cloud to predict the labels of these new categories. The target categories used during model training and testing do not overlap to ensure that the categories used during testing are all new categories and have not been seen by the model during training.审 The re -examining and correction of the task

Figure 1. The visualization of the two scenes (the prospects are Door and BOARD)

W Table 1. Existence (w/fg) and the performance of the previous model when there is no prospect leak (W/O FG)

This article re-examines the current FS-PCS task. It is found that the current task setting has two significant problems:

The first problem is

foreground leakage

: 3D tasks usually uniformly sample dense points in the scene point cloud as the input of the model. However, the sampling method used by FS-PCS is not uniform sampling. Instead, it will sample more points for the target category (foreground area) and sample fewer points for the non-target area (background area). The input point cloud obtained in this way There will be a denser distribution of points in the foreground, resulting in foreground leakage problems. As shown in Figure 1, the input point clouds in the fourth and sixth columns come from the current biased sampling and show a denser point distribution in the foreground area (door or board) than in the background, while the input point clouds in the third and fifth columns The input uses corrected consistency sampling, exhibiting a uniform point density distribution. This problem causes the information of the new class to be leaked by the density distribution of the point cloud, allowing the model to simply exploit the density difference in the input point cloud and predict denser areas as foregrounds to achieve good few-shot performance. It does not rely on learning the ability to transfer knowledge from support to query. Therefore, the current evaluation
• benchmark cannot reflect the real performance of past models
  . As shown in Table 1, after correcting the foreground leakage in the current setting, the past models showed a large performance drop, indicating that the past models relied heavily on density differences to achieve seemingly superior few-shot performance. The second problem is sparse point distribution: the current settingonly samples 2048 pointsfrom the scene as input to the model during training and testing. Such sparse point distribution severely limits the semantics of the input scene Clarity. As shown in Figure 1, in the first row and fifth column, it is difficult for the human eye to distinguish the semantic category door in the area from the surrounding category wall. For the second line, it is also difficult to distinguish whether the target area is the board class or another class such as window. These sparse input point clouds
  have very limited semantic information
,
• introducing significant ambiguity
  and limiting the model’s ability to effectively mine the semantics in the scene. Therefore, in order to correct these problems, the author proposed a new setting to standardize the FS-PCS task
  , using uniform sampling and increasing the number of sampling points 10 times to 20480 points. As shown in the third column of Figure 1, the input under the new setting has a consistent point distribution and clearer semantic information, making the task closer to the real application scenario.

New model COSeg

Under the newly corrected setting, the author introduced a new model called Correlation Optimization Segmentation (COSeg). Previous methods are based on the

feature optimization paradigm, focusing on optimizing the features of support or query, and inputting the improved features into the parameter-free prediction module to obtain prediction results, which can be regarded as implicit modeling between support and query Correlations

. On the contrary, it does not focus on optimizing features. The article proposes a

correlation optimization paradigm
, which directly inputs the correlations between support and query into a module with parameters, and

explicitly optimizes correlations

, allowing the model to directly shape the correlation between query and support. relationship, which enhances the generalization ability of the model.

                                                                                                                                                                                                          ^{Figure 2. COSeg architecture}

In COSeg, first calculate the Class-specific Multi-prototypical Correlation between each query point and the support prototypes, referred to as CMC, which means every point and all Relationships between category prototypes . The CMC is then input into the subsequent Hyper Correlation Augmentation (HCA) module.

HCA module utilizes two potential relationships to optimize correlations. First, the query points are all related to each other, so they are also related to the correlations between category prototypes. From this, we can get the relationship between points and points , which corresponds to the first half of HCA doing correlations in the point dimension. attention. Second, classifying a query point into a foreground or background class depends on the relative correlations between the foreground and background prototypes of the point. From this, we can get the relationship between the foreground and the background, which corresponds to the second half of HCA. Correlations do attention in the category dimension.

In addition, since the few-shot model is trained on the base category and tested on the novel category. These models will be easily interfered by familiar base categories existing in the test scene, affecting the segmentation of novel categories. In order to solve this problem, the article proposes to learn parameter-free prototypes for the base category (called base prototypes). When

segment a new class

, the query points belonging to the base class should be predicted as background . Therefore, using base prototypes, the author introduces the Base Prototypes Calibration (BPC) module inside the HCA layer to adjust the correlations between point and background categories, thereby alleviating the interference caused by the base class. Experimental results

                                                                                                                          

                                                                                                                                                                                                                                                                                    ^{benchmark, and proved The superior performance of the COSeg method achieves the best results in various few-shot tasks. The visualization also clearly shows that COSeg achieves better segmentation results. In addition, the author also provides extensive ablation experiments to prove the effectiveness of the design and the superiority of the correlation optimization paradigm.}

Summary

The contributions of this article’s research in the field of FS-PCS are as follows.

First, the authors identify two key issues in the current FS-PCS setting (

foreground leakage and sparse point distribution

), which reduce the accuracy of evaluation benchmarks against past methods. In order to solve the problems in the previous setting, the article introduces a new standardized setting and evaluation benchmark

.

In addition, under the standardized FS-PCS setting, the author proposes a new correlation optimization paradigm, which significantly improves the generalization performance of the model on few-shot tasks. The model in this article COSeg combines HCA to mine effective point cloud correlation information and BPC to adjust background prediction

, achieving the best performance on all few-shot tasks.

The standardized setting corrected in the article opens up more possibilities to improve the few-shot 3D segmentation task. At the same time, the new correlation optimization paradigm proposed also provides a new direction for future model design and development.

. As a new benchmark in the field of FS-PCS, this work is expected to inspire more researchers to explore and expand the boundaries of small-sample 3D scene understanding.

As a reference, the following points can be used as potential research directions to further promote the development of this field:

Under the new setting in the article, although COSeg achieves the best performance, it still There is a lot of room for improvement. The model can be improved to achieve better few-shot generalization: such as improving the prototype extraction method [1, 2], improving the correlation optimization module [3], and targeting each few-shot task. sexual training[4].

Solving the Base category interference problem is also a key factor affecting Few-shot performance. It can be optimized from the perspective of training or model design [5,6] to better reduce the Base category interference.

• Improve the training and inference efficiency of the model [7], especially when deployed to practical applications, the efficiency of the model is also a key consideration.
• In summary, this field has very broad prospects, and it is still in its emerging initial stage. For the majority of researchers, it is undoubtedly a research field full of hope and opportunities.

Reference link:

[1] Lang, Chunbo, et al. "Progressive parsing and commonality distillation for few-shot remote sensing segmentation." IEEE Transactions on Geoscience and Remote Sensing (2023 ).

[2] Liu, Yuanwei, et al. "Intermediate prototype mining transformer for few-shot semantic segmentation." Advances in Neural Information Processing Systems 35 (2022): 38020-38031.

[3] Zhang, Canyu, et al. "Few-shot 3d point cloud semantic segmentation via stratified class-specific attention based transformer network." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 3. 2023 .

[4] Bodiaf, Malik, et al. "Few-shot segmentation without meta-learning: A good transductive inference is all you need?." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[5] Wang, Jiahui, et al. "Few-shot point cloud semantic segmentation via contrastive self-supervision and multi-resolution attention." 2023 IEEE International Conference on Robotics and Automation (ICRA ). IEEE, 2023.

[6] Lang, Chunbo, et al. "Learning what not to segment: A new perspective on few-shot segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[7] Sun, Yanpeng, et al. "Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning." Advances in Neural Information Processing Systems 35 (2022 ): 37484-37496.

The above is the detailed content of CVPR 24｜ETH Zurich and other teams: Redefining the small sample 3D segmentation task, the new benchmark opens up broad potential for improvement!. For more information, please follow other related articles on the PHP Chinese website!