'It's hard to distinguish between true and false'! Clever use of autonomous driving simulation data generated by NeRF

Written before & the author’s personal understanding

Neural Radiation Field (NeRF) has become a tool to advance the prelude to the re-search of autonomous driving (AD), providing scalable closed-loop simulation and data enhancement capabilities. However, in order to trust the results obtained in the simulation, it is necessary to ensure that the AD system perceives the real data and the rendered data in the same way. Although the performance of rendering methods is improving, many scenes remain inherently challenging to faithfully reconstruct. To this end, we propose a new perspective to address the gap between real and simulated data. We not only focus on improving rendering fidelity, but explore simple yet effective methods to enhance the robustness of perceptual models to NeRF artifacts without affecting real data performance. Furthermore, we conduct the first large-scale investigation of the real-to-simulated data gap in AD settings using state-of-the-art neural rendering techniques. Specifically, our study evaluates object detectors and online mapping models on real and simulated data and investigates the effects of different pre-training strategies. Our results show significantly improved model accuracy on simulated data, even improving real-world performance in some cases. Finally, we delve into the real-to-simulated similarities, identifying FID and LPIPS as strong indicators.

In this article, we propose a new perspective to bridge the gap between intelligent driving systems and perception modules. Our goal is not to improve rendering quality but to make the perceptual model more robust to NeRF artifacts without degrading performance on real data. We believe that this direction is complementary to improving NeRF performance and is key to enabling virtual AV testing. As a first step in this direction, we show that even simple data augmentation techniques can have a strong impact on the model's robustness to NeRF artifacts.

We conduct the first extensive real2sim gap study on a large-scale AD dataset and evaluate the performance of multiple object detectors as well as online mapping models on real data and state-of-the-art (SOTA) neural rendering method data. . Our study includes the impact of different data augmentation techniques during training, and the fidelity of NeRF rendering during inference. We found that during model fine-tuning, these data showed the impact of the augmentation technique, and the fidelity of NeRF rendering even improved performance on real data in some cases. Finally, we study the correlation between implicit and common image reconstruction metrics in real2sim and gain insights into the significance of using NeRFs for CAD data simulators. We find that LPIPS and FID are powerful indicators of real2sim disparity, and further verify that our proposed enhancement reduces the sensitivity of contrast-poor visual compositionality.

Detailed explanation of method

To test and validate the AD functionality of the NeRF-driven simulation engine, they can use the data that has been collected to Explore new virtual scenes. However, in order for the results using such models to be trustworthy, the AD system must behave in the same way when processing the data and the actual data. In this work, we propose an alternative and complementary approach to adapt AD systems to make them less sensitive to differences between real and simulated data. In this way, we can tune the AD system to be less sensitive to differences between real and simulated data and thus better handle differences between real and simulated data.

As a first step in exploring how fine-tuning strategies can make perceptual models more robust to artifacts in rendered data, we used different fine-tuning strategies. Specifically, given an already trained model, we fine-tune the perceptual model using images that focus on improving the performance of rendered images while maintaining performance on real data, see Figure 2. In addition to reducing the real2sim gap, this may also reduce the requirements for sensor realism, pave the way for wider application of neural rendering methods, and reduce the computational requirements for the training and evaluation of description methods. Note that while we focus on perceptual models, our approach can be easily extended to end-to-end models as well.

Finally, we can imagine multiple ways to achieve the goal of making models more robust, such as drawing inspiration from the domain adaptation and multi-task learning literature. However, fine-tuning requires minimal model-specific adjustments, allowing us to easily study a range of models.

Image augmentations

Image augmentation is a commonly used method. The classic strategy for enhancing robustness against artifacts is to use image augmentation. Here, we choose enhancements to represent the various distortions present in the rendered image. More specifically, we add random Gaussian noise, convolve the image with a Gaussian blur kernel, and apply photometric distortion similar to that found in SimCLR. Finally, the image is downsampled and upsampled. Augments are applied sequentially, and each augmentation has a certain probability.

Fine-tuning with mixed-in rendered images

#NeRF是用來渲染3D場景的深度學習模型。在微調期間，模型可以適應另一種自然形式，即在微調期間包含這種數據。這有助於訓練NeRF模型的能力，以便可以在監控感知模型的相同資料集上訓練NeRF方法。然而，在大型資料集上訓練NeRF的成本可能高昂，其中一些可能需要進行3D物件偵測、語義分割或多種類別標籤等任務的標籤。此外，AD的NeRF常常也增加了中資料的順序要求。為了適應這些要求，標籤可能需要更多的特殊處理，例如3D物件偵測，語意分割或多種類別標籤等。

接下來，我們將中所選序列的影像分割為NeRF訓練集和保持集。感知模型的微調是在它們的整個訓練資料集D上完成的，對於在D中具有渲染對應關係的影像，我們使用具有機率p的渲染影像。這意味著用於微調的圖像沒有被NeRF模型看到。

Image-to-image translation

如前所述，渲染NeRF資料是一種昂貴的資料增強技術。此外，除了感知任務所需的資料外，它還需要順序資料和潛在的額外標記。也就是說，為了獲得可擴展的方法，我們理想地想要一種有效的策略來獲得單一影像的NeRF資料。為此，我們建議使用圖像對圖像的方法來學習生成類別NeRF圖像。給定真實影像，該模型將影像轉換到NeRF域，有效地引入了NeRF的典型偽影。這使我們能夠在有限的運算成本下，在微調過程中大幅增加類NeRF影像的數量。我們使用渲染圖像Dnerf及其對應的真實圖像來訓練圖像到圖像模型。不同增強策略的可視化範例見圖3。

結果

##神經輻射場（NeRF ）已成為模擬自動駕駛（AD）數據的一種很有前途的途徑。然而，為了實用，必須了解AD系統在模擬資料上執行的行為是如何轉換為真實的資料。我們的大規模調查揭示了暴露於模擬影像和真實影像的感知模型之間的表現差距。

與早期專注於提升渲染品質的方法不同，本文研究如何改變感知模型，使其對NeRF模擬資料更具穩健性。我們表明，使用NeRF或類似NeRF的資料進行微調，在不犧牲真實資料效能的情況下，大大減少了物件偵測和線上映射方法的real2sim間隙。此外，我們還表明，在現有的列車分佈之外產生新的場景，例如模擬車道偏離，可以提高實際數據的效能。對NeRF社群內常用影像指標的研究表明，LPIPS和FID分數與感知表現表現出最強的相關性。這表明，與單純的重建品質相比，感知相似性對感知模型具有更大的意義。

總之，我們認為NeRF模擬資料對AD有價值，尤其是當使用我們提出的方法來增強感知模型的穩健性時。此外，NeRF資料不僅有助於在模擬資料上測試AD系統，而且有助於提高感知模型在真實資料上的表現。