We present DC-Gaussian, a new method for generating novel views from in-vehicle dash cam videos. While neural rendering techniques have made significant strides in driving scenarios, existing methods are primarily designed for videos collected by autonomous vehicles. However, these videos are limited in both quantity and diversity compared to dash cam videos, which are more widely used across various types of vehicles and capture a broader range of scenarios. Dash cam videos often suffer from severe obstructions such as reflections and occlusions on the windshields, which significantly impede the application of neural rendering techniques. To address this challenge, we develop DC-Gaussian based on the recent real-time neural rendering technique 3D Gaussian Splatting (3DGS). Our approach includes an adaptive image decomposition module to model reflections and occlusions in a unified manner. Additionally, we introduce illumination-aware obstruction modeling to manage reflections and occlusions under varying lighting conditions. Lastly, we employ a geometry-guided Gaussian enhancement strategy to improve rendering details by incorporating additional geometry priors. Experiments on self-captured and public dash cam videos show that our method not only achieves state-of-the-art performance in novel view synthesis, but also accurately reconstructing captured scenes getting rid of obstructions.
我们提出了 DC-Gaussian，这是一种从车载仪表盘摄像头视频生成新视角的新方法。尽管神经渲染技术在驾驶场景中取得了重大进展，但现有方法主要是为自动驾驶车辆收集的视频设计的。然而，与仪表盘摄像头视频相比，这些视频在数量和多样性上都有限，仪表盘摄像头视频在各种类型的车辆中更为普遍使用，并且能捕捉到更广泛的场景。仪表盘摄像头视频经常受到严重的遮挡，如挡风玻璃上的反射和遮挡，这在很大程度上阻碍了神经渲染技术的应用。为了应对这一挑战，我们基于最近的实时神经渲染技术三维高斯喷溅（3DGS）开发了 DC-Gaussian。我们的方法包括一个自适应图像分解模块，以统一的方式模拟反射和遮挡。此外，我们引入了照明感知的遮挡建模，以在不同的光照条件下管理反射和遮挡。最后，我们采用几何引导的高斯增强策略，通过整合额外的几何先验来改善渲染细节。在自采集和公开的仪表盘摄像头视频上的实验表明，我们的方法不仅在新视角合成中达到了最先进的性能，而且还能准确地重构捕获的场景，消除遮挡。