RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS
(2403.13806)Abstract
Recent advances in view synthesis and real-time rendering have achieved photorealistic quality at impressive rendering speeds. While Radiance Field-based methods achieve state-of-the-art quality in challenging scenarios such as in-the-wild captures and large-scale scenes, they often suffer from excessively high compute requirements linked to volumetric rendering. Gaussian Splatting-based methods, on the other hand, rely on rasterization and naturally achieve real-time rendering but suffer from brittle optimization heuristics that underperform on more challenging scenes. In this work, we present RadSplat, a lightweight method for robust real-time rendering of complex scenes. Our main contributions are threefold. First, we use radiance fields as a prior and supervision signal for optimizing point-based scene representations, leading to improved quality and more robust optimization. Next, we develop a novel pruning technique reducing the overall point count while maintaining high quality, leading to smaller and more compact scene representations with faster inference speeds. Finally, we propose a novel test-time filtering approach that further accelerates rendering and allows to scale to larger, house-sized scenes. We find that our method enables state-of-the-art synthesis of complex captures at 900+ FPS.
Overview
-
RadSplat combines radiance field insights with Gaussian Splatting to render complex scenes in real-time, achieving over 900 FPS.
-
It uses radiance fields for optimization and supervision, introduces a pruning method for compact scene representation, and a test-time rendering approach for faster speeds.
-
The method significantly outperforms existing real-time rendering methods on conventional benchmarks, achieving high-quality results at unprecedented speeds.
-
Despite its high performance, RadSplat's longer training time and slight lag behind state-of-the-art in some metrics present areas for future improvement.
RadSplat: Achieving Real-Time Rendering of Complex Scenes with Radiance Field-Informed Gaussian Splatting
Overview
This work introduces RadSplat, a method that adeptly combines radiance field insights with Gaussian Splatting (GS) mechanics to robustly render complex scenes in real-time, achieving remarkable speeds of over 900 frames per second (FPS). At its core, RadSplat leverages the detail-oriented and stable optimization characteristics of radiance fields to inform the optimization of point-based scene representations. This approach not only improves the rendering quality and robustness but also significantly reduces computational overhead, leading to smaller, more compact scene representations. The main contributions include the use of radiance fields as both a prior and supervision signal for GS optimization, the introduction of a novel pruning technique to reduce the number of Gaussian primitives without quality loss, and a novel test-time rendering approach that further accelerates rendering speeds.
Radiance Field-Informed Optimization
Radiance Field-based methods, although achieving state-of-the-art quality in various scenarios, suffer from high compute requirements due to volumetric rendering's inherent complexity. In contrast, Gaussian Splatting-based methods that capitalize on rasterization naturally achieve real-time rendering but often underperform in more complex scenes due to fragile optimization heuristics. RadSplat addresses these challenges by leveraging radiance fields for both initializing and supervising the optimization of the point-based 3DGS representation, leading to enhanced quality and more robust optimization.
Pruning for Compact Scene Representation
The paper innovates a pruning strategy that reduces the overall point count in a scene by up to ten times while maintaining high quality. This is achieved through a novel importance score that assesses the contribution of each Gaussian primitive to the rendered views, allowing for the removal of non-contributing points during optimization. The result is a more compact scene representation that favors faster inference speeds without compromising rendering quality.
Visibility-Based Rendering for Real-Time Performance
RadSplat introduces a test-time filtering approach that enables even quicker rendering times. By clustering input camera views and employing a visibility filtering step, the method dynamically adjusts the set of rendered points to those relevant to the given viewpoint. This approach yields substantial FPS improvements, especially beneficial when scaling to larger, more complex scenes.
Notable Results
When evaluated on conventional benchmarks, RadSplat surpasses existing real-time rendering methods, including both NeRF- and GS-based models, achieving state-of-the-art synthesis of complex captures at unprecedented speeds. Specifically, on the MipNeRF360 dataset, RadSplat outperforms competing methods in quality metrics (SSIM, PSNR, LPIPS) while offering an incredible mean rendering speed of 907 FPS for its lightweight variant and 410 FPS for the default configuration. Even on the demanding large-scale scenes of the Zip-NeRF dataset, RadSplat exhibits superior SSIM values while furnishing a significant increase in rendering speed, up to 748 FPS for its lightweight variant.
Future Directions
Despite its achievements, RadSplat's training procedure, encompassing both a radiance field and a Gaussian Splatting representation, presents a longer training duration compared to single-representation models. Moreover, while RadSplat sets new performance benchmarks on the MipNeRF360 dataset, it marginally lags behind Zip-NeRF, the state-of-the-art in offline view synthesis, on the Zip-NeRF scenes primarily in PSNR and LPIPS metrics. The exploration of strategies to enhance training efficiency and further close the gap in quality on large-scale scenes will be intriguing future directions.
Conclusions
RadSplat stands as a pioneering contribution to the field of real-time 3D rendering, coupling the nuanced detail capture of radiance fields with the efficiency and real-time capability of Gaussian Splatting. Through novel pruning techniques and adaptive test-time filtering, it adeptly manages to balance quality and performance, rendering it an exemplary method for robust real-time rendering of complex scenes.
Create an account to read this summary for free: