Feature Splatting for Better Novel View Synthesis with Low Overlap

An Overview of Feature Splatting for Enhanced 3D Scene Representation

The paper introduces a novel approach to improving 3D scene representations, specifically targeting the limitations found in traditional methods such as Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS). The proposed method, termed Feature Splatting (FeatSplat), aims to address the key limitation of using spherical harmonics in 3DGS by adopting per-Gaussian feature vectors to encode color information. This essay explore the methodology, experiments, and implications of this approach, providing an insightful synthesis of the paper's contributions.

Introduction

The challenge of finding appropriate 3D scene representations is pivotal for applications in robotics, virtual reality (VR), and augmented reality (AR). Traditional scene representations like NeRFs are computationally intensive and scale poorly with scene size. The recent introduction of 3D Gaussian Splatting (3DGS) presents a faster alternative, but it relies on spherical harmonics to encode color, which limits the model's expressivity and generalization capabilities.

Methodology

Feature Splatting (FeatSplat) enhances 3DGS by replacing spherical harmonics with per-Gaussian feature vectors. This approach involves three key steps:

1. Feature Vector Encoding: Each 3D Gaussian is initialized with a feature vector sampled from a normal distribution.
2. Alpha Blending: During image synthesis, 3D Gaussians are projected into the image plane, and their corresponding feature vectors are alpha-blended.
3. Multi-Layer Perceptron (MLP) Decoding: The blended feature vector is concatenated with a camera embedding and decoded by a small MLP to render RGB pixel values.

The paper also extends FeatSplat to semantic segmentation, demonstrating the flexibility of feature vector representations in encoding both RGB values and per-pixel semantic labels.

Preliminaries: 3D Gaussian Splatting

3DGS uses a set of 3D Gaussians to encode scene geometry and color information through spherical harmonics. The rendering process involves converting SHs to RGB values, projecting Gaussians into 2D, sorting them, and finally alpha-blending their colors. This method achieves high-quality rendering at a significantly lower computational cost compared to NeRFs, though it incurs a higher memory usage and suffers from poor generalization for complex textures and distant viewpoints.

Experiments

The paper evaluates FeatSplat on several datasets, including Mip-360, Tanks and Temples (T&T), Deep Blending (DB), and ScanNet++. The evaluation metrics included SSIM, PSNR, LPIPS, rendering speed (FPS), and memory usage.

Results on Mip-360, T&T, and DB

FeatSplat achieved the best PSNR on all three datasets and showed improved SSIM on two of them. The qualitative results highlighted FeatSplat’s ability to render accurate and detailed images, though it slightly lagged in rendering speed compared to 3DGS. Notably, FeatSplat halved the memory usage compared to 3DGS.

Generalization to Novel Views

FeatSplat demonstrated superior performance in synthesizing novel views far from training distributions, substantially reducing the artifacts seen with 3DGS. This was evident in sequences where the camera moved through isolated or largely unexplored regions, showcasing FeatSplat’s enhanced ability to adapt Gaussians’ representation flexibly based on viewing conditions.

Results on ScanNet++

FeatSplat significantly outperformed both 3DGS and Compact-3DGS across all metrics. This dataset posed a greater challenge with its independent test set trajectories, underscoring FeatSplat’s robustness in lower visual overlap and distant viewpoints.

Semantic FeatSplat

The extension to per-pixel semantic segmentation achieved a weighted mIOU of 0.629 while maintaining high rendering performance (PSNR 24.64, SSIM 0.875, LPIPS 0.244) at 56 FPS. The resulting segmentation maps were coherent, albeit slightly noisy at the edges.

Limitations

While FeatSplat significantly enhances representation capacity and generalization, it introduces a trade-off between capacity and speed. The compact MLP limits texture complexity, sometimes resulting in over-smoothing. The increased feature vector dimension also leads to slower rendering speeds compared to 3DGS, though the authors argue it remains within acceptable real-time limits.

Implications and Future Directions

FeatSplat extends the utility of 3DGS by providing a more expressive and flexible scene representation. Its capacity to encode multiple colors within a single Gaussian and condition decoding on viewpoint information offers robust improvements in practical applications like novel view synthesis and semantic segmentation. Future research could explore optimizing the balance between rendering speed and texture complexity, as well as further extending the versatility of feature vector representations in other 3D vision tasks.

By addressing key limitations in existing 3D scene representations, Feature Splatting represents a meaningful advance in enhancing the performance and applicability of novel view synthesis techniques. The authors’ commitment to releasing the code upon acceptance promises further developments and community-driven insights into this promising approach.