Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering (2106.02634v2)

Abstract: Inferring representations of 3D scenes from 2D observations is a fundamental problem of computer graphics, computer vision, and artificial intelligence. Emerging 3D-structured neural scene representations are a promising approach to 3D scene understanding. In this work, we propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field parameterized via a neural implicit representation. Rendering a ray from an LFN requires only a single network evaluation, as opposed to hundreds of evaluations per ray for ray-marching or volumetric based renderers in 3D-structured neural scene representations. In the setting of simple scenes, we leverage meta-learning to learn a prior over LFNs that enables multi-view consistent light field reconstruction from as little as a single image observation. This results in dramatic reductions in time and memory complexity, and enables real-time rendering. The cost of storing a 360-degree light field via an LFN is two orders of magnitude lower than conventional methods such as the Lumigraph. Utilizing the analytical differentiability of neural implicit representations and a novel parameterization of light space, we further demonstrate the extraction of sparse depth maps from LFNs.

• The paper presents a novel approach using Light Field Networks that achieves single-evaluation rendering, making it over 100 times faster than traditional volumetric methods.
• It utilizes neural parameterization with Plücker coordinates and a meta-learning framework to enable continuous 360° scene reconstruction from sparse image observations.
• Experimental results show that LFNs outperform baseline methods with over 1 dB improvement in PSNR and enable real-time novel view synthesis across diverse ShapeNet object categories.

Light Field Networks: A Novel Approach to Neural Scene Representation

The paper "Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering" introduces a novel approach for inferring 3D scene representations. The paper transcends conventional 3D volumetric techniques by leveraging Light Field Networks (LFNs) to achieve real-time rendering with significant reductions in computational cost and memory requirements.

Key Contributions and Methodology

The authors propose LFNs as a neural-based method to represent 3D scenes using a four-dimensional light field parameterized via neural networks. This representation permits single-evaluation rendering, drastically improving speed over volumetric approaches such as Lumigraph by two orders of magnitude. A distinct feature of LFNs is their use of Plücker coordinates for parameterizing light fields, which allows continuous 360-degree scene representations. This aspect enables the inference of geometry details from the scene's light field, providing the capacity for sparse depth map extraction through analytical differentiation without traditional ray-casting methods.

Another innovative aspect of the research is the integration of LFNs into a meta-learning framework for training. This framework facilitates multi-view consistent reconstruction of scenes from sparse image observations (as minimal as a single image), showcasing LFNs as capable of efficient novel view synthesis and 3D reconstruction. The use of a hypernetwork allows LFNs to learn a prior over light fields, thereby ensuring that the generated light field representation aligns with the constraints of physically plausible 3D scenes.

Implications and Numerical Significance

The numerical results exhibit LFNs' ability to achieve real-time rendering speeds, offering an unprecedented leap in computational efficiency compared to traditional 3D volumetric rendering methods, which are notably expensive in terms of time and memory. Performance evaluations across various experiments reveal that LFNs outperform existing methods like Scene Representation Networks (SRNs) and Differentiable Volumetric Rendering (DVR) in single-shot reconstruction across multiple object categories from the ShapeNet dataset. The PSNR results consistently surpass these baselines by over 1 dB on average. Furthermore, LFNs manage view synthesis in real-time, which is three orders of magnitude faster than some stereo-consistent methods.

Potential Applications and Future Directions

The implications of LFNs extend to various domains such as computer vision, graphics, and robotics, offering new avenues for applications like real-time rendering, virtual and augmented reality, and efficient 3D indexing and retrieval. The proposed method demonstrates a clear advantage over locally-conditioned methods in scenarios where a compact scene representation is prioritized, though not exceeding local conditioning techniques such as pixelNeRF in some respects.

Future research could address extensions of LFNs to non-Lambertian scenes, the incorporation of local conditioning mechanisms to further enhance generalization, and enabling camera movement in obstructed environments. As LFNs do not inherently enforce multi-view consistency, addressing this limitation could further bolster their utility in more complex setups.

Overall, this paper provides a structured and substantial contribution to neural rendering techniques, introducing a framework that significantly outperforms traditional approaches in rendering speed and resource efficiency. The concept of parametrizing light fields neurally and resolving the associated challenges sets a promising precedent for subsequent investigations within the neural scene representation landscape.