What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs (2401.02411v1)

Abstract: 3D-aware Generative Adversarial Networks (GANs) have shown remarkable progress in learning to generate multi-view-consistent images and 3D geometries of scenes from collections of 2D images via neural volume rendering. Yet, the significant memory and computational costs of dense sampling in volume rendering have forced 3D GANs to adopt patch-based training or employ low-resolution rendering with post-processing 2D super resolution, which sacrifices multiview consistency and the quality of resolved geometry. Consequently, 3D GANs have not yet been able to fully resolve the rich 3D geometry present in 2D images. In this work, we propose techniques to scale neural volume rendering to the much higher resolution of native 2D images, thereby resolving fine-grained 3D geometry with unprecedented detail. Our approach employs learning-based samplers for accelerating neural rendering for 3D GAN training using up to 5 times fewer depth samples. This enables us to explicitly "render every pixel" of the full-resolution image during training and inference without post-processing superresolution in 2D. Together with our strategy to learn high-quality surface geometry, our method synthesizes high-resolution 3D geometry and strictly view-consistent images while maintaining image quality on par with baselines relying on post-processing super resolution. We demonstrate state-of-the-art 3D gemetric quality on FFHQ and AFHQ, setting a new standard for unsupervised learning of 3D shapes in 3D GANs.

• The paper introduces an SDF-based 3D GAN architecture that improves high-resolution 3D geometry rendering by reducing depth sample requirements up to fivefold.
• It employs learning-based samplers to efficiently select scene regions, ensuring consistent multi-view outputs without relying on super-resolution techniques.
• The method achieves state-of-the-art view-consistent results on datasets like FFHQ and AFHQ, advancing unsupervised 3D shape learning.

Introduction

Generative Adversarial Networks (GANs) have advanced dramatically over the last decade, especially in the field of image synthesis. One of the intriguing developments in this area is 3D-aware GANs, which can learn to recreate 3D scenes and geometries from 2D image collections. These 3D GANs rely on neural volume rendering techniques, but they have historically struggled with high computational and memory demands. This has limited their applications, making it challenging to achieve high-resolution outputs that maintain both geometric detail and multi-view consistency. A recent approach, however, presents a novel method to overcome these limitations, allowing for full-resolution rendering that captures fine-grained 3D geometry details without sacrificing image quality.

Scaling Challenges in 3D GANs

Previously, 3D GANs encountered roadblocks in scaling to high resolutions due to the intense resource requirements of volume rendering. For example, rendering a 512x512 pixel image might necessitate evaluating tens of millions of depth samples, demanding an impractical amount of GPU memory. To cope, researchers often used patch-based approaches or combined low-resolution neural rendering with 2D super-resolution (SR) techniques. Unfortunately, these workarounds compromised the consistency between different views and did not fully resolve 3D details.

Innovations in High-Fidelity Rendering

The new approach directly addresses the core issues that have held back high-resolution rendering in 3D GANs. The researchers present a set of techniques that constitute an end-to-end pipeline:

• SDF-based 3D GAN Architecture: Instead of relying on traditional volume rendering methods, the model uses a Signed Distance Function (SDF) based representation to encapsulate high-frequency geometry. This method rightly focuses on increasing the quality of surface geometry through training.
• Learning-based Samplers: By incorporating learning-based samplers into the rendering process, the model efficiently determines which parts of the scene need to be rendered at full resolution, reducing the number of depth samples needed by up to five times.
• Robust Sampling Strategy: The proposed sampling strategy ensures stable rendering with significantly fewer depth samples, maintaining image quality without resorting to super-resolution post-processing.

Impressive Results

Empirical demonstrations show that this methodology generates state-of-the-art 3D geometry quality, as tested on standard datasets such as FFHQ and AFHQ. Not only does the model produce strictly view-consistent images, but it does so with a level of detail previously unseen in unsupervised learning of 3D shapes in GANs. The results outperform existing methods in image quality, while also achieving unprecedented levels of geometric detail.

Conclusion

The proposed research represents a significant leap in the field of 3D-aware GANs, bridging the gap between 2D image quality and 3D geometric accuracy. With these advancements, theses GANs are poised to power a range of applications from content creation to novel view synthesis, providing tools that could revolutionize industries reliant on 3D modeling and visualization. As technology progresses, methods like these will continue to push the boundaries of what's possible in the synergy between artificial intelligence and 3D graphics.