SIGNeRF: Scene Integrated Generation for Neural Radiance Fields

Introduction

This paper introduces SIGNeRF, an innovative method for editing existing 3D scenes and integrating new objects with high fidelity using generative 2D diffusion models. Traditional approaches require complex pipelines, iterative optimizations, and lack precise control over the results. SIGNeRF addresses these challenges by leveraging a reference-sheet-based update strategy, ensuring 3D consistency across edits. This novel strategy for generating and updating image sets is based on a multi-view reference sheet made up of modified images, which then refine the original Neural Radiance Fields (NeRF) scene in a single operation.

Background and Related Work

In terms of related work, the paper reflects on the advancements in text-to-image and text-to-3D generation, acknowledging the integration of diffusion probabilistic models with large datasets. Using these models, researchers have been able to generate high-resolution and diverse images. Moreover, ControlNet, an image diffusion model with additional depth guidance, has shown inherent capabilities of generating coherent and consistent views. The paper also discusses the challenges of editing NeRF scenes and how current solutions provide a limited range of capabilities. Lastly, generative NeRF editing is considered, showing that there's a growing interest in modifying existing NeRF scenes by applying generative 3D models.

Methodology

The proposed SIGNeRF pipeline comprises several stages, starting with the training of an original NeRF scene. Upon selecting the 3D region to edit, reference cameras are placed, and corresponding color, depth, and mask images are rendered and arranged into image grids. ControlNet processes these grids to produce a reference sheet, which is then used to iteratively update images in the NeRF dataset to ensure multi-view consistency. The editing is fine-tuned by introducing two selection methods in the scene space: a mesh proxy and a bounding box selection mode. An optional second iteration can be performed if necessary, and due to the modular pipeline, individual components can be fine-tuned or exchanged easily.

Outcomes and Comparison

The SIGNeRF pipeline yields superior results in object generation and editing with consistent style across all views. When compared to existing methods such as Instruct-NeRF2NeRF and DreamEditor, SIGNeRF demonstrates improved performance in terms of scene preservation, selection precision, and generation quality. Notably, it enables more complex object edits and can preview edits before generating the complete updated image set. It also outperforms in terms of generation speed, taking half as much time as other methods. Quantitative evaluations using CLIP text-to-image directional similarity and metrics such as PSNR and SSIM indicate SIGNeRF's advantages in preserving unedited parts of the scene and achieving higher fidelity to the text prompts. However, the method has limitations, such as reduced edit quality for objects far from the camera and challenges in editing off-center objects or extensive scene modifications.

Conclusions

SIGNeRF presents a breakthrough in scene-integrated editing for NeRF scenes, offering a fast, controllable, and customizable approach to 3D generation. It leads to more consistent edits in single runs, optimizes the process compared to current editing methods, and provides an initial preview to users. Although focused on NeRF, the modularity of SIGNeRF enables adaptation to other 3D scene representations. Despite potential misuses of technology in creating convincing forgeries, the authors hope SIGNeRF will further democratize 3D content generation, ultimately benefiting the broader field.