Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LAENeRF: Local Appearance Editing for Neural Radiance Fields (2312.09913v2)

Published 15 Dec 2023 in cs.CV

Abstract: Due to the omnipresence of Neural Radiance Fields (NeRFs), the interest towards editable implicit 3D representations has surged over the last years. However, editing implicit or hybrid representations as used for NeRFs is difficult due to the entanglement of appearance and geometry encoded in the model parameters. Despite these challenges, recent research has shown first promising steps towards photorealistic and non-photorealistic appearance edits. The main open issues of related work include limited interactivity, a lack of support for local edits and large memory requirements, rendering them less useful in practice. We address these limitations with LAENeRF, a unified framework for photorealistic and non-photorealistic appearance editing of NeRFs. To tackle local editing, we leverage a voxel grid as starting point for region selection. We learn a mapping from expected ray terminations to final output color, which can optionally be supervised by a style loss, resulting in a framework which can perform photorealistic and non-photorealistic appearance editing of selected regions. Relying on a single point per ray for our mapping, we limit memory requirements and enable fast optimization. To guarantee interactivity, we compose the output color using a set of learned, modifiable base colors, composed with additive layer mixing. Compared to concurrent work, LAENeRF enables recoloring and stylization while keeping processing time low. Furthermore, we demonstrate that our approach surpasses baseline methods both quantitatively and qualitatively.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Unmixing-Based Soft Color Segmentation for Image Manipulation. ACM TOG, 36(2):61c:1–61c:19, 2017.
  2. SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field. In CVPR, 2023.
  3. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In CVPR, 2022.
  4. Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. In ICCV, 2023.
  5. NeRD: Neural Reflectance Decomposition from Image Collections. In CVPR, 2021.
  6. Hallucinated Neural Radiance Fields in the Wild. In CVPR, 2022.
  7. Unified Implicit Neural Stylization. In ECCV, 2022.
  8. Plenoxels: Radiance Fields without Neural Networks. In CVPR, 2022.
  9. Image Style Transfer Using Convolutional Neural Networks. In CVPR, 2016.
  10. Controlling Perceptual Factors in Neural Style Transfer. In CVPR, 2017.
  11. RecolorNeRF: Layer Decomposed Radiance Fields for Efficient Color Editing of 3D Scenes. In ACM MM, 2023.
  12. Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields. In ICCV AI3DCC, 2023.
  13. Learning to Stylize Novel Views. In CVPR, 2021.
  14. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. In CVPR, 2017.
  15. StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning. In CVPR, 2022.
  16. NeRFshop: Interactive Editing of Neural Radiance Fields. ACM CGIT, 6(1):1:1–1:21, 2023.
  17. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In ECCV, 2016.
  18. Segment Anything. In ICCV, 2023.
  19. Decomposing NeRF for Editing via Feature Field Distillation. In NeurIPS, 2022.
  20. PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields. In CVPR, 2023.
  21. AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields. In ECCV, 2022.
  22. Learning Blind Video Temporal Consistency. In ECCV, 2018.
  23. ICE-NeRF: Interactive Color Editing of NeRFs via Decomposition-Aware Weight Optimization. In ICCV, 2023.
  24. Language-driven Semantic Segmentation. In ICLR, 2022.
  25. StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields. In CVPR, 2023.
  26. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
  27. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM TOG, 38(4):29:1–29:14, 2019.
  28. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020.
  29. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM TOG, 41(4):102:1–102:15, 2022.
  30. DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. Comput. Graph. Forum, 40(4):45–59, 2021.
  31. SNeRF: Stylized Neural Implicit Representations for 3D Scenes. ACM TOG, 41(4):142:1–142:11, 2022.
  32. Locally Stylized Neural Radiance Fields. In ICCV, 2023.
  33. Floaters No More: Radiance Field Gradient Scaling for Improved Near-Camera Training. In EGSR, 2023.
  34. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In CVPR, 2022.
  35. Artistic Style Transfer for Videos. In GCPR, 2016.
  36. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR, 2014.
  37. Decomposing Images into Layers via RGB-Space Geometry. ACM TOG, 36(1):7:1–7:14, 2016.
  38. Efficient Palette-Based Decomposition and Recoloring of Images via RGBXY-Space Geometry. ACM TOG, 37(6):262:1–262:10, 2018.
  39. Block-NeRF: Scalable Large Scene Neural View Synthesis. In CVPR, 2022.
  40. Jinxiang Tang. Torch-ngp: a PyTorch implementation of instant-ngp, 2022. https://github.com/ashawkey/torch-ngp.
  41. Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement. In ICCV, 2022.
  42. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In ECCV, 2020.
  43. Recolorable Posterization of Volumetric Radiance Fields Using Visibility-Weighted Palette Extraction. Comput. Graph. Forum, 41(4):149–160, 2022.
  44. Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields. In CVPR, 2022.
  45. Eclipse: Disambiguating Illumination and Materials using Unintended Shadows. arXiv CoRR abs:2305.16321, 2023.
  46. NeRF-Art: Text-Driven Neural Radiance Fields Stylization. In IEEE TVCG, 2023a.
  47. InpaintNeRF360: Text-Guided 3D Inpainting on Unbounded Neural Radiance Fields. arXiv CoRR abs:2305.15094, 2023b.
  48. R. F. Woolson. Wilcoxon Signed-Rank Test, pages 1–3. John Wiley & Sons, Ltd, 2008.
  49. PaletteNeRF: Palette-based Color Editing for NeRFs. arXiv CoRR abs:2212.12871, 2022.
  50. IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis. In ICCV, 2023.
  51. ARF: Artistic Radiance Fields. In ECCV, 2022.
  52. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR, 2018.
  53. NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination. ACM TOG, 40(6):237:1–237:18, 2021.
  54. Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization. In CVPR, 2023a.
  55. Transforming Radiance Field with Lipschitz Network for Photorealistic 3D Scene Stylization. In CVPR, 2023b.
  56. EditableNeRF: Editing Topologically Varying Neural Radiance Fields by Key Points. In CVPR, 2023.
  57. Instant Volumetric Head Avatars. In CVPR, 2023.
Citations (2)
View on Semantic Scholar

Summary

  • The paper introduces LAENeRF, a framework that enables targeted local appearance edits in pre-trained Neural Radiance Fields with both photorealistic and stylized results.
  • It details a neural module that predicts editable color palettes via voxel-grid selection, facilitating rapid, memory-efficient recoloring while preserving 3D geometry.
  • Experimental results demonstrate that LAENeRF outperforms existing methods in processing time and editing quality, significantly reducing recoloring error rates.

Introduction to Local Appearance Editing

As digital rendering technologies progress, the manipulation and editing of 3D scenes become increasingly important. One of the primary tools for creating high-fidelity synthetic images is Neural Radiance Fields (NeRFs), which can faithfully reconstruct complex scenes from a set of input images. The work introduces LAENeRF, a novel approach to editing the localized appearance of objects within a pre-trained NeRF. This method allows both photorealistic and non-photorealistic alterations, such as recoloring and stylization, while maintaining the original fidelity of the scene and avoiding unwanted artifacts.

Challenges and Solutions in NeRF Editing

Editing NeRFs presents various challenges, including limited interactivity, lack of support for localized changes, and high memory consumption that result in impractical editing processes. LAENeRF addresses these challenges by enabling efficient local edits using a voxel grid that allows users to select specific content within a 3D scene for editing. It integrates a learning model that predicts an editable color palette based on ray termination points, facilitating rapid and memory-efficient optimizations.

Interactivity and Enhanced Editing Control

The LAENeRF’s architecture includes a neural module that is trained to predict color weights and offsets, which are composed into the output colors using a learned color palette. This setup allows users to alter colors interactively after the optimization process. The method incorporates novel losses, including a style loss, which guides the editing towards photorealistic or stylized outcomes while preserving 3D geometry and ensuring multi-view consistency.

Results and Contributions

The LAENeRF framework is thoroughly tested against concurrent methods and exhibits superior performance for local editing operations. Results show that LAENeRF outperforms other techniques in terms of processing time and quality of results, reducing error rates in recoloring by significant margins. The main contributions of this work include combining photorealistic and non-photorealistic edits for NeRFs, introducing the first interactive method for local and recolorable stylization, and proposing a unique architecture for efficient, 3D-aware stylization. The approach is a step towards practical, artist-friendly editing tools for complex 3D scenes.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. LAENeRF: Local Appearance Editing for Neural Radiance Fields