Texture-GS: Disentangling the Geometry and Texture for 3D Gaussian Splatting Editing (2403.10050v1)

Abstract: 3D Gaussian splatting, emerging as a groundbreaking approach, has drawn increasing attention for its capabilities of high-fidelity reconstruction and real-time rendering. However, it couples the appearance and geometry of the scene within the Gaussian attributes, which hinders the flexibility of editing operations, such as texture swapping. To address this issue, we propose a novel approach, namely Texture-GS, to disentangle the appearance from the geometry by representing it as a 2D texture mapped onto the 3D surface, thereby facilitating appearance editing. Technically, the disentanglement is achieved by our proposed texture mapping module, which consists of a UV mapping MLP to learn the UV coordinates for the 3D Gaussian centers, a local Taylor expansion of the MLP to efficiently approximate the UV coordinates for the ray-Gaussian intersections, and a learnable texture to capture the fine-grained appearance. Extensive experiments on the DTU dataset demonstrate that our method not only facilitates high-fidelity appearance editing but also achieves real-time rendering on consumer-level devices, e.g. a single RTX 2080 Ti GPU.

• The paper introduces a novel Texture-GS method that disentangles 3D geometry from 2D texture to enable efficient appearance editing.
• It leverages a UV mapping MLP and local Taylor expansion to approximate texture coordinates for fast and flexible 3D Gaussian Splatting.
• The approach achieves real-time rendering and high-fidelity scene reconstruction with strong performance on the DTU dataset, expanding applications in virtual reality and media production.

Disentangling Geometry and Texture in 3D Gaussian Splatting for Flexible Scene Editing

Introduction to Texture-GS

3D Gaussian Splatting (3D-GS) has been gaining traction as a compelling method for high-fidelity scene reconstruction and real-time rendering. However, its application to appearance editing, such as texture swapping, has been limited due to the entanglement of scene geometry and appearance within its representation. Addressing this limitation, the newly proposed Texture-GS method innovatively decouples geometry from texture in 3D-GS, significantly enhancing flexibility in appearance editing tasks. By representing appearance as a 2D texture map and employing a novel texture mapping module, Texture-GS maintains the advantages of 3D-GS while adding the ability to perform efficient and high-quality appearance modifications.

Texture Mapping Module

The core of Texture-GS lies in its texture mapping module, which achieves the disentanglement through several key components. This module introduces:

• A UV mapping MLP to determine UV coordinates for 3D Gaussian centers.
• A local Taylor expansion of the MLP, facilitating efficient approximation of UV coordinates for ray-Gaussian intersections during rendering.
• A learnable texture, representing the fine-grained appearance details.

This formulation effectively enables real-time rendering while also supporting complex appearance editing operations such as global texture swapping and fine-grained texture editing, showing strong performance on the DTU dataset.

Practical Implications and Advancements

Texture-GS offers several practical advancements in the field of neural rendering and scene editing:

• Editing Flexibility: It allows for seamless and efficient appearance changes, significantly expanding the use cases of 3D-GS in media, virtual reality, and game development.
• Real-Time Performance: Despite the added complexity of its disentangled representation, Texture-GS achieves real-time rendering speeds on consumer-grade hardware, ensuring its applicability in interactive applications.
• High-Fidelity Reconstruction: The method demonstrates an ability to recover detailed and high-quality 2D textures from multi-view images, facilitating various editing applications without compromising on visual quality.

Future Directions and Potential

The introduction of Texture-GS opens up new avenues for research and development in 3D scene editing and neural rendering. Potential future work could explore:

• Extending the method to support dynamic scenes and editing operations, enhancing its applicability in interactive and immersive experiences.
• Investigating the integration of Texture-GS with other deep learning approaches for improved scene understanding and manipulation capabilities.
• Exploring the trade-offs between rendering speed and visual fidelity in more complex or larger-scale scenes, aiming to optimize performance for specific application requirements.

Concluding Remarks

Texture-GS represents a significant step forward in the disentanglement of geometry and texture within the domain of 3D Gaussian Splatting, offering a robust solution for efficient and flexible scene editing. Its ability to combine real-time rendering capabilities with high-quality appearance modifications holds promise for a wide range of applications in computational photography, virtual reality, and digital media production.