LAGA: Layered 3D Avatar Generation and Customization via Gaussian Splatting (2405.12663v2)

Abstract: Creating and customizing a 3D clothed avatar from textual descriptions is a critical and challenging task. Traditional methods often treat the human body and clothing as inseparable, limiting users' ability to freely mix and match garments. In response to this limitation, we present LAyered Gaussian Avatar (LAGA), a carefully designed framework enabling the creation of high-fidelity decomposable avatars with diverse garments. By decoupling garments from avatar, our framework empowers users to conviniently edit avatars at the garment level. Our approach begins by modeling the avatar using a set of Gaussian points organized in a layered structure, where each layer corresponds to a specific garment or the human body itself. To generate high-quality garments for each layer, we introduce a coarse-to-fine strategy for diverse garment generation and a novel dual-SDS loss function to maintain coherence between the generated garments and avatar components, including the human body and other garments. Moreover, we introduce three regularization losses to guide the movement of Gaussians for garment transfer, allowing garments to be freely transferred to various avatars. Extensive experimentation demonstrates that our approach surpasses existing methods in the generation of 3D clothed humans.

• The paper presents a layered framework that decouples the human body and garments, allowing independent avatar customization.
• It employs a coarse-to-fine strategy that refines details from sparse initialization to capture high-quality avatars.
• The paper introduces novel dual-SDS and regularization loss functions to ensure visual coherence and effective garment transfer.

Summary of LAGA: Layered 3D Avatar Generation and Customization via Gaussian Splatting

The paper presents a novel approach to 3D avatar generation, namely LAyered Gaussian Avatar (LAGA), which leverages a layered representation framework using Gaussian Splatting (GS) for creating and customizing 3D clothed avatars. The primary objective is to address the limitations of traditional avatar generation methods where clothing and the human body are inseparable, therefore limiting customization options.

The authors propose a multi-layered system where each layer corresponds to either the human body or individual garments. This enables the decoupling of the clothing from the avatar, allowing users to edit and customize clothing independently. Such a decomposable system is beneficial in fields where avatar personalization is crucial, such as gaming and virtual reality.

Key Contributions

1. Layered Avatar Generation Framework: The authors introduce a framework that models avatars using layers of Gaussian points, each layer representing either the human body or garments. This structure supports customization by allowing individual garment modification and transfer.
2. Coarse-to-Fine Strategy: To generate high-quality, diverse garments, the method employs a two-stage coarse-to-fine approach. It starts with a sparse initialization and evolves the avatar to capture fine details.
3. Dual-SDS Loss Function: The framework introduces a novel dual-SDS loss function designed to maintain coherence between the garments and the avatar components. This function helps in generating avatars that are visually consistent and natural.
4. Regularization Losses for Garment Transfer: To facilitate garment transfer between avatars of different shapes, the method incorporates three regularization losses—Human Fitting Loss, Similarity Loss, and Visibility Loss—that guide the motion of Gaussian points.

Experimental Results

The paper reports extensive experimental validation demonstrating the superior quality of avatars generated with LAGA compared to state-of-the-art methods. The quantitative measures include CLIP Score and Fréchet Inception Distance (FID), both showing enhanced performance. The user paper further confirms the improved realism and texture coherence achieved by LAGA.

Practical and Theoretical Implications

The LAGA framework offers a significant step forward in avatar customization, providing a more flexible system that can potentially be expanded and enhanced for various applications in entertainment and virtual interactions. The ability to transfer garments between avatars of different body shapes without degradation in quality presents notable practical implications for dynamic content generation and interactive environments.

From a theoretical perspective, the application of Gaussian Splatting to layered 3D modeling opens new avenues for efficient 3D representation and rendering, suggesting potential future developments in reducing computational costs and improving the scalability of avatar modeling systems.

Future Prospects

In terms of future research, the exploration of this layered approach coupled with GS could lead to advancements in real-time avatar rendering and advancements in interactive gaming. Additionally, the authors' framework might inspire further research into more sophisticated customization options, potentially integrating more complex garment physics and color variations directly influenced by user interactions.

Overall, the LAGA framework represents a promising direction in the generation and customization of 3D avatars, offering a flexible and comprehensive approach to avatar personalization and interaction.