ClotheDreamer: Text-Guided Garment Generation with 3D Gaussians
(2406.16815)Abstract
High-fidelity 3D garment synthesis from text is desirable yet challenging for digital avatar creation. Recent diffusion-based approaches via Score Distillation Sampling (SDS) have enabled new possibilities but either intricately couple with human body or struggle to reuse. We introduce ClotheDreamer, a 3D Gaussian-based method for generating wearable, production-ready 3D garment assets from text prompts. We propose a novel representation Disentangled Clothe Gaussian Splatting (DCGS) to enable separate optimization. DCGS represents clothed avatar as one Gaussian model but freezes body Gaussian splats. To enhance quality and completeness, we incorporate bidirectional SDS to supervise clothed avatar and garment RGBD renderings respectively with pose conditions and propose a new pruning strategy for loose clothing. Our approach can also support custom clothing templates as input. Benefiting from our design, the synthetic 3D garment can be easily applied to virtual try-on and support physically accurate animation. Extensive experiments showcase our method's superior and competitive performance. Our project page is at https://ggxxii.github.io/clothedreamer.
Overview
-
ClotheDreamer introduces a framework for generating high-fidelity 3D garment assets guided by textual descriptions, utilizing Disentangled Clothe Gaussian Splatting (DCGS) to optimize garment and body components separately.
-
The methodology involves Bidirectional Score Distillation Sampling (SDS) for detailed geometry and texture control, a novel pruning strategy for preserving loose garment geometry, and template-guided garment synthesis for personalized designs.
-
Experimental results demonstrate ClotheDreamer's superiority over state-of-the-art methods, with high alignment to textual descriptions and preferences from user studies; future directions include refining decoupling processes and mitigating color oversaturation issues.
ClotheDreamer: Text-Guided Garment Generation with 3D Gaussians
The paper "ClotheDreamer: Text-Guided Garment Generation with 3D Gaussians" proposes a framework for generating high-fidelity, production-ready 3D garment assets guided by textual descriptions. This innovative approach leverages the concept of Disentangled Clothe Gaussian Splatting (DCGS) to independently optimize garment and body components, offering a significant leap forward in the field of text-driven 3D garment synthesis.
Methodology
The authors introduce the concept of DCGS, which represents a clothed avatar as a single Gaussian model, while freezing the body Gaussian splats for separate optimization. This novel representation is crucial for creating versatile, reusable garment assets that are disentangled from the body model. The initialization process includes leveraging SMPL-X segmentation parts, categorized into six common groups, to select the clothing points relative to text instructions. This ID-based initialization significantly enhances the control over garment generation, ensuring that the garments align closely with the human body while remaining separate entities.
A pivotal part of this methodology is the Bidirectional Score Distillation Sampling (SDS) guidance. This dual supervision mechanism, applied through pre-trained diffusion models, provides robust control over the detailed geometry and texture of the clothing. The framework accommodates both clothing and body renderings by parsing Gaussian Splatting (GS) renderings, which are then subjected to SDS loss computations.
Additionally, the paper addresses the challenge of loose clothing with a new pruning strategy that maintains the integrity of the garments. Conventional strategies often result in the mistaken elimination of useful Gaussian points, particularly for loose clothing like long dresses and gowns. The proposed method prunes Gaussian splats during the middle training stages with an increased scaling factor range, preserving the completeness of the garment geometry.
The authors also present a technique for template-guided garment generation. By using custom template meshes, the method allows for personalized garment synthesis, providing more control over the overall garment style without constraining intricate details. This versatility extends the practical applications of the method.
For animation, the DCGS garments offer two primary approaches: animation using SMPL-X pose sequences for tight garments, and mesh-based garment animation for loose clothing. The latter involves using simulated mesh sequences to drive the transformation of Gaussian points, facilitated by ICP registration and KD-tree binding.
Experimental Results
The experimental evaluation showcases the superiority of ClotheDreamer over state-of-the-art methods, including Shap-E, ProlificDreamer, LGM, GaussianDreamer, and DreamGaussian. The quantitative assessment using CLIP scores indicates a markedly higher alignment with textual descriptions, and a user study further corroborates the method's appeal, with participants favoring ClotheDreamer for overall quality and textual consistency.
Ablation studies underscore the efficacy of Bidirectional SDS guidance in reducing artifacts and enhancing the quality of the generated garments. Further, the new pruning strategy for loose clothing demonstrates clear advantages in maintaining the garment's overall geometry and texture.
Implications and Future Directions
ClotheDreamer offers broad implications for the fields of virtual try-on, fashion design, and digital avatar creation. The ability to generate detailed, production-ready 3D garments from textual prompts opens new avenues for creative and commercial applications. The prospect of dynamic animation further enhances the usability of generated assets in interactive and immersive environments.
Future work could address some of the limitations identified, such as refining the decoupling process for more complex scenarios involving multiple garment components, and mitigating color oversaturation issues common to SDS-based techniques. Improvements in SDS formulations and lighting disentanglement for 3D Gaussian representations could also enhance the realism and versatility of this approach.
In summary, ClotheDreamer presents a robust, adaptable framework for text-guided 3D garment generation, establishing a new benchmark in the domain with its novel representation and dual-guidance methodologies. The paper’s contributions lay a strong foundation for further innovation and practical advancements in digital fashion and avatar technologies.
Create an account to read this summary for free: