AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text (2311.17917v1)
Abstract: We study the problem of creating high-fidelity and animatable 3D avatars from only textual descriptions. Existing text-to-avatar methods are either limited to static avatars which cannot be animated or struggle to generate animatable avatars with promising quality and precise pose control. To address these limitations, we propose AvatarStudio, a coarse-to-fine generative model that generates explicit textured 3D meshes for animatable human avatars. Specifically, AvatarStudio begins with a low-resolution NeRF-based representation for coarse generation, followed by incorporating SMPL-guided articulation into the explicit mesh representation to support avatar animation and high resolution rendering. To ensure view consistency and pose controllability of the resulting avatars, we introduce a 2D diffusion model conditioned on DensePose for Score Distillation Sampling supervision. By effectively leveraging the synergy between the articulated mesh representation and the DensePose-conditional diffusion model, AvatarStudio can create high-quality avatars from text that are ready for animation, significantly outperforming previous methods. Moreover, it is competent for many applications, e.g., multimodal avatar animations and style-guided avatar creation. For more results, please refer to our project page: http://jeff95.me/projects/avatarstudio.html
- Metahuman. https://www.unrealengine.com/en-US/metahuman, 2023.
- Torchmetrics. https://torchmetrics.readthedocs.io/en/stable/multimodal/clip_score.html, 2023.
- imghum: Implicit generative models of 3d human shape and articulated pose. In ICCV, 2021.
- Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models. ArXiv, 2023.
- Implicit fairing of irregular meshes using diffusion and curvature flow. In Seminal Graphics Papers: Pushing the Boundaries. 2023.
- Torchmetrics-measuring reproducibility in pytorch. Journal of Open Source Software, 2022.
- Densepose: Dense human pose estimation in the wild. In CVPR, 2018.
- threestudio: A unified framework for 3d content generation. https://github.com/threestudio-project/threestudio, 2023.
- Eva3d: Compositional 3d human generation from 2d image collections. ArXiv, 2022a.
- Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. ACM Trans. on Graphics, 2022b.
- Dreamwaltz: Make a scene with complex 3d animatable avatars. ArXiv, 2023.
- Zero-shot text-guided object generation with dream fields. CVPR, 2021.
- Avatarcraft: Transforming text into neural human avatars with parameterized shape and pose control. ArXiv, 2023.
- 3d gaussian splatting for real-time radiance field rendering. ACM Trans. on Graphics, 2023.
- Clip-mesh: Generating textured meshes from text using pretrained image-text models. ACM Transactions on Graphics, 2022.
- Adam: A method for stochastic optimization. In ICLR, 2015.
- Pare: Part attention regressor for 3d human body estimation. In ICCV, 2021.
- Dreamhuman: Animatable 3d avatars from text. ArXiv, 2023.
- Modular primitives for high-performance differentiable rendering. ACM Trans. on Graphics, 2020.
- Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In CVPR, 2021.
- Tada! text to animatable digital avatars. ArXiv, 2023.
- Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023a.
- Common diffusion noise schedules and sample steps are flawed. ArXiv, 2023b.
- Smpl: A skinned multi-person linear model. ACM Trans. on Graphics, 2015.
- Marching cubes: A high resolution 3d surface construction algorithm. 1998.
- Latent-nerf for shape-guided generation of 3d shapes and textures. ArXiv, 2022.
- Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
- Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. on Graphics, 2022.
- Laplacian mesh optimization. In CGIT, 2006.
- Expressive body capture: 3d hands, face, and body from a single image. In CVPR, 2019.
- Dreamfusion: Text-to-3d using 2d diffusion. In ICCV, 2022.
- Learning transferable visual models from natural language supervision. In ICML, 2021.
- Texture: Text-guided texturing of 3d shapes. ArXiv, 2023.
- High-resolution image synthesis with latent diffusion models, 2021.
- Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, 2022.
- Clip-forge: Towards zero-shot text-to-shape generation. CVPR, 2021.
- Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
- Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. In NeurIPS, 2022.
- Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In NeurIPS, 2021.
- Mvdream: Multi-view diffusion for 3d generation. ArXiv, 2023.
- Human motion diffusion model. In ICLR, 2023.
- Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. ArXiv, 2023.
- Xagen: 3d expressive human avatars generation. In NeurIPS, 2023.
- Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. ArXiv, 2023.
- Geometry-consistent neural shape representation with implicit displacement fields. ArXiv, 2021.
- Avatarverse: High-quality & stable 3d avatar creation from text and pose. ArXiv, 2023a.
- Avatargen: a 3d generative model for animatable human avatars. ArXiv, 2022.
- Adding conditional control to text-to-image diffusion models. ICCV, 2023b.
- Getavatar: Generative textured meshes for animatable human avatars. In ICCV, 2023c.