In recent times, the generation of 3D assets from text prompts has shown impressive results. Both 2D and 3D diffusion models can help generate decent 3D objects based on prompts. 3D diffusion models have good 3D consistency, but their quality and generalization are limited as trainable 3D data is expensive and hard to obtain. 2D diffusion models enjoy strong abilities of generalization and fine generation, but 3D consistency is hard to guarantee. This paper attempts to bridge the power from the two types of diffusion models via the recent explicit and efficient 3D Gaussian splatting representation. A fast 3D object generation framework, named as GaussianDreamer, is proposed, where the 3D diffusion model provides priors for initialization and the 2D diffusion model enriches the geometry and appearance. Operations of noisy point growing and color perturbation are introduced to enhance the initialized Gaussians. Our GaussianDreamer can generate a high-quality 3D instance or 3D avatar within 15 minutes on one GPU, much faster than previous methods, while the generated instances can be directly rendered in real time. Demos and code are available at https://taoranyi.com/gaussiandreamer/.
GaussianDreamer is a new model for quickly creating high-quality 3D assets from text.
It combines 2D and 3D diffusion models to leverage the detail and consistency of both dimensions.
The model begins with a basic 3D structure from a 3D diffusion model which is then detailed by a 2D diffusion model.
Training on a single GPU can be completed in just 15 minutes, allowing for rapid content creation.
This method benefits industries like gaming and virtual reality, where real-time rendering of detailed 3D objects is required.
The paper introduces GaussianDreamer, a novel framework designed to efficiently produce high-quality 3D assets from textual prompts. The process marries the strengths of 3D and 2D diffusion models, utilizing a recent efficient representation known as 3D Gaussian Splatting. This innovative approach enables the rapid generation of 3D objects with rich details and consistency while offering the capability of real-time rendering.
To capitalize on the distinct advantages of 2D and 3D diffusion models — the former's detail richness and the latter's three-dimensional consistency — GaussianDreamer employs a recent advancement called 3D Gaussian Splatting. This method uses 3D diffusion models to provide a basic geometric form as a starting point, and then enriches that form with details via a 2D diffusion model. The integration of both models mitigates the limitations of working exclusively in either dimension and accelerates the training process considerably as compared to techniques that use 3D training data alone.
GaussianDreamer operates in two major steps:
Additional operations such as noisy point growing and color perturbation are applied to enhance the initial geometric structure. The process is notably swift, allowing for the completion of the training within 15 minutes on a single GPU.
The achievement of merging 3D and 2D diffusion model capabilities is significant. Not only does this lead to faster generation times and high-quality output, but it also allows for real-time rendering which is a considerable step forward in the field. The method has practical implications in various industries such as gaming, virtual reality, and film, where speed and quality of 3D asset generation are crucial.
Moreover, the authors claim that the method can be adapted to a wide array of prompts, showing versatility and the potential to generate a broad range of detailed 3D models. Consequently, GaussianDreamer stands out as a user-friendly and powerful tool for rapid 3D content creation.
In summary, GaussianDreamer showcases a leap in 3D asset generation technology by swiftly producing realistic 3D models that blend coherence and detail, ultimately benefiting various industries reliant on 3D content.