GaussianBody: Clothed Human Reconstruction via 3d Gaussian Splatting
(2401.09720)Abstract
In this work, we propose a novel clothed human reconstruction method called GaussianBody, based on 3D Gaussian Splatting. Compared with the costly neural radiance based models, 3D Gaussian Splatting has recently demonstrated great performance in terms of training time and rendering quality. However, applying the static 3D Gaussian Splatting model to the dynamic human reconstruction problem is non-trivial due to complicated non-rigid deformations and rich cloth details. To address these challenges, our method considers explicit pose-guided deformation to associate dynamic Gaussians across the canonical space and the observation space, introducing a physically-based prior with regularized transformations helps mitigate ambiguity between the two spaces. During the training process, we further propose a pose refinement strategy to update the pose regression for compensating the inaccurate initial estimation and a split-with-scale mechanism to enhance the density of regressed point clouds. The experiments validate that our method can achieve state-of-the-art photorealistic novel-view rendering results with high-quality details for dynamic clothed human bodies, along with explicit geometry reconstruction.
Overview
-
Introduction of GaussianBody, a method for reconstructing high-fidelity clothed human figures from images or videos.
-
Utilizes 3D Gaussian Splatting (3D-GS) to reduce training time and enhance rendering quality.
-
Overcomes challenges of non-rigid human motion and clothing details with pose-guided deformation and density enhancement strategies.
-
Includes a physically-based regularization and pose refinement for more accurate reconstructions.
-
Demonstrated superior performance and efficiency in rigorous testing compared to baselines and state-of-the-art methods.
Overview
Researchers have introduced a groundbreaking method for reconstructing high-fidelity clothed human figures from single images or monocular videos. This innovative technique, named GaussianBody, uses 3D Gaussian Splatting (3D-GS), a differentiable rendering approach that significantly reduces training time and renders with exceptional quality. Traditional techniques face challenges with complex non-rigid human motion and intricate clothing details; however, GaussianBody successfully addresses these by applying explicit pose-guided deformation, a physically-based regularization, and advanced strategies to refine pose estimation and enhance point cloud density.
The Problem Domain
Reconstructing realistic 3D models of clothed humans from images or videos is a crucial task with applications in various industries, from entertainment to virtual reality. Mesh-based and implicit methods have shown promising results, offering both speed and detailed representations. However, they struggle with dynamic motion and require extensive computational resources. By contrast, 3D-GS, previously applied to static models, has proven effective but needed adaptation for the dynamic nature of human motion. GaussiansBody represents an advancement in this space, offering both improved precision and efficiency.
The GaussianBody Approach
In the heart of GaussianBody lies the blend of 3D-GS and articulated human models. The method uses forward linear blend skinning (LBS) to deform 3D Gaussians from canonical to observation space, closely following the human pose throughout frames. A scalable strategy dynamically enhances the point cloud density, handling areas of sparseness that occur due to the lack of multi-view information. Moreover, pose refinement is introduced to correct any initial pose estimation inaccuracies. This results in high-quality, detailed renderings of dynamic clothed humans in near-real-time.
A Leap in Performance
GaussianBody has been tested and validated through rigorous experiments against baseline models and state-of-the-art methods. Noteworthy is its performance in areas that typically challenge reconstruction accuracy, such as the fidelity of clothing details and the recovery of geometry. By comparison, GaussianBody offers superior reconstruction quality while significantly reducing the required training time. It demonstrates its efficacy, particularly in rendering dynamic clothed humans, marking significant progress in the field of 3D human modeling.
Create an account to read this summary for free: