SPEC: Seeing People in the Wild with an Estimated Camera (2110.00620v2)

Abstract: Due to the lack of camera parameter information for in-the-wild images, existing 3D human pose and shape (HPS) estimation methods make several simplifying assumptions: weak-perspective projection, large constant focal length, and zero camera rotation. These assumptions often do not hold and we show, quantitatively and qualitatively, that they cause errors in the reconstructed 3D shape and pose. To address this, we introduce SPEC, the first in-the-wild 3D HPS method that estimates the perspective camera from a single image and employs this to reconstruct 3D human bodies more accurately. First, we train a neural network to estimate the field of view, camera pitch, and roll given an input image. We employ novel losses that improve the calibration accuracy over previous work. We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose. SPEC is more accurate than the prior art on the standard benchmark (3DPW) as well as two new datasets with more challenging camera views and varying focal lengths. Specifically, we create a new photorealistic synthetic dataset (SPEC-SYN) with ground truth 3D bodies and a novel in-the-wild dataset (SPEC-MTP) with calibration and high-quality reference bodies. Both qualitative and quantitative analysis confirm that knowing camera parameters during inference regresses better human bodies. Code and datasets are available for research purposes at https://spec.is.tue.mpg.de.

Understanding SPEC: An Approach for 3D Human Pose Estimation with Perspective Camera

The paper "SPEC: Seeing People in the Wild with an Estimated Camera" proposes a novel framework for estimating 3D human pose and shape from single images captured in-the-wild. This is accomplished by taking into account the complexities of perspective projection, which have been traditionally overlooked in favor of simpler assumptions such as weak-perspective projection models. The paper presents a detailed methodology for the integration of perspective camera estimation into the human pose and shape (HPS) estimation process, offering improvements over existing methods in both accuracy and applicability to real-world scenarios.

Problem Definition and Approach

The challenge addressed in the paper arises from the common assumptions made in state-of-the-art HPS estimation techniques that rely on simplified camera models. These models frequently employ weak-perspective projection, assuming constant large focal lengths and no camera rotation, which are rarely valid in real-world conditions. This paper highlights the limitations of such assumptions through quantitative and qualitative analysis and proposes SPEC, a model that estimates perspective camera parameters directly from an RGB image and utilizes them in regressing accurate 3D human body shapes and poses.

The framework developed comprises two core components:

1. CamCalib Network: This network is trained to estimate the camera's vertical field of view (vfov), pitch, and roll from an input image. Novel loss functions are introduced to improve the regression accuracy of these parameters, emphasizing the differential impact of over- and underestimation of the vfov.
2. SPEC Network: By integrating the estimated camera parameters into the body reconstruction process, the SPEC network applies this additional information to refine the estimated 3D pose and shape. This is achieved through conditioning on camera parameters in both optimization- and regression-based approaches.

Implementation Details and Evaluation

The authors have conducted extensive experiments on both synthetic and real-world datasets. Notably, they created a new synthetic dataset and an in-the-wild dataset with varying camera views and focal lengths to benchmark the performance of their model. The SPEC framework, coupled with CamCalib, demonstrates a higher accuracy than existing techniques, especially in datasets with challenging camera conditions that include foreshortening effects and diverse focal lengths.

In empirical evaluations, performance improvements are especially pronounced in settings where the perspective effects deviate from the assumptions made in weak-perspective models. The proposed metric, World-MPJPE, which measures errors in world coordinates rather than camera space, underscores SPEC's capability to accurately position and orient 3D poses in a global context, reducing dependency on Procrustes alignment.

Contributions and Implications

The paper's contributions lie in the following areas:

• Development of a framework that incorporates realistic camera modeling into 3D human pose and shape estimation.
• Introduction of novel loss functions to improve camera parameter estimation accuracy.
• Creation of datasets that allow for a comprehensive evaluation of perspective-aware 3D HPS methodologies.
• Demonstration of improved accuracy and robustness across various camera conditions and datasets, expanding the applicability of HPS estimation techniques in real-life applications including robotics, AR/VR, and computer graphics.

Theoretical implications suggest a significant shift in the approach towards camera-aware modeling in 3D vision tasks, particularly in conditions that break common orthographic assumptions. Future research directions could explore the integration of this approach into multi-view systems, real-time applications, and further refinement of the network architecture to leverage camera information even more effectively. By aligning camera estimation with human pose regression, this work lays essential groundwork for more precise interpretations of visual data in automated systems.