Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation (2106.05969v3)

Abstract: We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinematics or dynamics-based approaches where the two components are used disjointly, we synergize the two approaches via dynamics-regulated training. At each timestep, a kinematic model is used to provide a target pose using video evidence and simulation state. Then, a prelearned dynamics model attempts to mimic the kinematic pose in a physics simulator. By comparing the pose instructed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model. By factoring in the 6DoF pose of objects (e.g., chairs, boxes) in the scene, we demonstrate for the first time, the ability to estimate physically-plausible 3D human-object interactions using a single wearable camera. We evaluate our egocentric pose estimation method in both controlled laboratory settings and real-world scenarios.

• The paper introduces a hybrid approach that combines kinematic predictions with dynamic corrections for physically plausible 3D pose and interaction estimation.
• It leverages a Universal Humanoid Controller and dynamics-regulated training to significantly reduce pose estimation errors across controlled and real-world datasets.
• Incorporating object-aware kinematics, the method enhances human-object interaction recognition, paving the way for advanced VR and AR applications.

Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation

This paper presents a novel approach to egocentric pose estimation, addressing the inadequacies of purely kinematic or dynamics-based methodologies by integrating both aspects into a cohesive dynamics-regulated training framework. The task here is to infer 3D human pose and interactions from videos recorded by wearable, head-mounted cameras, a problem compounded by the lack of direct visibility into the wearer's body and dynamic interactions with objects within the scene.

Key Contributions and Methodology

The paper introduces several key contributions:

1. Hybrid Modeling of Kinematics and Dynamics: By synergizing kinematic and dynamics models, the authors aim to estimate physically plausible 3D poses and interactions. The kinematic model provides initial pose predictions from the video, while the dynamics model refines these predictions by enforcing physical plausibility through a physics simulator. This dual approach mitigates the limitations inherent in using either method alone.
2. Universal Humanoid Controller (UHC): Leveraging a large database of human motion data, the UHC is a dynamics-based model that generalizes well across a wide range of human behaviors within a physics-based framework. This controller serves as a foundation, mimicking nuanced human activities such as dance or sports, and adapting easily to egocentric objectives.
3. Dynamics-Regulated Training: This distinctive training scheme integrates the kinematic estimations with physical insights provided by the dynamics model, optimizing the system through both supervised learning and reinforcement learning techniques. This operational stratagem not only enhances the robustness of pose estimations in real-world settings but also introduces anisotropic influences from motion capture data into the kinematic policy.
4. Object-aware Kinematic Policy: The paper proposes to account for the interactions between humans and objects in the scene by incorporating the six degrees of freedom (6DoF) object pose into the kinematic modeling. This object-awareness allows the system to accurately evaluate human-object interactions using purely egocentric video data.

Experimental Evaluations

The proposed approach is evaluated using two datasets, one recorded in a controlled motion capture environment and another in the wild, capturing naturalistic human-object interactions. The model's performance is benchmarked against existing methods like EgoPose and PoseReg, showing superior performance across both pose-based and physics-based metrics.

Numerical Results:

• In controlled environments, the dynamics-regulated method reduced root error (Eroot) and mean per joint position error (Empipe) significantly compared to competitive baselines.
• In real-world applications, demonstrated an interaction success rate (Sinter) of 93.4%, a marked improvement in accuracy over other methods.

Implications and Future Directions

Practically, this method advances the application of virtual and augmented reality systems, where understanding complex human-object interactions through wearable devices can enhance interaction realism. Theoretically, it enriches the domain of hybrid kinematic-dynamic modeling, proposing an architecture capable of bridging the gap between static pose estimation and dynamic physical interaction modeling.

The research sets the stage for several future explorations:

• Expanding the repertoire of recognizable actions by introducing learned motion priors for more complex activities.
• Extending the current dynamics-regulated framework to incorporate real-time third-person pose estimation tasks.
• Investigating deeper into handling larger domain shifts inherent in real-world environments to enhance adaptive performance.

Concluding Remarks

This paper makes substantial progress in the domain of egocentric pose estimation by innovatively harmonizing kinematic predictions with dynamic corrections within a physics simulation environment. The findings underscore the potential for future AI systems to reliably interpret human pose and interaction through consumer-grade devices, albeit with careful consideration of privacy and ethical ramifications.