Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input (1610.04889v1)

Abstract: Real-time simultaneous tracking of hands manipulating and interacting with external objects has many potential applications in augmented reality, tangible computing, and wearable computing. However, due to difficult occlusions, fast motions, and uniform hand appearance, jointly tracking hand and object pose is more challenging than tracking either of the two separately. Many previous approaches resort to complex multi-camera setups to remedy the occlusion problem and often employ expensive segmentation and optimization steps which makes real-time tracking impossible. In this paper, we propose a real-time solution that uses a single commodity RGB-D camera. The core of our approach is a 3D articulated Gaussian mixture alignment strategy tailored to hand-object tracking that allows fast pose optimization. The alignment energy uses novel regularizers to address occlusions and hand-object contacts. For added robustness, we guide the optimization with discriminative part classification of the hand and segmentation of the object. We conducted extensive experiments on several existing datasets and introduce a new annotated hand-object dataset. Quantitative and qualitative results show the key advantages of our method: speed, accuracy, and robustness.

• The paper introduces a novel real-time method using 3D Gaussian mixture alignment for joint hand-object tracking from RGB-D input.
• It employs dual-proposal optimization and a two-layer random forest for hand part classification to enhance robustness against occlusions and rapid motions.
• Empirical evaluations on a new annotated dataset demonstrate 30Hz performance and high precision, promising interactive application potential.

Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input

The paper under review presents a sophisticated method for real-time simultaneous tracking of a hand and an object with the use of a single commodity RGB-D camera. This research addresses the complexity inherent in jointly tracking hand and object poses, which includes challenges such as occlusions, fast motions, and uniform hand appearances. Prior methodologies predominantly relied on multi-camera configurations or computationally expensive steps that hindered the feasibility of real-time interaction.

Methodology

The approach is founded on a 3D articulated Gaussian mixture alignment strategy tailored specifically for hand-object interaction scenarios. This enables efficient pose optimization through alignment energies and novel regularizers which accommodate occlusions and hand-object contacts. The optimization process is further fortified by discriminative part classification of the hand and object segmentation. The methodology proposed delivers enhanced robustness through this dual-guided optimization, simultaneously tracking hand and object efficiently.

The core components of the system are:

1. Gaussian Mixture Model Representation: The human hand's motion is parameterized using a kinematic skeleton, articulated across approximately 26 degrees of freedom, allowing for detailed motion capture. The object handled is considered rigid, represented by an automatically fitted set of Gaussian mixtures to its geometry.
2. Multiple Proposal Optimization: The system employs two distinct hand-object tracking energies to compute concurrent proposals. Such an approach aids in reaching a robust estimate by evaluating two potential solutions and selecting the more optimal one.
3. Discriminative Hand Part Classification: Part classification employs a two-layer random forest model that segments the depth map into hand and object components and refines hand parts further. It adapts to the view of the hand, enhancing accuracy and reliability in classification.
4. Tracking Objectives and Energies: The tracking framework introduces energies that consider spatial and semantic alignment, anatomical plausibility, temporal smoothness, contact points, and occlusion handling. These intricacies help maintain the tracking's fidelity and robustness despite challenging interactions.

Results and Contributions

The empirical results emphasize the method's speed, accuracy, and robustness, benchmarking against existing datasets and introducing a new one for comprehensive evaluation. The quantitative analyses illustrate that the approach achieves a 30Hz frame rate, ensuring real-time performance with substantial precision in joint and object positioning. The new dataset provides annotated hand-object interactions, enriching the potential for future research comparison.

Limitations and Future Directions

While the paper demonstrates success in tracking different object sizes, shapes, and hand movements, the presented method faces constraints under prolonged occlusions or rapid motions. Such challenges hint at the need for further research into more sophisticated occlusion handling and potential integration of higher frame-rate sensors, which could improve temporal coherence and mitigate tracking errors.

Additionally, augmenting the system to manage multiple objects and more intricate interactions could expand the method's applicability, especially in complex augmented reality setups or intricate industrial applications.

Conclusion

This work introduces a significant advancement in the real-time joint tracking of hands and objects with simple hardware, enhancing potential applications in augmented and tangible computing. By combining discriminative classification with 3D articulated tracking informed by innovative computational energies, this method shows impactful results, paving the way for broader deployment in user-interactive environments. Exploring more profound learning-based methods and enhancing occlusion modeling could further propel this domain toward achieving seamless and comprehensive interactive tracking solutions.