ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions (2401.10232v2)

Abstract: To enable machines to understand the way humans interact with the physical world in daily life, 3D interaction signals should be captured in natural settings, allowing people to engage with multiple objects in a range of sequential and casual manipulations. To achieve this goal, we introduce our ParaHome system designed to capture dynamic 3D movements of humans and objects within a common home environment. Our system features a multi-view setup with 70 synchronized RGB cameras, along with wearable motion capture devices including an IMU-based body suit and hand motion capture gloves. By leveraging the ParaHome system, we collect a new human-object interaction dataset, including 486 minutes of sequences across 207 captures with 38 participants, offering advancements with three key aspects: (1) capturing body motion and dexterous hand manipulation motion alongside multiple objects within a contextual home environment; (2) encompassing sequential and concurrent manipulations paired with text descriptions; and (3) including articulated objects with multiple parts represented by 3D parameterized models. We present detailed design justifications for our system, and perform key generative modeling experiments to demonstrate the potential of our dataset.

• The paper presents ParaHome, which captures both full-body and fine hand movements using synchronized multi-camera and motion capture devices.
• It introduces a comprehensive dataset that integrates dexterous human actions with dynamic object movements across natural home activities.
• The findings enable probabilistic generative modeling of nuanced human-object interactions, offering advances for robotics and augmented reality.

Overview of ParaHome

The research presented introduces a breakthrough system, ParaHome, for capturing and parameterizing 3D interactions between humans and objects within a home setting. The core of this system is a specialized setup combining 70 synchronized RGB cameras and wearable motion capture devices, which track both the gross movements of the body across a room and the fine dexterous movements of the hands.

Data Collection and Unique Features

ParaHome system's data collection has been extensive, with a particular focus on the authenticity and variety of human-object interactions. The dataset, which will be publicly available, stands out for its comprehensiveness by capturing 3D full-body and hand movements, movements of various objects, and their articulated parts within a real-world room setting. A summary of the key advancements offered by the dataset includes:

• Integration of dexterous human actions and object movements in a shared parameterized space.
• Capture of human interaction with multiple objects in an array of naturally occurring activities.
• Inclusion of objects with articulated parts, such as laptops and kitchen drawers, providing a new layer of interaction complexity.

Modeling Human-Object Interactions

ParaHome's goal extends beyond tracking to understanding and predicting human-object interactions (HOI). To facilitate this, the system and associated paper introduce a parameterized 3D space with human pose parameters and object parameters to capture the nuanced dynamics of these interactions. Moreover, the paper suggests probabilistic modeling approaches to predict or infer plausible configurations and dynamics from the data.

Implications and Future Directions

The innovative ParaHome system enables the deep paper of the causal and spatiotemporal relationships within human-object interactions. The resulting dataset not only provides significant improvements over existing datasets but also paves the way for future research in generative modeling of HOI. The researchers recognize the system's current limitations, such as the inability to use RGB videos for training models due to markers on suits, and plan enhancements, including more diverse environments and objects. This endeavor reflects the ongoing research commitment to understanding complex interactions in home environments that are crucial for advancements in robotics as well as virtual and augmented reality simulations.