Learning to Detect Human-Object Interactions (1702.05448v2)

Abstract: We study the problem of detecting human-object interactions (HOI) in static images, defined as predicting a human and an object bounding box with an interaction class label that connects them. HOI detection is a fundamental problem in computer vision as it provides semantic information about the interactions among the detected objects. We introduce HICO-DET, a new large benchmark for HOI detection, by augmenting the current HICO classification benchmark with instance annotations. To solve the task, we propose Human-Object Region-based Convolutional Neural Networks (HO-RCNN). At the core of our HO-RCNN is the Interaction Pattern, a novel DNN input that characterizes the spatial relations between two bounding boxes. Experiments on HICO-DET demonstrate that our HO-RCNN, by exploiting human-object spatial relations through Interaction Patterns, significantly improves the performance of HOI detection over baseline approaches.

• The paper presents the HICO-DET benchmark and HO-RCNN framework, significantly enhancing the detection of human-object interactions.
• The methodology employs a multi-stream CNN architecture that integrates Interaction Patterns to capture spatial relationships between humans and objects.
• Extensive experiments show that modeling spatial configurations yields improved performance, setting a new benchmark for HOI detection.

Overview of "Learning to Detect Human-Object Interactions"

The paper "Learning to Detect Human-Object Interactions" by Yu-Wei Chao et al. addresses the foundational problem of detecting human-object interactions in static images, an important task within the field of computer vision. To this end, the authors introduce a new benchmark, HICO-DET, which expands the existing HICO classification dataset with instance-level annotations specifically for HOI detection. Alongside the dataset, the authors propose a novel detection framework named Human-Object Region-based Convolutional Neural Networks (HO-RCNN).

Key Contributions

1. HICO-DET Benchmark: By complementing the HICO classification dataset with instance annotations, HICO-DET emerges as a comprehensive benchmark for HOI detection. It includes over 150,000 annotated human-object pairs across 600 interaction categories, providing a rich resource for evaluating HOI detection methods.
2. HO-RCNN Framework: The proposed HO-RCNN extends conventional region-based object detectors to simultaneously detect human and object bounding boxes, along with the interaction labels connecting them. At its core, the HO-RCNN incorporates Interaction Patterns, which are novel deep neural network (DNN) inputs that encode the spatial relationships between humans and objects.
3. Experiments and Results: Extensive experiments on the HICO-DET benchmark demonstrate that the proposed HO-RCNN significantly enhances HOI detection performance compared to existing baseline methods. The results confirm the utility of encoding spatial relationships between humans and objects for accurate interaction detection.

Technical Approach

The HO-RCNN leverages a multi-stream architecture:

• Human and Object Streams: These streams independently extract local features from the human and object bounding boxes identified in an image. Each stream contributes towards identifying HOIs by processing local spatial and contextual information using convolutional neural networks (CNNs).
• Pairwise Stream with Interaction Patterns: The pairwise stream processes the spatial configuration of human-object pairs through Interaction Patterns, which represent the relative positioning of bounding boxes. The authors explore different configurations of Interaction Patterns and their impact on detection performance. Notably, the network architectures considered include fully connected and convolutional variants, with the latter showing superior results in aligning with spatial complexities.

Implications and Future Directions

This paper advances the state-of-the-art in HOI detection by demonstrating the feasibility and effectiveness of integrating spatial relational modeling through deep learning. The introduction of HICO-DET sets a new standard for evaluating HOI detection models, encouraging further research in this direction.

In the future, one could expect advancements in object relationship reasoning and possibly the extension of these methods to video-based interactions, which would inherently involve temporal dynamics. Moreover, developing techniques to balance computational feasibility with the scalability of models to handle an increasing number of interactions and complex scenes remains a potential area for development.

Concluding Remarks

This paper paves the way for a more nuanced understanding and detection of visual semantics through human-object interactions. By establishing a new benchmark and proposing a robust detection framework, the authors contribute significantly to the synergistic integration of object detection and relational inference within static image contexts. The results encourage further exploration of spatial relationship modeling in complex visual tasks.