Semantic-Based Active Perception for Humanoid Visual Tasks with Foveal Sensors

Abstract: The aim of this work is to establish how accurately a recent semantic-based foveal active perception model is able to complete visual tasks that are regularly performed by humans, namely, scene exploration and visual search. This model exploits the ability of current object detectors to localize and classify a large number of object classes and to update a semantic description of a scene across multiple fixations. It has been used previously in scene exploration tasks. In this paper, we revisit the model and extend its application to visual search tasks. To illustrate the benefits of using semantic information in scene exploration and visual search tasks, we compare its performance against traditional saliency-based models. In the task of scene exploration, the semantic-based method demonstrates superior performance compared to the traditional saliency-based model in accurately representing the semantic information present in the visual scene. In visual search experiments, searching for instances of a target class in a visual field containing multiple distractors shows superior performance compared to the saliency-driven model and a random gaze selection algorithm. Our results demonstrate that semantic information, from the top-down, influences visual exploration and search tasks significantly, suggesting a potential area of research for integrating it with traditional bottom-up cues.

• The paper introduces a semantic-based active perception model that integrates object detection and Bayesian methods to update semantic maps for efficient visual tasks.
• It employs predictive gaze strategies to reduce shifts and improve accuracy in visual search and scene exploration compared to traditional saliency-based models.
• The approach, despite higher computational demands, shows promise for achieving human-like visual cognition in complex, dynamic environments.

Semantic-Based Active Perception for Humanoid Visual Tasks with Foveal Sensors

Introduction

The paper "Semantic-Based Active Perception for Humanoid Visual Tasks with Foveal Sensors" introduces an advanced methodology for performing visual tasks on humanoid robots using a combination of semantic information and active perception. The study extends a semantic-based foveal active perception model to address both scene exploration and visual search tasks. By leveraging modern object detectors for identifying objects and updating a semantic scene description over successive fixations, the approach demonstrates superior performance compared to traditional saliency-based models.

Methodology Overview

The proposed methodology integrates semantic data from object detectors with active visual perception to emulate human-like visual cognition. The approach utilizes foveal vision, which increases processing efficiency by focusing on high-resolution central vision while reducing peripheral visual information.

System Architecture

Figure 1: Our general methodological approach to semantic visual search and scene exploration tasks.

Object detection frameworks are employed to generate bounding boxes and classification scores for detected objects. These semantic predictions are fused into a semantic map using Bayesian methods, specifically using Dirichlet distributions that account for classification uncertainties.

Figure 2: Representation of the dependencies between the variables that participate in the methodology.

Active Perception Strategy

Active perception in this model involves determining the next gaze location to minimize uncertainty, measured by metrics such as Kullback-Leibler divergence or entropy. For predictive tasks like visual search, the model simulates future updates of the semantic map to predict and select the next best fixation point.

Experimental Results

The methodology was validated through a series of experiments comparing it to traditional saliency-based models and random selection algorithms. The experiments focused on visual search and scene exploration tasks in complex visual environments with multiple distractors.

Visual Search

The semantic model displayed superior performance in visual search tasks compared to saliency-based models like VOCUS2. The predictive approach significantly improved accuracy, reducing the number of gaze shifts required to locate target objects.

Figure 3: Comparison between the mean values of the cumulative performance for predictive and non-predictive approaches.

Scene Exploration

For scene exploration, the semantic model outperformed saliency-based models in accurately mapping the semantic content of scenes with fewer actions. The use of foveal calibration added some benefits to the non-predictive approach, improving its efficacy.

Figure 4: Mean values of the average success rate indicating advantages of the semantic-based method over the random and VOCUS2 approaches.

Computational Considerations

The computational requirements for the proposed methodology are higher than those for traditional saliency models, primarily due to the complexity of semantic map simulations and updates. However, the semantic approach provides a more accurate representation of scene content with fewer actions, compensating for its higher computational cost in applications where precision is critical.

Conclusion

The semantic-based active perception model for humanoid visual tasks offers a promising approach to achieving human-level visual cognition in robots. By integrating semantic information with active perception, the model effectively balances processing efficiency with task accuracy. Future work may explore further integration with top-down cognitive models and extensions to real-world mobile robotic platforms.

Overall, this research presents a robust framework for addressing complex visual tasks in dynamic environments, highlighting the potential of semantic information to enhance robotic perception systems.