MOT-DETR: 3D Single Shot Detection and Tracking with Transformers to build 3D representations for Agro-Food Robots

Introduction

The agro-food industry relies heavily on robotics to face labor shortages and meet production demands. Precise 3D detection and localization of objects are essential for robotic systems in complex agricultural environments. However, occlusions and sensor noise can pose significant challenges.

Background and Contributions

Traditional multi-object tracking (MOT) techniques, including two-stage and recurrent methods, have made significant strides but often fall short in agricultural scenarios with low frame rates and significant occlusions. Despite advancements, popular algorithms like SORT and DeepSORT struggle in environments with obstructed viewpoints and drastic perspective changes inherent to robot operations. This paper introduces MOT-DETR, a transformative method for both detecting and tracking objects over time using convolutional networks and transformers. This technique is particularly designed for 3D environments, enabling robots to construct accurate representations even in occluded settings.

The paper's contributions can be summarized as follows:

• MOT-DETR: a pioneering deep learning method employing transformers for efficient MOT.
• A strategy for integrating 3D data to enhance MOT in environments with complicated occlusions.
• Comparisons between MOT-DETR and existing state-of-the-art tracking methods.
• Testing the robustness of MOT-DETR under varied noise levels in camera pose estimations.

Approach and Architecture

The proposed MOT-DETR processes both 2D images and 3D point clouds. It leverages transformers' self- and cross-attention mechanisms to fuse data from color images and point clouds for improved object differentiation. This method creates 2D bounding boxes, classifies objects, and employs re-identification (re-ID) features for tracking across views. An important distinction of MOT-DETR is its adaptability to single-shot detection and tracking, simplifying the training and operation compared to recurrent methods.

Experiments and Results

MOT-DETR's performance is evaluated in real and synthetic scenarios. Synthetic 3D models of tomato plants are generated to provide training and testing data, allowing the creation of a vast dataset to effectively train the deep neural network. Moreover, the implementation displays resilience when exposed to noise in camera pose estimations, suggesting suitability for real-world robotic applications with inherent sensor inaccuracies. In comparison to state-of-the-art methods, MOT-DETR demonstrates superior tracking performance, especially in sequences with long-term occlusions and significant viewpoint changes.

Conclusion

By integrating 3D data and transformer architecture, the novel MOT-DETR provides a significant advancement in the way robots could navigate, track, and interact with their environments in the agro-food industry. Its application paves the way for improved automation and efficiency in contexts where visual occlusion and sensor noise are prevalent. The model’s robustness against camera pose noise also indicates its potential utility across various robot-operated systems beyond agriculture.