DronePaint: Swarm Light Painting with DNN-based Gesture Recognition (2107.11288v1)

Abstract: We propose a novel human-swarm interaction system, allowing the user to directly control a swarm of drones in a complex environment through trajectory drawing with a hand gesture interface based on the DNN-based gesture recognition. The developed CV-based system allows the user to control the swarm behavior without additional devices through human gestures and motions in real-time, providing convenient tools to change the swarm's shape and formation. The two types of interaction were proposed and implemented to adjust the swarm hierarchy: trajectory drawing and free-form trajectory generation control. The experimental results revealed a high accuracy of the gesture recognition system (99.75%), allowing the user to achieve relatively high precision of the trajectory drawing (mean error of 5.6 cm in comparison to 3.1 cm by mouse drawing) over the three evaluated trajectory patterns. The proposed system can be potentially applied in complex environment exploration, spray painting using drones, and interactive drone shows, allowing users to create their own art objects by drone swarms.

• The paper presents DronePaint, which integrates DNN-based gesture recognition with swarm control to create an intuitive human-drone interface.
• It employs MediaPipe for hand tracking, achieving 99.75% gesture accuracy while using alpha-beta filtering for smooth trajectory conversion.
• User experiments reveal a 5.6 cm error in gesture-based drawing, indicating strong potential for interactive art and practical applications like search and rescue.

Overview of "DronePaint: Swarm Light Painting with DNN-based Gesture Recognition"

The paper presents an innovative approach to human-swarm interaction (HSI) through the development of DronePaint, a system that enables direct control of drone swarms using gesture-based input via deep neural network (DNN) technology. The core contribution of this work is the integration of a computer vision (CV) system that allows users to manipulate drone swarms in real-time using only human gestures, without reliance on traditional input devices. This offers a more intuitive and flexible interface for controlling complex drone formations and trajectories.

System Components and Architecture

The DronePaint system architecture is structured into three primary modules:

1. Human-Swarm Interface: This module incorporates hand tracking and gesture recognition functionalities. Using the MediaPipe framework for hand tracking, the system infers 21 key points on a human hand per frame, which are then processed by a DNN-based gesture recognition module. High precision was achieved with a gesture recognition accuracy of 99.75%.
2. Trajectory Processing: The trajectory processing module ensures smooth trajectory execution by employing alpha-beta filtering and linear interpolation. This module transforms generated trajectories from pixel coordinates to real-world coordinates, providing seamless integration with the drone control system.
3. Drone Control System: Utilizing the potential field approach, the control system combines attractive and repulsive forces to guide each drone along the desired path, ensuring collision avoidance within the swarm.

Experimental Evaluation

The system was evaluated through user experiments involving gesture and computer mouse-based trajectory drawing. Notably, the system achieved an average gesture-based trajectory drawing error of 5.6 cm, compared to 3.1 cm for mouse-based drawing. The gesture interface, though less precise than the mouse, presents a more engaging interaction mode that does not require specific hardware, thus democratizing drone painting among non-technical users.

Practical and Theoretical Implications

DronePaint extends the frontier of HSI by integrating advancements in gesture recognition with swarm control, indicating potential applications beyond art, such as search and rescue, exploration, and interactive entertainment. The trajectory generation and fine control allowed by the system can also be leveraged in precision agriculture, environmental monitoring, and architectural modeling.

From a theoretical standpoint, the research highlights the capability of DNNs to effectively model and interpret complex human gestures, signifying advancements in both CV and HSI domains. The use of potential field techniques for autonomous coordination in a swarm also contributes to the body of knowledge in multi-agent systems.

Future Directions

Future work could explore whole-body gesture integration to enhance control dimensions, allowing for gesture-based modulation of swarm speed, orientation, and formation complexity. Expanding the system to function with GPS in outdoor environments would further broaden its applicability. Additionally, developing algorithms for optimal task distribution within a swarm can lead to more efficient path planning and execution, enhancing potential applications.

In summary, this paper provides a detailed exploration of DronePaint, showcasing the integration of gesture recognition and swarm robotics. It outlines a path forward for both research and practical applications, emphasizing the growing intersection of artificial intelligence and robotics in solving complex interaction challenges.