Dual-arm Coordinated Manipulation for Object Twisting with Human Intelligence (2108.12007v1)

Abstract: Robotic dual-arm twisting is a common but very challenging task in both industrial production and daily services, as it often requires dexterous collaboration, a large scale of end-effector rotating, and good adaptivity for object manipulation. Meanwhile, safety and efficiency are preliminary concerns for robotic dual-arm coordinated manipulation. Thus, the normally adopted fully automated task execution approaches based on environmental perception and motion planning techniques are still inadequate and problematic for the arduous twisting tasks. To this end, this paper presents a novel strategy of the dual-arm coordinated control for twisting manipulation based on the combination of optimized motion planning for one arm and real-time telecontrol with human intelligence for the other. The analysis and simulation results showed it can achieve collision and singularity free for dual arms with enhanced dexterity, safety, and efficiency.

• The paper introduces a dual-arm system that integrates optimized motion planning with real-time teleoperation to achieve coordinated object twisting.
• It employs self-collision detection and directional manipulability evaluation to ensure safe, efficient, and collision-free manipulation.
• Simulation results with a flexible object demonstrate improved dexterity and alignment control under varying stiffness conditions.

Dual-arm Coordinated Manipulation with Human-in-the-Loop

This paper introduces a dual-arm robotic system designed for coordinated object twisting, leveraging human intelligence through real-time teleoperation to enhance adaptability and overcome challenges associated with object flexibility and imprecise grasping (2108.12007). The proposed system combines optimized motion planning for one arm with real-time telecontrol for the other, achieving collision and singularity-free manipulation with improved dexterity, safety, and efficiency.

Problem Formulation

The paper identifies key constraints in dual-arm twisting, including the need for coordinated end-effector rotation, collision avoidance, and accommodation of environmental restrictions. It highlights the difficulty introduced by flexible objects, where grasping inaccuracies and deformations complicate axis alignment. The authors define the dual-arm twisting task through several states:

• Initial State: Both arms are in a predefined starting configuration.
• Grasping Process: One arm (right) grasps one end of the object and moves it to a preparation location.
• Twisting Alignment: The other arm (left), controlled via teleoperation, approaches the object's other end, adjusting its pose to achieve axis alignment.
• Twisting Process: Both arms rotate in opposite directions to twist the object.

The paper acknowledges that variations in object stiffness present unique alignment challenges (Figure 1). Highly stiff objects may experience pose inaccuracies due to grasping errors. Flexible objects introduce bending, and soft objects exhibit large deformations, complicating the alignment process for the second arm.

Figure 1: Illustration for twisting axis alignment problem caused by the grasped object, and the end-effectors' grasping coordinates are indicated along the both ends of the object.

Proposed Solution

The core concept involves using teleoperation to address the twisting alignment problems. The human operator can make real-time decisions to adjust the pose of the second arm, achieving accurate grasping and axis alignment. The authors justify the use of position mode control, simplifying the problem by assuming negligible interactive forces and resistance torque for elastic or deformable objects. The overall architecture of the dual-arm system is shown in Figure 2.

Figure 2: The systematic architecture of the dual-arm coordinated manipulation for object twisting with human intelligence.

Implementation Details

Teleoperation Architecture

The master side employs a haptic device (Omega 7) for human input, while the slave side executes the dual-arm manipulation. The Cartesian space kinematic mapping between the master and slave is defined as:

$$X_s = g(X_m) = g\left(^m f_{k}\left(q_{m}\right)\right)$$

where $X$ is the end pose, $q$ is the joint state, and $m$ and $s$ represent the master and slave sides, respectively. The target joint state of the slave robot arm is then calculated using the inverse kinematics function:

$q_{s} = ^s f_{k}^{-1}\left(X_{s}\right) = \,^s f_{k}^{-1}\left(g\left(\,^m f_{k}\left(q_{m}\right)\right)\right)$

Self-Collision Detection

To ensure safe dual-arm coordination, the minimal distance between the manipulators is derived using models for line-to-line and point-to-line distances (Figure 3).

Figure 3: Collision detection models.

The spatial linear equations of two links are:

$\left\{\begin{array}{l} P_{Li}\left(c_{Li}\right)=P_{L i}+c_{Li} \boldsymbol{n}_{Li} \quad (i = 1,2, \dots, N_L)\[0.5ex] P_{Rj}\left(c_{Rj}\right)=P_{R j}+c_{Rj} \boldsymbol{n}_{Rj} \quad (j = 1,2, \dots, N_R) \end{array}\right.$

where $n_{Li}$ and $n_{Rj}$ are the direction vectors of $\overrightarrow{P_{Li}P_{L,i+1}}$ and $\overrightarrow{P_{Rj}P_{R,j+1}}$ respectively, $N_L$ and $N_R$ are the joint number of the two arms, and $c_{Li}$ and $c_{Rj}$ are the coefficients.

The distance $d_{lij}$ between the two lines is:

$$d_{lij}=\frac{\left(\boldsymbol{n}_{Li} \times \boldsymbol{n}_{Rj}\right) \cdot\left[P_{Li}\left(c_{Li}\right)-P_{Rj}\left(c_{Rj}\right)\right]}{\left|\boldsymbol{n}_{Li} \times \boldsymbol{n}_{Rj}\right|}$$

Directional Manipulability Evaluation

The manipulability of a manipulator, which measures its kinematic performance, is evaluated using:

$M= 1 / k^{T}\left(J J^{T}\right)^{-1} k$

where $k$ is the directional unit vector of velocity, $J =[J_v \ J_\omega]^T$ is the Jacobian matrix of the manipulator.

Weighted Configuration Variation

The fitness function is expressed as:

$\begin{array}{l} \arg \min f_{a}=\sum_{i,j=1}^{7}\left(A_{L i}+A_{R j}\right) \[1mm] \text { s.t. }\left\{\begin{array}{l} | J_{L}|\neq 0, \ | J_{R} \mid \neq 0 \[1mm] \theta_{R j} \in\left[\theta_{R j, \min }, \theta_{R j, \max }\right] \[1mm] \theta_{L i} \in\left[\theta_{L i, \min }, \theta_{L i, \max }\right] \end{array}\right. \end{array}$

where $A_{Li}$ and $A_{Rj}$ represent the weighted angle changes for both arms from initial configuration to final target configuration, and $J_L$ and $J_R$ are the Jacobian matrices of the left and right arms.

Simulation Results

The proposed method was simulated using PyBullet with a flexible cylindrical object. The simulation involved the right arm grasping one end, followed by teleoperated grasping and alignment by the left arm (Figure 4). The teleoperated left arm's trajectories are shown in Figure 5, while joint state variations are depicted in Figure 6.

Figure 4: Simulation states: (a) Starting state: the right arm starts to grasp at one end of a cylindrical object which is flexible but with a certain stiffness. (b) Idea twisting alignment state: the right arm lifts the grasped end to a preparation location, the teleoperated left arm grasps the other end and adjusts its spatial pose by trying to keep the both ends in a same line to reconstruct the cylinder and achieve twisting alignment.

Figure 5: Left arm end-effector's teleoperating trajectories during the simulation tests of dual-arm coordinated twisting manipulation.

Figure 6: Joint state variation during the manipulation.

Conclusion

The paper presents a dual-arm coordinated manipulation framework for object twisting, integrating teleoperation to handle challenges posed by object flexibility and alignment. The inclusion of self-collision detection, directional manipulability evaluation, and weighted configuration variation enhances the safety, efficiency, and stability of the manipulation. The simulation results demonstrate the feasibility of the approach. Further quantitative analysis and real robot experiments are planned to validate the method.