Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots

Abstract: We present Universal Manipulation Interface (UMI) -- a data collection and policy learning framework that allows direct skill transfer from in-the-wild human demonstrations to deployable robot policies. UMI employs hand-held grippers coupled with careful interface design to enable portable, low-cost, and information-rich data collection for challenging bimanual and dynamic manipulation demonstrations. To facilitate deployable policy learning, UMI incorporates a carefully designed policy interface with inference-time latency matching and a relative-trajectory action representation. The resulting learned policies are hardware-agnostic and deployable across multiple robot platforms. Equipped with these features, UMI framework unlocks new robot manipulation capabilities, allowing zero-shot generalizable dynamic, bimanual, precise, and long-horizon behaviors, by only changing the training data for each task. We demonstrate UMI's versatility and efficacy with comprehensive real-world experiments, where policies learned via UMI zero-shot generalize to novel environments and objects when trained on diverse human demonstrations. UMI's hardware and software system is open-sourced at https://umi-gripper.github.io.

• The paper proposes **UMI (Universal Manipulation Interface)**, combining a portable gripper and wide-angle camera/fisheye lens SLAM system/html> for robust, transferable visuomotor robotic control. It achieves 87.5% success in dynamic tasks and 90% cross-robot success.

Problem formulation and contribution

The paper addresses a persistent bottleneck in robot learning from demonstration: teleoperation provides robot-compatible actions but is expensive, embodiment-specific, and difficult to deploy outside laboratory settings, whereas passive human videos are scalable but lack explicit robot actions and exhibit substantial observation and embodiment gaps. “Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots” proposes UMI, a hardware–software framework intended to occupy the intermediate regime: demonstrations are collected by humans in arbitrary environments using portable hand-held grippers, while the resulting visuomotor policies can be deployed on multiple robot platforms without collecting demonstrations on those robots (2402.10329).

The central claim is that transferability depends less on a single learning algorithm than on jointly designing the demonstration interface, state representation, action parameterization, temporal synchronization, and policy architecture. UMI therefore combines a low-cost hand-held gripper, a wrist-mounted wide-angle camera, mirror-based depth cues, visual–inertial SLAM, continuous gripper control, latency compensation, relative end-effector trajectories, and Diffusion Policy (2402.10329). The resulting system is evaluated on single-arm, dynamic, bimanual, deformable-object, and long-horizon tasks, as well as on out-of-distribution environments and objects.

Demonstration interface

The physical interface is a trigger-activated, 3D-printed parallel-jaw gripper weighing approximately 780 g. Its bill of materials is reported as $73, excluding a GoPro camera and accessories costing approximately $298. The gripper uses soft TPU fingers and continuous width tracking, rather than a binary open–close command. The same camera–gripper geometry is reproduced on the deployment robot, reducing the observation discrepancy between demonstrations and execution and avoiding explicit camera-to-robot-world calibration.

The interface is deliberately sensor-minimal: the GoPro supplies RGB video and inertial measurements, while fiducial markers on the gripper enable estimation of finger width. This design is important for portability, but it shifts the burden of state estimation onto visual–inertial tracking. A 155-degree fisheye lens supplies broad scene coverage from the wrist-mounted viewpoint. The authors use the raw fisheye image rather than rectifying it into a pinhole projection. Their argument is that rectification severely expands peripheral regions while reducing the effective resolution of the central task-relevant area.

Figure 1: UMI uses a portable hand-held gripper with a wrist-mounted GoPro, fisheye observation, and physical side mirrors.

The fisheye camera addresses a specific failure mode of wrist-mounted sensing: the manipulated object can dominate the image while relevant context, support surfaces, targets, and obstacles remain outside the field of view. The paper’s ablation supports this design choice. On the cup-arrangement task, rectifying and cropping the image to a 69-degree field of view reduces success from 20/20 to 11/20, or 55%. The degradation persists even when the object remains visible, with the authors reporting jittery and unnecessarily multimodal behavior. This result suggests that the useful contribution of the fisheye lens is not merely object visibility but preservation of contextual cues needed for action disambiguation.

Figure 2: Rectification of the 155-degree fisheye image distorts peripheral context and compresses central task-relevant information.

The interface also places two mirrors in the peripheral camera view. These mirrors generate virtual viewpoints with different optical centers, providing implicit stereo cues without additional cameras or depth sensors. Because reflected objects have reversed orientation, UMI digitally reflects the mirror crops and swaps the left and right views before policy training. The distinction is empirically consequential: directly supplying uncorrected mirror images achieves 17/20 success, compared with 18/20 without mirrors, whereas digitally corrected mirror views achieve 20/20. Thus, the mirrors are not automatically beneficial; their utility depends on representation-level normalization.

Figure 3: Side mirrors provide virtual stereo views, which are digitally reflected to restore consistent object orientation.

Visual–inertial action recovery

UMI recovers six-degree-of-freedom gripper motion using a modified monocular–inertial SLAM pipeline based on ORB-SLAM3 (2402.10329). The GoPro IMU supplies metric scale and maintains short-term tracking through intervals of motion blur or insufficient visual texture. This capability is particularly relevant to dynamic manipulation, for which monocular structure-from-motion can suffer from scale ambiguity and tracking loss.

The system uses a map-then-localize procedure. A scene map is built before demonstrations, and each subsequent video is relocalized to that map. The authors modify the SLAM system so that relocalization serves as initialization while normal SLAM optimization continues, allowing the map to adapt to scene changes. Fiducial markers can additionally be used during mapping to resolve ambiguities caused by distant features, repeated patterns, or outdoor scenes; the markers need not appear during demonstrations.

A motion-capture benchmark containing seven single-gripper and seven bimanual tasks yields a mean absolute trajectory error of 6.1 mm in position and 3.5 degrees in rotation. Relative pose error between two grippers is 10.1 mm and 0.8 degrees, respectively. These measurements establish that the recovered trajectories are sufficiently precise for the demonstrated tasks, but they do not establish uniformly accurate tracking across textureless or highly dynamic environments. The paper explicitly concedes that the method inherits the texture requirements of visual SLAM.

Figure 4: Visual–inertial SLAM provides metric gripper trajectories and shared-map relative pose for bimanual demonstrations.

Continuous gripper control is another important interface decision. The authors argue that release timing in dynamic tossing depends on object width and therefore cannot be represented reliably by a binary action. Soft fingers also provide passive compliance during contact-rich behaviors, including faucet manipulation and cloth pickup. This mechanical compliance is not equivalent to force sensing, however; the system infers grasp-related behavior through finger width and finger deformation rather than directly measuring interaction forces.

Policy interface and temporal alignment

The policy receives synchronized sequences of RGB images, relative end-effector poses, and gripper widths, and predicts sequences of relative end-effector poses and gripper widths. Diffusion Policy is used throughout the experiments to represent multimodal action distributions, including the clockwise and counter-clockwise solutions to cup reorientation (2402.10329).

The action representation is a sequence of $SE(3)$ transforms, each defined relative to the same current end-effector pose. This differs from delta actions, which compose successive local increments and accumulate tracking error, and from absolute actions, which require a globally calibrated coordinate frame. Relative trajectories also make the policy less sensitive to camera displacement and robot-base placement.

Figure 5: Relative trajectories reference all predicted poses to the current end-effector pose, avoiding delta-action error accumulation and absolute-frame calibration.

The experimental comparison strongly favors the proposed representation. On cup arrangement, relative trajectories achieve 20/20 success. Delta actions achieve 16/20, or 80%, while absolute actions achieve only 5/20, or 25%. The absolute-action result is particularly revealing: although absolute actions are theoretically expressive, the required calibration between SLAM coordinates and the robot base introduces a severe systematic bias. The implication is that coordinate-frame reliability can dominate the nominal advantages of an action representation.

UMI also represents proprioceptive history as a relative trajectory. With a short observation horizon, this supplies velocity-like information while remaining invariant to the robot base and scene-level coordinate choice. In bimanual settings, relative inter-gripper pose is explicitly provided. Removing this signal reduces cloth-folding success from 14/20, or 70%, to 6/20, or 30%. The observed failure is asynchronous grasping of the sweater hem, indicating that visual overlap alone is insufficient for precise two-arm coordination in this setup.

Temporal alignment is treated as part of the policy interface rather than as an implementation detail. During data collection, camera, IMU, gripper, and pose measurements are synchronized within the recording pipeline. During deployment, camera, proprioception, inference, arm execution, and gripper execution introduce heterogeneous delays. UMI measures these delays independently, interpolates lower-latency streams to the camera timestamp, discards actions that are already outdated, and transmits future action commands early enough to compensate for execution latency.

Figure 6: UMI synchronizes heterogeneous observation streams and advances action commands to compensate for execution delay.

The dynamic tossing experiment isolates the effect of this procedure. With latency matching, the policy successfully tosses 105 of 120 objects, or 87.5%. Disabling latency matching reduces performance to 69/120, or 57.5%, a 30 percentage-point drop. The degradation is attributed to jittery arm motion, mismatched gripper and arm execution, and incorrect release timing. This result supports the paper’s stronger systems-level claim: for high-speed manipulation, temporal calibration can be as consequential as model architecture or data volume.

Capability experiments

UMI is evaluated on four tasks with randomized initial states and 20 evaluation episodes for most narrow-domain experiments. The tasks span different sources of difficulty rather than merely different object categories.

Task	Demonstrations	Main capability	Success
Cup arrangement	305	Prehensile, non-prehensile, multimodal actions	20/20
Dynamic tossing	280	Rapid motion and release timing	105/120 objects
Bimanual cloth folding	250	Deformable-object and two-arm coordination	14/20
Dish washing	258	Long horizon, articulated and deformable objects	14/20

The cup task requires placing an espresso cup upright on a saucer with its handle within $\pm 15$ degrees of the desired orientation. It combines grasping, pushing, relative-depth estimation, and multimodal reorientation. The policy achieves 100% success on the primary robot and 90% on a Franka FR2, with both FR2 failures attributed to joint-limit violations. This cross-robot result is evidence for hardware-agnostic deployment, although the failures also demonstrate that the policy is not intrinsically embodiment-aware. UMI instead relies on kinematic filtering and suitable robot placement.

Figure 7: Narrow-domain evaluations show high success on cup arrangement and substantial sensitivity to ablated sensing, action, and proprioceptive interfaces.

Dynamic tossing tests whether human rapid motions can be transferred into robot trajectories when target bins lie outside the robot’s kinematic reach. The 87.5% object-level success rate demonstrates that UMI can represent velocity-dependent behavior and synchronize release with the arm trajectory. The no-latency baseline shows that this capability is conditional on accurate timing compensation.

Cloth folding requires coordinated sleeve folding, hem lifting, rotation, and final folding of a deformable sweater. A centralized policy controlling both arms achieves 70% success. The inter-gripper ablation falls to 30%, establishing that relative two-arm proprioception is not a minor auxiliary feature in this task. The result also clarifies the scope of visual imitation: the policy does not simply learn independent arm behaviors; it uses explicit geometric coupling to coordinate simultaneous contact events.

Dish washing is a seven-step sequential task involving a faucet, plate, sponge, water, ketchup, and recovery behavior when additional sauce is introduced. A CLIP-pretrained ViT-B/16 vision encoder is fine-tuned with Diffusion Policy, producing 14/20 successful episodes, or 70%. By contrast, a ResNet-34 trained from scratch achieves 0/10. The paper therefore makes a strong claim about representation capacity and initialization: for visually complex, long-horizon manipulation with limited task-specific data, pretrained ViT features are necessary in this experiment. The comparison is not a controlled test of architecture alone because it also contrasts pretraining regimes and training configurations, so the result should not be interpreted as establishing universal superiority of ViT over ResNet.

Figure 8: UMI transfers dynamic tossing, bimanual cloth folding, and long-horizon dish washing in addition to cup arrangement.

In-the-wild generalization and collection efficiency

The most extensive generalization experiment uses 1,400 cup-arrangement demonstrations collected by three demonstrators in 30 locations within 12 person-hours. The data include homes, offices, restaurants, and outdoor environments, and 15 cup styles during training. Evaluation takes place in an outdoor cafe and on a black water fountain with a continuously flowing water film, using both training and unseen cup designs.

The policy obtains 28/40 successes on training cups, 15/20 on held-out cups, and 43/60 overall, or 71.7%. On the unseen cups specifically, it achieves 75%, slightly higher than its 70% performance on training cups. The paper describes this as zero-shot generalization because no demonstrations are collected in the evaluation environments or with the held-out cup styles. A model trained only on narrow-domain laboratory data, despite using the same pretrained ViT backbone, achieves 0% in these environments and does not move toward the cup.

Figure 9: Diverse in-the-wild demonstrations support transfer to novel environments and objects, whereas narrow-domain data fail in the same evaluation settings.

The comparison supports the conclusion that pretrained visual representations alone do not substitute for environmental and object diversity. Nevertheless, the evaluation is limited in scale: two unseen environments, two held-out cup styles, and 60 total trials do not characterize generalization over a broad task distribution. Moreover, the evaluation states are manually aligned and success is judged by an operator, which makes the metric operationally meaningful but introduces subjectivity.

UMI also improves data collection throughput relative to space-mouse teleoperation. For cup arrangement, UMI operates at 48% of bare-hand speed and is reported to be more than three times faster than teleoperation. For tossing, it operates at 64% of bare-hand speed, while the space-mouse interface produces no successful demonstration within 15 minutes. The comparison captures reset time, object randomization, and robot faults, making it more representative of end-to-end collection cost than a pure trajectory-recording rate. However, UMI remains slower than direct human demonstration because of its mass, bulk, and lower degrees of freedom.

Figure 10: UMI improves practical demonstration throughput over space-mouse teleoperation while remaining slower than bare-hand demonstration.

Limitations and open questions

The framework depends on kinematic filtering because the deployment robot is not necessarily known during data collection. Filtering removes demonstrations that are infeasible for a target embodiment, but it does not solve the more general problem of adapting valid human trajectories to different kinematic structures, joint limits, or dynamic capabilities. The 90% FR2 result, with failures caused by joint-limit violations, directly illustrates this limitation.

The visual–inertial SLAM pipeline requires sufficient environmental texture and can be vulnerable to feature scarcity, repeated patterns, or severe motion blur. Marker-enhanced mapping improves initialization but does not eliminate the underlying dependence on visual features. The paper leaves open whether a hybrid sensing configuration could preserve portability while supporting texture-deficient spaces.

The mechanical interface is also less dexterous than the human hand. Continuous parallel-jaw control and compliant fingers enable a broader behavior set than binary grippers, but the demonstrated tasks do not establish transfer of highly dexterous in-hand manipulation, force-sensitive assembly, or multi-contact hand behaviors. Finally, the principal success metrics are manually judged, the number of evaluation episodes is modest, and the operator can terminate episodes for safety, faults, or timeout. These conditions are appropriate for physical robot experiments but complicate exact comparison and reproducibility.

Conclusion

UMI presents a coherent systems approach to robot teaching without requiring robots during demonstration collection. Its contribution is the integration of portable hardware, fisheye and mirror-based observation, visual–inertial SLAM, continuous gripper control, relative trajectory representations, explicit inter-gripper geometry, latency matching, and diffusion-based multimodal policy learning. The empirical results show 87.5% object-level success for dynamic tossing, 70% success for bimanual cloth folding and dish washing, 90% cross-robot success on cup arrangement, and 71.7% success in out-of-distribution environments. The ablations indicate that these results depend materially on temporal synchronization, relative action representations, corrected mirror observations, broad visual context, and bimanual proprioception. The framework’s unresolved issues are embodiment feasibility, texture-dependent tracking, limited dexterity, and evaluation subjectivity, but the paper establishes a technically viable route for collecting transferable manipulation data outside conventional robot laboratories.