Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence (2405.06903v1)

Published 11 May 2024 in cs.CV

Abstract: Garment manipulation (e.g., unfolding, folding and hanging clothes) is essential for future robots to accomplish home-assistant tasks, while highly challenging due to the diversity of garment configurations, geometries and deformations. Although able to manipulate similar shaped garments in a certain task, previous works mostly have to design different policies for different tasks, could not generalize to garments with diverse geometries, and often rely heavily on human-annotated data. In this paper, we leverage the property that, garments in a certain category have similar structures, and then learn the topological dense (point-level) visual correspondence among garments in the category level with different deformations in the self-supervised manner. The topological correspondence can be easily adapted to the functional correspondence to guide the manipulation policies for various downstream tasks, within only one or few-shot demonstrations. Experiments over garments in 3 different categories on 3 representative tasks in diverse scenarios, using one or two arms, taking one or more steps, inputting flat or messy garments, demonstrate the effectiveness of our proposed method. Project page: https://warshallrho.github.io/unigarmentmanip.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Speedfolding: Learning efficient bimanual folding of garments. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–8. IEEE, 2022.
  2. Bag all you need: Learning a generalizable bagging strategy for heterogeneous objects. IROS, 2023.
  3. Cloth3d: clothed 3d humans. In European Conference on Computer Vision, pages 344–359. Springer, 2020.
  4. Cloth funnels: Canonicalized-alignment for multi-purpose garment manipulation. In International Conference of Robotics and Automation (ICRA), 2022.
  5. Autobag: Learning to open plastic bags and insert objects. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3918–3925. IEEE, 2023a.
  6. Learning to grasp clothing structural regions for garment manipulation tasks. arXiv preprint arXiv:2306.14553, 2023b.
  7. Garmentnets: Category-level pose estimation for garments via canonical space shape completion. In The IEEE International Conference on Computer Vision (ICCV), 2021.
  8. Preafford: Universal affordance-based pre-grasping for diverse objects and environments, 2024.
  9. Learning part motion of articulated objects using spatially continuous neural implicit representations. In British Machine Vision Conference (BMVC), 2023.
  10. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. Conference on Robot Learning, 2018.
  11. Physical edge detection in clothing items for robotic manipulation. In 2017 18th International Conference on Advanced Robotics (ICAR), pages 524–529. IEEE, 2017.
  12. Learning dense visual correspondences in simulation to smooth and fold real fabrics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11515–11522. IEEE, 2021.
  13. Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2978–2988, 2023a.
  14. Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7081–7091, 2023b.
  15. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference on Robot Learning, pages 24–33. PMLR, 2022.
  16. Unsupervised learning of dense shape correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4370–4379, 2019.
  17. Surfemb: Dense and continuous correspondence distributions for object pose estimation with learnt surface embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6749–6758, 2022.
  18. Efficient deformable shape correspondence via multiscale spectral manifold wavelets preservation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14536–14545, 2021.
  19. Dynamic cloth manipulation with deep reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4630–4636. IEEE, 2020.
  20. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  21. Cheng-I Lai. Contrastive predictive coding based feature for automatic speaker verification. arXiv preprint arXiv:1904.01575, 2019.
  22. The functional correspondence problem. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15772–15781, 2021.
  23. Regrasping and unfolding of garments using predictive thin shell modeling. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1382–1388. IEEE, 2015.
  24. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, 2019.
  25. Mobileafford: Mobile robotic manipulation through differentiable affordance learning. In 2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA 2024, 2024a.
  26. Unidoormanip: Learning universal door manipulation policy over large-scale and diverse door manipulation environments. arXiv preprint arXiv:2403.02604, 2024b.
  27. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, 2021a.
  28. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 432–448. PMLR, 2021b.
  29. Articulated object manipulation with coarse-to-fine affordance for mitigating the effect of point cloud noise. ICRA, 2024.
  30. Unified particle physics for real-time applications. ACM Transactions on Graphics (TOG), 33(4):1–12, 2014.
  31. Sim-to-real reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 734–743. PMLR, 2018.
  32. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6813–6823, 2021.
  33. Where2explore: Few-shot affordance learning for unseen novel categories of articulated objects. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
  34. Dgcm-net: dense geometrical correspondence matching network for incremental experience-based robotic grasping. Frontiers in Robotics and AI, 7:120, 2020.
  35. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
  36. Franka Robotics. Franka emika panda, a.
  37. Franka Robotics. Libfranka, b.
  38. Deep imitation learning of sequential fabric smoothing from an algorithmic supervisor. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9651–9658. IEEE, 2020.
  39. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4568–4575. IEEE, 2021.
  40. Skeleton merger: an unsupervised aligned keypoint detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 43–52, 2021.
  41. Neural descriptor fields: Se(3)-equivariant object representations for manipulation. In ICRA, 2022.
  42. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9411–9418. IEEE, 2020.
  43. Visuotactile affordances for cloth manipulation with local control. In Proceedings of The 6th Conference on Robot Learning, pages 1596–1606. PMLR, 2023.
  44. Learning to singulate layers of cloth using tactile feedback. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7773–7780, 2022.
  45. Efficient deformable shape correspondence via kernel matching. In 2017 international conference on 3D vision (3DV), pages 517–526. IEEE, 2017.
  46. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. European conference on computer vision (ECCV 2022), 2022.
  47. One policy to dress them all: Learning to dress people with diverse poses and garments. In Robotics: Science and Systems (RSS), 2023.
  48. Fabricflownet: Bimanual cloth manipulation with a flow-based policy. In Conference on Robot Learning, 2021.
  49. VAT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects. In International Conference on Learning Representations, 2022.
  50. Learning environment-aware affordance for 3d articulated object manipulation under occlusions. In Advances in Neural Information Processing Systems (NeurIPS), 2023a.
  51. Learning foresightful dense visual affordance for deformable object manipulation. In IEEE International Conference on Computer Vision (ICCV), 2023b.
  52. Learning to manipulate deformable objects without demonstrations. In 16th Robotics: Science and Systems, RSS 2020. MIT Press Journals, 2020.
  53. Lie-x: Depth image based articulated object pose estimation, tracking, and action recognition on lie groups. International Journal of Computer Vision, 123:454–478, 2017.
  54. Naturalvlm: Leveraging fine-grained natural language for affordance-guided visual manipulation. arXiv preprint arXiv:2403.08355, 2024.
  55. Dextairity: Deformable manipulation can be a breeze. RSS, 2022.
  56. Unifolding: Towards sample-efficient, scalable, and generalizable robotic garment folding. In 7th Annual Conference on Robot Learning, 2023a.
  57. Garmenttracking: Category-level garment pose tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21233–21242, 2023b.
  58. Useek: Unsupervised se (3)-equivariant 3d keypoints for generalizable manipulation. ICRA, 2023c.
  59. Nerf-supervision: Learning dense object descriptors from neural radiance fields. In 2022 International Conference on Robotics and Automation (ICRA), pages 6496–6503. IEEE, 2022.
  60. Learning grasping points for garment manipulation in robot-assisted dressing. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9114–9120. IEEE, 2020.
  61. Learning garment manipulation policies toward robot-assisted dressing. Science Robotics, 7(65):eabm6010, 2022.
  62. Dualafford: Learning collaborative visual affordance for dual-gripper object manipulation. International Conference on Learning Representations (ICLR), 2023.
  63. Clothesnet: An information-rich 3d garment model repository with simulated clothes environment. ICCV, 2023.
  64. Deep fashion3d: A dataset and benchmark for 3d garment reconstruction from single images. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 512–530. Springer, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Ruihai Wu (28 papers)
  2. Haoran Lu (20 papers)
  3. Yiyan Wang (7 papers)
  4. Yubo Wang (53 papers)
  5. Hao Dong (175 papers)
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets