Multi-modal perception for soft robotic interactions using generative models (2404.04220v1)
Abstract: Perception is essential for the active interaction of physical agents with the external environment. The integration of multiple sensory modalities, such as touch and vision, enhances this perceptual process, creating a more comprehensive and robust understanding of the world. Such fusion is particularly useful for highly deformable bodies such as soft robots. Developing a compact, yet comprehensive state representation from multi-sensory inputs can pave the way for the development of complex control strategies. This paper introduces a perception model that harmonizes data from diverse modalities to build a holistic state representation and assimilate essential information. The model relies on the causality between sensory input and robotic actions, employing a generative model to efficiently compress fused information and predict the next observation. We present, for the first time, a study on how touch can be predicted from vision and proprioception on soft robots, the importance of the cross-modal generation and why this is essential for soft robotic interactions in unstructured environments.
- R. Bajcsy, Y. Aloimonos and J.K. Tsotsos “Revisiting active perception” In Autonomous Robots 42, 2018 DOI: 10.1007/s10514-017-9615-3
- “Learning Factorized Multimodal Representations” In International Conference on Representation Learning, 2019
- “Connecting Touch and Vision via Cross-Modal Prediction” In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019
- Q. Tang, J. Liang and F. Zhu “A comparative review on multi-modal sensors fusion based on deep learning” In Signal Processing 213, 2023 DOI: 10.1016/j.sigpro.2023.109165
- F. Nadon, A.J. Valencia and P. Payeur “Multi-Modal Sensing and Robotic Manipulation of Non-Rigid Objects: A Survey” In Robotics 7.4, 2018 DOI: 10.3390/robotics7040074
- J.T. Lee, D. Bollegala and S. Luo ““Touching to See” and “Seeing to Feel”: Robotic Cross-modal Sensory Data Generation for Visual-Tactile Perception” In IEEE ICRA, 2019
- “See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation” In Proceedings of Machine Learning Research, 2022
- “Multimodel Sensor Fusion for Learning Rich Models for Interacting Soft Robots” In IEEE RoboSoft, 2023 DOI: 10.1109/RoboSoft55895.2023.10121992
- “Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks” In IEEE Transactions on Robotics 36.3, 2020 DOI: 10.1109/TRO.2019.2959445
- “Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors” In IEEE ICRA, 2021 DOI: 10.1109/ICRA48506.2021.9561847
- “Soft Robots Modeling: A Structured Overview” In IEEE Transactions on Robotics 39.3, 2023 DOI: 10.1109/TRO.2022.3231360
- S. Grazioso, G. Di Gironimo and B. Siciliano “A Geometrically Exact Model for Soft Continuum Robots: The Finite Element Deformation Space Formulation” In Soft Robotics 6.6, 2019 DOI: 10.1089/soro.2018.0047
- “Forward dynamics of continuum and soft robots: a strain parametrization based approach” In IEEE Transactions on Robotics 37.3, 2020 DOI: 10.1109/TRO.2020.3036618
- C. Alessi, E. Falotico and A. Lucantonio “Ablation Study of a Dynamic Model for a 3D-Printed Pneumatic Soft Robotic Arm” In IEEE Access 11, 2023 DOI: 10.1109/ACCESS.2023.3266282
- D. Kim and al. “Review of machine learning methods in soft robotics” In Plos One, 2021 DOI: 10.1371/journal.pone.0246102
- “Learning-Based Control Strategies for Soft Robots: Theory, Achievements, and Future Challenges” In IEEE Control Systems Magazine 43.3, 2023 DOI: 10.1109/MCS.2023.3253421
- “To Enabling Plant-like Movement Capabilities in Continuum Arms” In 2022 I-RIM Conference DOI: 10.5281/zenodo.7531338
- “Plant-inspired behavior-based controller to enable reaching in redundant continuum robot arms” In 2023 IEEE RoboSoft DOI: 10.1109/RoboSoft55895.2023.10122017
- E. Coevoet, A. Escande and C. Duriez “Optimization-Based Inverse Model of Soft Robots With Contact Handling” In IEEE Robotics and Automation Letters 2.3, 2017 DOI: 10.1109/LRA.2017.2669367
- “Auto-Encoding Variational Bayes” In International Conference on Learning Representations (ICLR2014)
- K. Sohn, H. Lee and X. Yan “Learning Structured Output Representation using Deep Conditional Generative Models” In Advances in Neural Information Processing Systems 28 (NIPS 2015)
- “SoMo: Fast and Accurate Simulations of Continuum Robots in Complex Environments” In IEEE-RSJ IROS, 2021 DOI: 10.1109/IROS51168.2021.9636059
- “PyBullet, a Python module for physics simulation for games, robotics and machine learning”, 2020
- “Stretchable e-skin and transformer enable high-resolution morphological reconstruction for soft robots” In nature machine intelligence 5, 2023 DOI: 10.1038/s42256-023-00622-8
- Enrico Donato (5 papers)
- Egidio Falotico (13 papers)
- Thomas George Thuruthel (6 papers)