2000 character limit reached
Visual Contact Pressure Estimation for Grippers in the Wild (2303.07344v2)
Published 13 Mar 2023 in cs.RO
Abstract: Sensing contact pressure applied by a gripper can benefit autonomous and teleoperated robotic manipulation, but adding tactile sensors to a gripper's surface can be difficult or impractical. If a gripper visibly deforms, contact pressure can be visually estimated using images from an external camera that observes the gripper. While researchers have demonstrated this capability in controlled laboratory settings, prior work has not addressed challenges associated with visual pressure estimation in the wild, where lighting, surfaces, and other factors vary widely. We present a model and associated methods that enable visual pressure estimation under widely varying conditions. Our model, Visual Pressure Estimation for Robots (ViPER), takes an image from an eye-in-hand camera as input and outputs an image representing the pressure applied by a soft gripper. Our key insight is that force/torque sensing can be used as a weak label to efficiently collect training data in settings where pressure measurements would be difficult to obtain. When trained on this weakly labeled data combined with fully labeled data that includes pressure measurements, ViPER outperforms prior methods, enables precision manipulation in cluttered settings, and provides accurate estimates for unseen conditions relevant to in-home use.
- P. Grady, J. A. Collins, S. Brahmbhatt, C. D. Twigg, C. Tang, J. Hays, and C. C. Kemp, “Visual pressure estimation and control for soft robotic grippers,” 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
- A. A. Nazari, F. Janabi-Sharifi, and K. Zareinia, “Image-based force estimation in medical applications: A review,” IEEE Sensors Journal, vol. 21, no. 7, pp. 8805–8830, 2021.
- C. W. Kennedy and J. P. Desai, “A vision-based approach for estimating contact forces: Applications to robot-assisted surgery,” Applied Bionics and Biomechanics, vol. 2, no. 1, pp. 53–60, 2005.
- E. Noohi, S. Parastegari, and M. Žefran, “Using monocular images to estimate interaction forces during minimally invasive surgery,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2014, pp. 4297–4302.
- A. Marban, V. Srinivasan, W. Samek, J. Fernández, and A. Casals, “A recurrent convolutional neural network approach for sensorless force estimation in robotic surgery,” Biomedical Signal Processing and Control, vol. 50, pp. 134–150, 2019.
- Z. Chua, A. M. Jarc, and A. M. Okamura, “Toward force estimation in robot-assisted surgery using deep learning with vision and robot state,” in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 12 335–12 341.
- D. Kim, H. Cho, H. Shin, S.-C. Lim, and W. Hwang, “An efficient three-dimensional convolutional neural network for inferring physical interaction force from video,” Sensors, vol. 19, no. 16, p. 3579, 2019.
- Y. Wang, D. Held, and Z. Erickson, “Visual haptic reasoning: Estimating contact forces by observing deformable object interactions,” IEEE Robotics Autom. Lett., vol. 7, no. 4, pp. 11 426–11 433, 2022. [Online]. Available: https://doi.org/10.1109/LRA.2022.3199684
- N. Kuppuswamy, A. Alspach, A. Uttamchandani, S. Creasey, T. Ikeda, and R. Tedrake, “Soft-bubble grippers for robust and perceptive manipulation,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 9917–9924.
- B. Ward-Cherrier, N. Pestell, L. Cramphorn, B. Winstone, M. E. Giannaccini, J. Rossiter, and N. F. Lepora, “The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies,” Soft Robotics, vol. 5, no. 2, pp. 216–227, 2018.
- A. Yamaguchi and C. G. Atkeson, “Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables,” in 2016 IEEE-RAS 16th International Conference on Humanoid Robots. IEEE, 2016, pp. 1045–1051.
- W. Yuan, S. Dong, and E. H. Adelson, “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017.
- M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, et al., “Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation,” IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 3838–3845, 2020.
- J. A. Collins, P. Grady, and C. C. Kemp, “Force/torque sensing for soft grippers using an external camera,” IEEE International Conference on Robotics and Automation (ICRA), 2023.
- P. Grady, J. A. Collins, C. Tang, C. D. Twigg, J. Hays, and C. C. Kemp, “Visual estimation of fingertip pressure on diverse surfaces using easily captured data,” 2023. [Online]. Available: https://arxiv.org/abs/2301.02310
- M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable features with deep adaptation networks,” in International conference on machine learning. PMLR, 2015, pp. 97–105.
- E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell, “Deep domain confusion: Maximizing for domain invariance,” arXiv preprint arXiv:1412.3474, 2014.
- M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Unsupervised domain adaptation with residual transfer networks,” Advances in neural information processing systems, vol. 29, 2016.
- Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” The journal of machine learning research, vol. 17, no. 1, pp. 2096–2030, 2016.
- J. Ahn and S. Kwak, “Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4981–4990.
- Y.-T. Chang, Q. Wang, W.-C. Hung, R. Piramuthu, Y.-H. Tsai, and M.-H. Yang, “Weakly-supervised semantic segmentation via sub-category exploration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8991–9000.
- S. Paul, Y.-H. Tsai, S. Schulter, A. K. Roy-Chowdhury, and M. Chandraker, “Domain adaptive semantic segmentation using weak labels,” in European conference on computer vision. Springer, 2020, pp. 571–587.
- S. Xie, R. B. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2017.
- T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2017, pp. 2117–2125.
- F. Massa, R. Marlet, and M. Aubry, “Crafting a multi-task CNN for viewpoint estimation,” BVMC, 2016.
- D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, (ICLR) 2015, Y. Bengio and Y. LeCun, Eds., 2015.
- C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich, “The design of Stretch: A compact, lightweight mobile manipulator for indoor human environments,” in IEEE International Conference on Robotics and Automation (ICRA), 2022.
- S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marín-Jiménez, “Automatic generation and detection of highly reliable fiducial markers under occlusion,” Pattern Recognition, vol. 47, no. 6, pp. 2280–2292, 2014.
- G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000.
- P. Grady, C. Tang, S. Brahmbhatt, C. D. Twigg, C. Wan, J. Hays, and C. C. Kemp, “PressureVision: estimating hand pressure from a single RGB image,” European Conference on Computer Vision (ECCV), 2022.
- A. Lukezic, T. Vojir, L. ˇCehovin Zajc, J. Matas, and M. Kristan, “Discriminative correlation filter with channel and spatial reliability,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6309–6318.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
