Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the Fly Robotic-Assisted Medical Instrument Planning and Execution Using Mixed Reality (2404.05887v1)

Published 8 Apr 2024 in cs.RO

Abstract: Robotic-assisted medical systems (RAMS) have gained significant attention for their advantages in alleviating surgeons' fatigue and improving patients' outcomes. These systems comprise a range of human-computer interactions, including medical scene monitoring, anatomical target planning, and robot manipulation. However, despite its versatility and effectiveness, RAMS demands expertise in robotics, leading to a high learning cost for the operator. In this work, we introduce a novel framework using mixed reality technologies to ease the use of RAMS. The proposed framework achieves real-time planning and execution of medical instruments by providing 3D anatomical image overlay, human-robot collision detection, and robot programming interface. These features, integrated with an easy-to-use calibration method for head-mounted display, improve the effectiveness of human-robot interactions. To assess the feasibility of the framework, two medical applications are presented in this work: 1) coil placement during transcranial magnetic stimulation and 2) drill and injector device positioning during femoroplasty. Results from these use cases demonstrate its potential to extend to a wider range of medical scenarios.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. J. Klodmann, C. Schlenk, A. Hellings-Kuß, T. Bahls, R. Unterhinninghofen, A. Albu-Schäffer, and G. Hirzinger, “An introduction to robotically assisted surgical systems: Current developments and focus areas of research,” Current Robotics Reports, vol. 2, no. 3, pp. 321–332, 2021.
  2. H. H. Hassanalideh and S. Gholampour, “Finding the optimal drill bit material and proper drilling condition for utilization in the programming of robot-assisted drilling of bone,” CIRP Journal of Manufacturing Science and Technology, vol. 31, pp. 34–47, 2020.
  3. N. Sugano, “Computer-assisted orthopaedic surgery and robotic surgery in total hip arthroplasty,” Clinics in orthopedic surgery, vol. 5, no. 1, pp. 1–9, 2013.
  4. M. Zhou, Q. Yu, K. Huang, S. Mahov, A. Eslami, M. Maier, C. P. Lohmann, N. Navab, D. Zapp, A. Knoll, et al., “Towards robotic-assisted subretinal injection: A hybrid parallel–serial robot system design and preliminary evaluation,” IEEE Transactions on Industrial Electronics, vol. 67, no. 8, pp. 6617–6628, 2019.
  5. Z. Qi, Y. Li, X. Xu, J. Zhang, F. Li, Z. Gan, R. Xiong, Q. Wang, S. Zhang, and X. Chen, “Holographic mixed-reality neuronavigation with a head-mounted device: technical feasibility and clinical application,” Neurosurgical Focus, vol. 51, no. 2, p. E22, 2021.
  6. S. Skyrman, M. Lai, E. Edström, G. Burström, P. Förander, R. Homan, F. Kor, R. Holthuizen, B. H. Hendriks, O. Persson, et al., “Augmented reality navigation for cranial biopsy and external ventricular drain insertion,” Neurosurgical Focus, vol. 51, no. 2, p. E7, 2021.
  7. L. Qian, J. Y. Wu, S. P. DiMaio, N. Navab, and P. Kazanzides, “A review of augmented reality in robotic-assisted surgery,” IEEE Transactions on Medical Robotics and Bionics, vol. 2, no. 1, pp. 1–16, 2019.
  8. R. Wen, L. Yang, C.-K. Chui, K.-B. Lim, and S. Chang, “Intraoperative visual guidance and control interface for augmented reality robotic surgery,” in IEEE ICCA 2010.   IEEE, 2010, pp. 947–952.
  9. R. Wen, C.-B. Chng, C.-K. Chui, K.-B. Lim, S.-H. Ong, and S. K.-Y. Chang, “Robot-assisted rf ablation with interactive planning and mixed reality guidance,” in 2012 IEEE/SICE International Symposium on System Integration (SII), 2012, pp. 31–36.
  10. R. Wen, W.-L. Tay, B. P. Nguyen, C.-B. Chng, and C.-K. Chui, “Hand gesture guided robot-assisted surgery based on a direct augmented reality interface,” Computer Methods and Programs in Biomedicine, vol. 116, no. 2, pp. 68–80, 2014, new methods of human-robot interaction in medical practice.
  11. R. Wen, B. P. Nguyen, C.-B. Chng, and C.-K. Chui, “In situ spatial ar surgical planning using projector-kinect system,” in Proceedings of the 4th Symposium on Information and Communication Technology, ser. SoICT ’13.   New York, NY, USA: Association for Computing Machinery, 2013, p. 164–171.
  12. S. Y. Gadre, E. Rosen, G. Chien, E. Phillips, S. Tellex, and G. Konidaris, “End-user robot programming using mixed reality,” in 2019 International conference on robotics and automation (ICRA).   IEEE, 2019, pp. 2707–2713.
  13. M. Ostanin, S. Mikhel, A. Evlampiev, V. Skvortsova, and A. Klimchik, “Human-robot interaction for robotic manipulator programming in mixed reality,” in 2020 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2020, pp. 2805–2811.
  14. M. Q. Tram, J. M. Cloud, and W. J. Beksi, “Intuitive robot integration via virtual reality workspaces,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 11 654–11 660.
  15. J. Krieglstein, G. Held, B. A. Bálint, F. Nägele, and W. Kraus, “Skill-based robot programming in mixed reality with ad-hoc validation using a force-enabled digital twin,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 11 612–11 618.
  16. J. Fu, M. C. Palumbo, E. Iovene, Q. Liu, I. Burzo, A. Redaelli, G. Ferrigno, and E. De Momi, “Augmented reality-assisted robot learning framework for minimally invasive surgery task,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 11 647–11 653.
  17. E. Rosen, D. Whitney, E. Phillips, G. Chien, J. Tompkin, G. Konidaris, and S. Tellex, “Communicating robot arm motion intent through mixed reality head-mounted displays,” in Robotics Research: The 18th International Symposium ISRR.   Springer, 2020, pp. 301–316.
  18. U. Gruenefeld, L. Prädel, J. Illing, T. Stratmann, S. Drolshagen, and M. Pfingsthorn, “Mind the arm: realtime visualization of robot motion intent in head-mounted augmented reality,” in Proceedings of Mensch und Computer 2020, 2020, pp. 259–266.
  19. G. Bolano, Y. Fu, A. Roennau, and R. Dillmann, “Deploying multi-modal communication using augmented reality in a shared workspace,” in 2021 18th International Conference on Ubiquitous Robots (UR).   IEEE, 2021, pp. 302–307.
  20. J. Elsdon and Y. Demiris, “Augmented reality for feedback in a shared control spraying task,” in 2018 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2018, pp. 1939–1946.
  21. E. Azimi, L. Qian, N. Navab, and P. Kazanzides, “Alignment of the virtual scene to the tracking space of a mixed reality head-mounted display,” arXiv preprint arXiv:1703.05834, 2017.
  22. A. Martin-Gomez, H. Li, T. Song, S. Yang, G. Wang, H. Ding, N. Navab, Z. Zhao, and M. Armand, “Sttar: surgical tool tracking using off-the-shelf augmented reality head-mounted displays,” IEEE Transactions on Visualization and Computer Graphics, 2023.
  23. A. Fedorov, R. Beichel, J. Kalpathy-Cramer, J. Finet, J.-C. Fillion-Robin, S. Pujol, C. Bauer, D. Jennings, F. Fennessy, M. Sonka, et al., “3d slicer as an image computing platform for the quantitative imaging network,” Magnetic resonance imaging, vol. 30, no. 9, pp. 1323–1341, 2012.
  24. P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, and G. Ranzuglia, “MeshLab: an Open-Source Mesh Processing Tool,” in Eurographics Italian Chapter Conference, V. Scarano, R. D. Chiara, and U. Erra, Eds.   The Eurographics Association, 2008.
  25. Y. Liu, A. Kheradmand, and M. Armand, “Toward process controlled medical robotic system,” arXiv preprint arXiv:2308.05809, 2023.
  26. Y. Liu, S. J. Liu, S. Sefati, T. Jing, A. Kheradmand, and M. Armand, “Inside-out tracking and projection mapping for robot-assisted transcranial magnetic stimulation,” in Optical architectures for displays and sensing in augmented, virtual, and mixed reality (AR, VR, MR) III, vol. 11931.   SPIE, 2022, pp. 57–70.
  27. D. Coleman, I. Sucan, S. Chitta, and N. Correll, “Reducing the barrier to entry of complex robotic software: a moveit! case study,” arXiv preprint arXiv:1404.3785, 2014.
  28. K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-D point sets,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 9, no. 5, pp. 698–700, 1987.
  29. P. J. Besl and N. D. McKay, “Method for registration of 3-d shapes,” in Sensor fusion IV: control paradigms and data structures, vol. 1611.   Spie, 1992, pp. 586–606.
  30. C. Maurer, J. Fitzpatrick, M. Wang, R. Galloway, R. Maciunas, and G. Allen, “Registration of head volume images using implantable fiducial markers,” IEEE Trans. on Medical Imaging, vol. 16, no. 4, pp. 447–462, Aug 1997.
  31. Y. Liu, J. Zhang, Z. She, A. Kheradmand, and M. Armand, “Gbec: Geometry-based hand-eye calibration,” In press, accepted for publication in: The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).
  32. A. Keller, “Hololens 2 infrared retro-reflector tracking,” https://github.com/andreaskeller96/HoloLens2-IRTracking, 2023.
  33. G. Fattori, A. J. Lomax, D. C. Weber, and S. Safai, “Technical assessment of the ndi polaris vega optical tracking system,” Radiation oncology, vol. 16, no. 1, pp. 1–4, 2021.
  34. D. Ungureanu, F. Bogo, S. Galliani, P. Sama, X. Duan, C. Meekhof, J. Stühmer, T. J. Cashman, B. Tekin, J. L. Schönberger, et al., “Hololens 2 research mode as a tool for computer vision research,” arXiv preprint arXiv:2008.11239, 2020.
  35. A. T. Barker, R. Jalinous, and I. L. Freeston, “Non-invasive magnetic stimulation of human motor cortex,” The Lancet, vol. 325, no. 8437, pp. 1106–1107, 1985.
  36. D. E. Bohning, A. Shastri, Z. Nahas, J. P. Lorberbaum, S. W. Andersen, W. R. Dannels, E.-U. Haxthausen, D. J. Vincent, and M. S. George, “Echoplanar bold fmri of brain activation induced by concurrent transcranial magnetic stimulation,” Investigative radiology, vol. 33, no. 6, pp. 336–340, 1998.
  37. A. Chail, R. K. Saini, P. Bhat, K. Srivastava, and V. Chauhan, “Transcranial magnetic stimulation: a review of its evolution and current applications,” Industrial psychiatry journal, vol. 27, no. 2, p. 172, 2018.
  38. V. Di Lazzaro, R. Bella, A. Benussi, M. Bologna, B. Borroni, F. Capone, K.-H. S. Chen, R. Chen, A. V. Chistyakov, J. Classen, M. C. Kiernan, G. Koch, G. Lanza, J.-P. Lefaucheur, H. Matsumoto, J.-P. Nguyen, M. Orth, A. Pascual-Leone, I. Rektorova, P. Simko, J.-P. Taylor, S. Tremblay, Y. Ugawa, R. Dubbioso, and F. Ranieri, “Diagnostic contribution and therapeutic perspectives of transcranial magnetic stimulation in dementia,” Clinical Neurophysiology, vol. 132, no. 10, pp. 2568–2607, 2021.
  39. S. Harquel, J. Diard, E. Raffin, B. Passera, G. Dall’Igna, C. Marendaz, O. David, and A. Chauvin, “Automatized set-up procedure for transcranial magnetic stimulation protocols,” Neuroimage, vol. 153, pp. 307–318, 2017.
  40. A. Noccaro, A. Mioli, M. D’Alonzo, M. Pinardi, G. Di Pino, and D. Formica, “Development and validation of a novel calibration methodology and control approach for robot-aided transcranial magnetic stimulation (tms),” IEEE Transactions on Biomedical Engineering, vol. 68, no. 5, pp. 1589–1600, 2021.
  41. J. Beckmann, S. J. Ferguson, M. Gebauer, C. Luering, B. Gasser, and P. Heini, “Femoroplasty–augmentation of the proximal femur with a composite bone cement–feasibility, biomechanical properties and osteosynthesis potential,” Medical engineering & physics, vol. 29, no. 7, pp. 755–764, 2007.
  42. E. G. Sutter, S. C. Mears, and S. M. Belkoff, “A biomechanical evaluation of femoroplasty under simulated fall conditions,” Journal of orthopaedic trauma, vol. 24, no. 2, p. 95, 2010.
  43. J. Beckmann, R. Springorum, E. Vettorazzi, S. Bachmeier, C. Lüring, M. Tingart, K. Püschel, O. Stark, J. Grifka, T. Gehrke, et al., “Fracture prevention by femoroplasty—cement augmentation of the proximal femur,” Journal of Orthopaedic Research, vol. 29, no. 11, pp. 1753–1758, 2011.
  44. A. Farvardin, E. Basafa, M. Bakhtiarinejad, and M. Armand, “Significance of preoperative planning for prophylactic augmentation of osteoporotic hip: A computational modeling study,” Journal of biomechanics, vol. 94, pp. 75–81, 2019.
  45. E. Basafa, R. J. Murphy, Y. Otake, M. D. Kutzer, S. M. Belkoff, S. C. Mears, and M. Armand, “Subject-specific planning of femoroplasty: an experimental verification study,” Journal of biomechanics, vol. 48, no. 1, pp. 59–64, 2015.
  46. A. Farvardin, M. Bakhtiarinejad, R. J. Murphy, E. Basafa, H. Khanuja, J. K. Oni, and M. Armand, “A biomechanically-guided planning and execution paradigm for osteoporotic hip augmentation: Experimental evaluation of the biomechanics and temperature-rise,” Clinical Biomechanics, vol. 87, p. 105392, 2021.
  47. M. Bakhtiarinejad, C. Gao, A. Farvardin, G. Zhu, Y. Wang, J. K. Oni, R. H. Taylor, and M. Armand, “A surgical robotic system for osteoporotic hip augmentation: System development and experimental evaluation,” IEEE Transactions on Medical Robotics and Bionics, vol. 5, no. 1, pp. 18–29, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Letian Ai (2 papers)
  2. Yihao Liu (85 papers)
  3. Mehran Armand (51 papers)
  4. Amir Kheradmand (7 papers)
  5. Alejandro Martin-Gomez (9 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets