EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting
(2401.11535)Abstract
Surgical 3D reconstruction is a critical area of research in robotic surgery, with recent works adopting variants of dynamic radiance fields to achieve success in 3D reconstruction of deformable tissues from single-viewpoint videos. However, these methods often suffer from time-consuming optimization or inferior quality, limiting their adoption in downstream tasks. Inspired by 3D Gaussian Splatting, a recent trending 3D representation, we present EndoGS, applying Gaussian Splatting for deformable endoscopic tissue reconstruction. Specifically, our approach incorporates deformation fields to handle dynamic scenes, depth-guided supervision with spatial-temporal weight masks to optimize 3D targets with tool occlusion from a single viewpoint, and surface-aligned regularization terms to capture the much better geometry. As a result, EndoGS reconstructs and renders high-quality deformable endoscopic tissues from a single-viewpoint video, estimated depth maps, and labeled tool masks. Experiments on DaVinci robotic surgery videos demonstrate that EndoGS achieves superior rendering quality. Code is available at https://github.com/HKU-MedAI/EndoGS.
Overview
-
EndoGS introduces Gaussian Splatting (GS) for high-fidelity 3D reconstruction of deformable tissues in endoscopic videos, enhancing robotic surgery applications.
-
This technique models dynamic surgical scenes, effectively handles tool occlusions, and employs deformation fields and depth-guided supervision for quality reconstruction.
-
Experimental evaluation on robotic surgery videos shows EndoGS surpassing existing methods in rendering quality, confirmed by image quality metrics and rendering speed.
-
Despite challenges with single-viewpoint inputs, EndoGS's advances offer significant potential for surgical training, planning, and simulation, with future research expected to expand its applications.
Enhancing Deformable Tissue Reconstruction in Robotic Surgery with Gaussian Splatting
Method Overview
The paper introduces EndoGS, a novel technique employing Gaussian Splatting (GS) for the reconstruction of deformable tissues in endoscopic video footage, targeted at applications in robotic surgery. By incorporating deformation fields and depth-guided supervision, EndoGS effectively deals with the dynamics of surgical scenes and tool occlusion issues, resulting in high-fidelity 3D reconstructions from a single viewpoint. This method builds upon the foundation of 3D Gaussian Splatting, further extending its utility in the medical domain by specifically tackling the challenges associated with endoscopic video analysis.
Technical Contributions
Key contributions of this study are as follows:
- Introduction of Gaussian Splatting for Medical 3D Reconstruction: It pioneers the application of Gaussian Splatting in the medical imaging field, showcasing its potential for endoscopic surgical procedures.
- Dynamic Scene Modeling: The method models surgical scenes as combinations of static and deformable parameters over time, leveraging a multi-resolution voxel plane approach for encoding spatial-temporal information.
- Depth-Guided Supervision and Occlusion Handling: It innovates by incorporating depth estimation and spatiotemporal weighting masks for occlusion handling, alongside total variation regularization to maintain quality across dimensions.
Implementation & Optimization
The EndoGS pipeline demonstrates a structured approach to deformable tissue reconstruction:
- The initial 3D Gaussian models capture the static scene structure.
- Deformation fields, modeled via MLPs, account for dynamic scene changes over time.
- Spatial-temporal weight masks, combined with depth maps, enhance the model's learning from visible tissue areas while addressing tool occlusions.
- The model employs depth-guided and total variation losses to regulate the 3D reconstruction process, ensuring both local coherence and global accuracy.
Experimental Evaluation
Evaluations on DaVinci robotic surgery videos revealed that EndoGS outperforms existing methods in rendering quality significantly. With respect to image quality metrics (PSNR, SSIM, LPIPS) and rendering speed (FPS), EndoGS presented superior performance, affirming its efficacy and efficiency for real-time applications. Ablation studies further validated the importance of depth regularization and spatial total variation loss, underscoring their roles in enhancing the reconstruction quality and consistency.
Future Directions and Limitations
Despite its advantages, the technique encounters challenges related to the single-viewpoint nature of the input videos, which may limit the method's applicability for more complex surgical tasks. Potential improvements could involve exploring multi-viewpoint reconstruction approaches and introducing surface-oriented Gaussian optimization strategies to refine the model's accuracy and reliability.
Conclusion
EndoGS represents a substantive advance in the reconstruction of deformable tissues from endoscopic video footage. By adeptly addressing the intricacies of dynamic scene modeling and occlusion, the method sets a new standard for 3D visualization in robotic surgeries. Its implications extend beyond immediate surgical applications, offering avenues for enhanced surgical training, planning, and simulation technologies. Future research will undoubtedly expand on this foundation, further unlocking the potential of Gaussian Splatting in medical imaging and beyond.
Create an account to read this summary for free: