Abstract
Purpose
To detect specularities as elliptical blobs in endoscopy. The rationale is that in the endoscopic setting, specularities are generally small and that knowing the ellipse coefficients allows one to reconstruct the surface normal. In contrast, previous works detect specular masks as free-form shapes and consider the specular pixels as nuisance.
Methods
A pipeline combining deep learning with handcrafted steps for specularity detection. This pipeline is general and accurate in the context of endoscopic applications involving multiple organs and moist tissues. A fully convolutional network produces an initial mask which specifically finds specular pixels, being mainly composed of sparsely distributed blobs. Standard ellipse fitting follows for local segmentation refinement in order to only keep the blobs fulfilling the conditions for successful normal reconstruction.
Results
Convincing results in detection and reconstruction on synthetic and real images, showing that the elliptical shape prior improves the detection itself in both colonoscopy and kidney laparoscopy. The pipeline achieved a mean Dice of 84% and 87% respectively in test data for these two use cases, and allows one to exploit the specularities as useful information for inferring sparse surface geometry. The reconstructed normals are in good quantitative agreement with external learning-based depth reconstruction methods manifested, as shown by an average angular discrepancy of \(12.11^{\circ } \pm 9.86^{\circ }\) in colonoscopy.
Conclusion
First fully automatic method to exploit specularities in endoscopic 3D reconstruction. Because the design of current reconstruction methods can vary considerably for different applications, our elliptical specularity detection could be of potential interest in clinical practice thanks to its simplicity and generalisability. In particular, the obtained results are promising towards future integration with learning-based depth inference and SfM methods.
This is a preview of subscription content, log in via an institution to check access.
References
Morgand A, Tamaazousti M, Bartoli A (2017) A geometric model for specularity prediction on planar surfaces with multiple light sources. IEEE Trans Visual Comput Graph 24(5):1691–1704
Daher R, Vasconcelos F, Stoyanov D (2022) A temporal learning approach to inpainting endoscopic specularities and its effect on image correspondence. arXiv preprint arXiv:2203.17013
Ciuti G, Visentini-Scarzanella M, Dore A, Menciassi A, Dario P, Yang G-Z (2012) Intra-operative monocular 3d reconstruction for image-guided navigation in active locomotion capsule endoscopy. In: 2012 4th IEEE RAS &EMBS international conference on biomedical robotics and biomechatronics (BioRob), pp 768–774. IEEE
Okatani T, Deguchi K (1997) Shape reconstruction from an endoscope image by shape from shading technique for a point light source at the projection center. Comput Vis Image Underst 66(2):119–131
Yeung S, Tsui H-T, Yim A (1999) Global shape from shading for an endoscope image. In: MICCAI. Springer, Berlin, pp 318–327
Makki K, Bartoli A (2022) Normal reconstruction from specularity in the endoscopic setting. arXiv preprint arXiv:2211.05642
Rodriguez-Puigvert J, Recasens D, Civera J, Martinez-Cantin R (2022) On the uncertain single-view depths in colonoscopies. In: MICCAI, pp 130–140
Sengupta A, Bartoli A (2021) Colonoscopic 3D reconstruction by tubular non-rigid structure-from-motion. Int J Comput Assist Radiol Surg 16(7):1237–1241
Halır R, Flusser J (1998) Numerically stable direct least squares fitting of ellipses. In: Proceedings of the 6th international conference in central europe on computer graphics and visualization. WSCG, vol 98, pp 125–132. Citeseer
Fitzgibbon A, Pilu M, Fisher RB (1999) Direct least square fitting of ellipses. IEEE Trans Pattern Anal Mach Intell 21(5):476–480
Forsyth D, Mundy JL, Zisserman A, Coelho C, Heller A, Rothwell C (1991) Invariant descriptors for 3D object recognition and pose. IEEE Trans Pattern Anal Mach Intell 13(10):971–991
Tchoulack S, Langlois JP, Cheriet F (2008) A video stream processor for real-time detection and correction of specular reflections in endoscopic images. In: 2008 joint 6th international IEEE northeast workshop on circuits and systems and TAISA conference, pp 49–52. IEEE
Rau A, Edwards PE, Ahmad OF, Riordan P, Janatka M, Lovat LB, Stoyanov D (2019) Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy. Int J Comput Assist Radiol Surg 14:1167–1176
Acknowledgements
This work has been supported by the Endomapper H2020 project.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest. This is a retrospective analysis, no interventional procedures were performed, and the data was already collected. All patients gave written informed consent in accordance with the UroCCR project (French network of research on kidney cancer, NCT03293563). This article does not contain any studies with animals performed by any of the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Makki, K., Chandelon, K. & Bartoli, A. Elliptical specularity detection in endoscopy with application to normal reconstruction. Int J CARS 18, 1323–1328 (2023). https://doi.org/10.1007/s11548-023-02904-3
Received:
Accepted:
Published:
Version of record:
Issue date:
DOI: https://doi.org/10.1007/s11548-023-02904-3
