Several barriers prevent the integration and adoption of augmented reality (AR) in robotic renal surgery despite the increased availability of virtual three-dimensional (3D) models. Apart from correct model alignment and deformation, not all instruments are clearly visible in AR. Superimposition of a 3D model on top of the surgical stream, including the instruments, can result in a potentially hazardous surgical situation. We demonstrate real-time instrument detection during AR-guided robotassisted partial nephrectomy and show the generalization of our algorithm to ARguided robot-assisted kidney transplantation. We developed an algorithm using deep learning networks to detect all nonorganic items. This algorithm learned to extract this information for 65 927 manually labeled instruments on 15 100 frames. Our setup, which runs on a standalone laptop, was deployed in three different hospitals and used by four different surgeons. Instrument detection is a simple and feasible way to enhance the safety of AR-guided surgery. Future investigations should strive to optimize efficient video processing to minimize the 0.5-s delay

Improving Augmented Reality Through Deep Learning: Real-time Instrument Delineation in Robotic Renal Surgery / De Backer, Pieter; Van Praet, Charles; Simoens, Jente; Peraire Lores, Maria; Creemers, Heleen; Mestdagh, Kenzo; Allaeys, Charlotte; Vermijs, Saar; Piazza, Pietro; Mottaran, Angelo; Bravi, Carlo A; Paciotti, Marco; Sarchi, Luca; Farinha, Rui; Puliatti, Stefano; Cisternino, Francesco; Ferraguti, Federica; Debbaut, Charlotte; De Naeyer, Geert; Decaestecker, Karel; Mottrie, Alexandre. - In: EUROPEAN UROLOGY. - ISSN 1873-7560. - STAMPA. - 84:1(2023), pp. 86-91. [10.1016/j.eururo.2023.02.024]

Improving Augmented Reality Through Deep Learning: Real-time Instrument Delineation in Robotic Renal Surgery

Piazza, Pietro;Mottaran, Angelo;
2023

Abstract

Several barriers prevent the integration and adoption of augmented reality (AR) in robotic renal surgery despite the increased availability of virtual three-dimensional (3D) models. Apart from correct model alignment and deformation, not all instruments are clearly visible in AR. Superimposition of a 3D model on top of the surgical stream, including the instruments, can result in a potentially hazardous surgical situation. We demonstrate real-time instrument detection during AR-guided robotassisted partial nephrectomy and show the generalization of our algorithm to ARguided robot-assisted kidney transplantation. We developed an algorithm using deep learning networks to detect all nonorganic items. This algorithm learned to extract this information for 65 927 manually labeled instruments on 15 100 frames. Our setup, which runs on a standalone laptop, was deployed in three different hospitals and used by four different surgeons. Instrument detection is a simple and feasible way to enhance the safety of AR-guided surgery. Future investigations should strive to optimize efficient video processing to minimize the 0.5-s delay
2023
Improving Augmented Reality Through Deep Learning: Real-time Instrument Delineation in Robotic Renal Surgery / De Backer, Pieter; Van Praet, Charles; Simoens, Jente; Peraire Lores, Maria; Creemers, Heleen; Mestdagh, Kenzo; Allaeys, Charlotte; Vermijs, Saar; Piazza, Pietro; Mottaran, Angelo; Bravi, Carlo A; Paciotti, Marco; Sarchi, Luca; Farinha, Rui; Puliatti, Stefano; Cisternino, Francesco; Ferraguti, Federica; Debbaut, Charlotte; De Naeyer, Geert; Decaestecker, Karel; Mottrie, Alexandre. - In: EUROPEAN UROLOGY. - ISSN 1873-7560. - STAMPA. - 84:1(2023), pp. 86-91. [10.1016/j.eururo.2023.02.024]
De Backer, Pieter; Van Praet, Charles; Simoens, Jente; Peraire Lores, Maria; Creemers, Heleen; Mestdagh, Kenzo; Allaeys, Charlotte; Vermijs, Saar; Piazza, Pietro; Mottaran, Angelo; Bravi, Carlo A; Paciotti, Marco; Sarchi, Luca; Farinha, Rui; Puliatti, Stefano; Cisternino, Francesco; Ferraguti, Federica; Debbaut, Charlotte; De Naeyer, Geert; Decaestecker, Karel; Mottrie, Alexandre
File in questo prodotto:
File Dimensione Formato  
PIIS0302283823026337.pdf

accesso aperto

Tipo: Versione (PDF) editoriale
Licenza: Licenza per Accesso Aperto. Creative Commons Attribuzione - Non commerciale - Non opere derivate (CCBYNCND)
Dimensione 2.02 MB
Formato Adobe PDF
2.02 MB Adobe PDF Visualizza/Apri
mmc1 (1).docx

accesso aperto

Tipo: File Supplementare
Licenza: Licenza per Accesso Aperto. Creative Commons Attribuzione - Non commerciale - Non opere derivate (CCBYNCND)
Dimensione 33.62 kB
Formato Microsoft Word XML
33.62 kB Microsoft Word XML Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11585/959070
Citazioni
  • ???jsp.display-item.citation.pmc??? 6
  • Scopus 18
  • ???jsp.display-item.citation.isi??? 12
social impact