AENEAS Project: Live Image-Based Navigation and Roadmap Generation in Endoscopic Neurosurgery Using Machine Vision

BACKGROUND AND OBJECTIVES: Artificial intelligence algorithms have proven capable of replicating cognitive processes. Our aim was to replicate human roadmap generation for endoscopic neurosurgery with a live image-based machine vision method. METHODS: Surgical videos of a highly standardized surgical approach are labeled and used for algorithm training. After object detection (YOLOv7) to generate bounding boxes for landmark anatomical structures, an autoencoder first encodes the currently detected structures into an estimated position within this anatomical roadmap and then enables extrapolation of structures that are expected to be encountered in forward or backward directions. Average precision of the model applied to the test videos at an intersection-over-union threshold of 0.5 is reported. RESULTS: In total, 166 anonymized endoscopic recording (3 × 106 labeled video frames) were included. We performed model development using 146 videos and held out 20 videos for evaluation (test set). The performance regarding bounding box detection among the 20 test set videos on average was 53.4. Evaluation of the performance of the autoencoder model in detecting the current position within the roadmap of the surgical approach is evaluated semiquantitatively, showing that the first detection of anatomical structures by the model corresponds well to their label distribution along the latent variable encoding the anatomical roadmap. We also provide videos demonstrating the mixed reality head’s up display for anatomical navigation. CONCLUSION: Our method enables reliable identification of key anatomical structures during endoscopic endonasal trans-sphenoidal surgery in mixed reality. Through encoding detected landmark anatomical structures, a surgical roadmap is encoded. This approach allows for detection of visible anatomical structures and enables extrapolation toward the location of those yet to be dissected in deeper anatomical layers. Further development of such algorithms may pave the way toward adding a mixed reality, real-time anatomical navigation software to the neurosurgeon’s armamentarium.