Real-time Navigation Research Articles

SURG-26. IMPROVEMENT IN PRECISION AND SURGICAL EFFICIENCY USING THE STEALTH AUTOGUIDE™

Abstract INTRODUCTION The Stealth Autoguide™ (Medtronic) automatically determines the path for guiding instruments based on the created surgical plan. It seamlessly integrates continuous real-time navigation with robotic alignment capabilities, enhancing intraoperative workflow efficiency and supporting precise positioning. Our institution has achieved favorable results with surgeries utilizing the Stealth Autoguide™, and we report these outcomes and our future prospects. METHODS We conducted 15 surgeries at our institution using the StealthStation S8 (Medtronic) linked with the Stealth Autoguide™. The average age was 56.2 years, with 10 males and 5 females, all under general anesthesia. RESULTS The cases included 11 gliomas, 2 primary central nervous system lymphomas, 1 germ cell tumor, and 1 brain abscess. Applications of the Stealth Autoguide™ included 7 biopsies (6 burr holes, 1 craniotomy), 6 placements of fenestrations, and 2 tubing placements. The biopsies allowed for quick and reliable tissue sampling. In fenestration cases, it was possible to position quickly and precisely at the target, placing it at the planned site with high accuracy. Additionally, by using a 3D endoscopic system, surgeons could view both the surgical field and navigation screens simultaneously without shifting their gaze, enhancing safety during procedures. In the case of an intraventricular germ cell tumor, tubing preceded by ViewSite™ allowed for a safe and clear surgical field. CONCLUSION Surgeries using the Stealth Autoguide™ were conducted safely and efficiently, regardless of the surgical technique, achieving high-precision positioning. Continuing to utilize this system is expected to improve the accuracy, safety, and reproducibility of surgeries, thereby reducing the burden on patients.

Chat-Bot Based Ticketing System Using Dialogflow and Llama LLM

MyMuseum represents an innovative solution designed to revolutionize the ticket booking process for museums across India, with the primary goal of enhancing engagement and increasing visitor numbers. At its core, the system incorporates a sophisticated conversational AI chatbot that streamlines the ticket booking process, eliminating the hassles traditionally associated with museum visits. This chatbot is complemented by an additional AI-powered feature that interactively presents exhibit information, creating a comprehensive and engaging user experience. The platform boasts a user-friendly interface accessible around the clock, significantly simplifying ticket booking procedures while offering real-time navigation support and comprehensive guides for all participating museums. Key features of MyMuseum include automated event management, a system for notifications and alerts, virtual tour capabilities, and interactive content delivery, collectively forming a holistic solution for museum visitors. This report provides an in-depth examination of the underlying technology stack, which encompasses the ticket-booking chatbot, a Large Language Model (LLM) for disseminating information about museums, their artifacts, and exhibits, integrated payment gateways, and robust backend services. It highlights the myriad benefits of the system, including enhanced automation, scalability, and the generation of data-driven insights. Beyond improving convenience for visitors, MyMuseum is projected to drive increased museum attendance, boost revenue through strategic upselling opportunities, and play a crucial role in enhancing cultural awareness and education. Furthermore, it promises to alleviate operational burdens on museum staff by providing real-time analytics and improving overall efficiency and sustainability in museum operations

Advanced Edge AI Navigation and Assistance System for Blind User

- Despite substantial advancements in visual assistance technology, many existing systems are limited by sensor capabilities, computational resources, and power consumption. Traditional computer vision algorithms struggle to perform complex tasks required for real-time navigation. This paper presents the design and implementation of an advanced computer vision-based navigation system specifically tailored for Blender users, aimed at facilitating independent navigation in diverse environments .By leveraging edge Artificial Intelligence (AI) and deep learning methodologies, the proposed system achieves real-time object detection, person recognition, and environmental awareness, addressing the critical challenges posed by conventional systems. The use of cost-effective, low-power mobile computing platforms, such as smart depth sensors like the OpenCV AI Kit-Depth (OAK-D), enables efficient processing while ensuring portability. This system not only enhances the ability to identify and navigate obstacles but also incorporates additional functionalities, including reading written notices aloud and responding to traffic signals. Key design considerations, such as training data collection, computational efficiency, and portability, have been meticulously addressed to ensure reliable performance in real-world scenarios. The incorporation of an AI-driven voice interface facilitates user-friendly interaction, making this system an innovative and unobtrusive visual assistance device.

Three-Dimensional Holographic-Guided Robotic Lung Segmentectomy for Deep Pulmonary Nodules: Technique and Initial Results.

Background: Diagnosis and treatment of small and isolated lung nodules remain challenging issues. Purpose: The aim of this article is to report the technique of real-time navigation using holographic reconstruction technology combined with a robot assisted thoracic surgery (RATS) platform for lung resection in patients withsmall deep nodules.Research Design: The pre-surgery 3D planning was based on the chest CT scan. The reconstruction was uploaded to a head-mounted display for real-time navigation during mini invasive robot assisted surgery performed with an open console platform. We evaluated this technique with the success rate of diagnosis, the operative time and the post-operative course.Study Sample: This technique was performed in 6 patients (4 female, mean age 65 years) to date.Results: The precision of the head-mounted display based localization system was effective in all cases without the need of open conversion. The mean diameter of the nodules was 8mm (6-9). The diagnosis was a lung cancer (n = 5) and tuberculoma (n = 1). The mean operative time was 125min (100-145). The mean hospital stay was 2.5days (1-3).Conclusions: In conclusion, the intraoperative navigation using the 3D holographic assistance was an helpful tool for mini invasive RATS lung segmentectomy without the need of preoperative localization.

Polar Motion Ultra-Short-Term Prediction of Least-Squares+Multivariate Autoregressive Hybrid Method by Using the Kalman Filter.

The polar motion (PM, including two parameters PMx and PMy) ultra-short-term prediction (1-10 days) is demanded in the real-time navigation of satellites and spacecrafts. Improving the PMx and PMy ultra-short-term predictions accuracies are a key to optimize the performance of these related applications. Currently, the least squares (LS)+autoregressive (AR) hybrid method is regarded as one of the most capable approaches for ultra-short-term predictions of PMx and PMy. The Kalman filter has proven to be effective in improving the ultra-short-term prediction performance of the LS+AR hybrid method, but the PMx and PMy ultra-short-term predictions accuracies are still not able to satisfy some related applications. In order to improve the performance of PM ultra-short-term prediction, it is worth exploring the combinations of existing methods. Throughout the existing predicted methods, the LS+multivariate autoregressive (MAR) hybrid method by using the Kalman filter has the potential to improve the accuracy of PM ultra-short-term prediction. In addition, a PM prediction performance analysis of the LS+MAR hybrid method by using the Kalman filter, namely the LS+MAR+Kalman hybrid method, is still missing. In this contribution, we proposed the LS+MAR+Kalman hybrid method for PM ultra-short-term prediction. The data sets for PM predictions, which range from 1 to 10 days, have been tested based on the International Earth Rotation and Reference Systems Service Earth Orientation Parameter (IERS EOP) 14 C04 series to assess the performance of the LS+MAR+Kalman hybrid model. The experimental results illustrated that the LS+MAR+Kalman hybrid method can effectively execute PMy ultra-short-term predictions. The improvement of PMy prediction accuracy can rise up to 12.69% for 10-day predictions, and the improvement of ultra-short-term predictions is 7.64% on average.

Full-course NIR-II imaging-navigated fractionated photodynamic therapy of bladder tumours with X-ray-activated nanotransducers

The poor 5-year survival rate for bladder cancers is associated with the lack of efficient diagnostic and treatment techniques. Despite cystoscopy-assisted photomedicine and external radiation being promising modalities to supplement or replace surgery, they remain invasive or fail to provide real-time navigation. Here, we report non-invasive fractionated photodynamic therapy of bladder cancer with full-course real-time near-infrared-II imaging based on engineered X-ray-activated nanotransducers that contain lanthanide-doped nanoscintillators with concurrent emissions in visible and the second near-infrared regions and conjugated photosensitizers. Following intravesical instillation in mice with carcinogen-induced autochthonous bladder tumours, tumour-homing peptide-labelled nanotransducers realize enhanced tumour regression, robust recurrence inhibition, improved survival rates, and restored immune homeostasis under X-ray irradiation with accompanied near-infrared-II imaging. On-demand fractionated photodynamic therapy with customized doses is further achieved based on quantifiable near-infrared-II imaging signal-to-background ratios. Our study presents a promising non-invasive strategy to confront the current bladder cancer dilemma from diagnosis to treatment and prognosis.

A novel edge intelligence-based solution for safer footpath navigation of visually impaired using computer vision

Navigating through a tactile paved footpath surrounded by various sizes of static and dynamic obstacles is one of the biggest impediments visually impaired people face, especially in Dhaka, Bangladesh. This problem is important to address, considering the number of accidents in such densely populated footpaths. We propose a novel deep-edge solution using Computer Vision to make people aware of the obstacles in the vicinity and reduce the necessity of a walking cane. This study introduces a diverse novel tactile footpath dataset of Dhaka covering different city areas. Additionally, existing state-of-the-art deep neural networks for object detection have been fine-tuned and investigated using this dataset. A heuristic-based breadth-first navigation algorithm (HBFN) is developed to provide navigation directions that are safe and obstacle-free, which is then deployed in a smartphone application that automatically captures images of the footpath ahead to provide real-time navigation guidance delivered by speech. The findings from this study demonstrate the effectiveness of the object detection model, YOLOv8s, which outperformed other benchmark models on this dataset, achieving a high mAP of 0.974 and an F1 score of 0.934. The model’s performance is analyzed after quantization, reducing its size by 49.53% while retaining 98.97% of the original mAP.

Low-latency intelligent service combination caching strategy with density peak clustering algorithm in vehicle edge computing

In the dynamic field of Vehicle Edge Computing (VEC), the demand for intelligent vehicular systems to process vast amounts of data is escalating, driven by advancements in autonomous driving and real-time navigation technologies. Optimizing service latency and minimizing transmission costs are crucial for enhancing the performance of vehicular networks. Traditional service caching strategies, which largely rely on the popularity of individual services, often fail to account for the intricate interdependencies between services. The oversight can result in redundant data transfers and inefficient use of storage resources. In response, our paper introduces a novel approach to service combination caching within a heterogeneous computational framework comprising vehicles, edge servers, and the cloud. Our strategy focuses on minimizing user wait times and data transmission costs during task execution, while adhering to the caching budget constraints of service providers. Key contributions include the development of an Improved Density Peak Clustering (IDPC) algorithm to facilitate cooperative clustering among edge servers and the design of a Service Combination Caching Strategy (SCCS). The SCCS approach reduces caching costs by categorizing servers, forming efficient clusters, and strategically allocating storage. Simulation results demonstrate that the method outperforms existing strategies by significantly decreasing task execution delays and transmission costs, thereby greatly enhancing the quality of service in vehicular applications.

Effect of Spectral Filtering and Segmental X-ray Tube Current Switch-Off on Interventionalist's Scatter Exposure during CT Fluoroscopy.

Dose optimization in computed tomography (CT) is crucial, especially in CT fluoroscopy (fluoro-CT) used for real-time navigation, affecting both patient and operator safety. This study evaluated the impact of spectral X-ray filtering using a tin filter (Sn filter), and a method called partial-angle computed tomography (PACT), which involves segmentally switching off the X-ray tube current at the ambient dose rate H˙*(10) at the interventional radiologist's (IR) position. Measurements were taken at two body regions (upper body: head/neck; lower body: lower legs/feet) using a 120 kV X-ray tube voltage, 3 × 5.0 mm CT collimation, 0.5 s rotation speed, and X-ray tube currents of 43 Eff.mAs (without Sn filter) and 165 Eff.mAs (with Sn filter). The study found significant dose reductions in both body regions when using the Sn filter and PACT together. For instance, in the upper body region, the combination protocol reduced H˙*(10) from 11.8 µSv/s to 6.1 µSv/s (p < 0.0001) compared to the protocol without using these features. Around 8% of the reduction (about 0.5 µSv/s) is attributed to the Sn filter (p = 0.0005). This approach demonstrates that using the Sn filter along with PACT effectively minimizes radiation exposure for the IR, particularly protecting areas like the head/neck, which can only be insufficiently covered by (standard) radiation protection material.

Spatial disparities in urban park accessibility: Integrating real-time traffic data and housing prices in Ningbo, China

The urban park accessibility is a vital measure of urban livability, indicating either the presence or absence of environmental equity. This study introduces a novel approach by integrating real-time navigation data with traditional accessibility assessments to examine the spatial distribution of parks in Ningbo, China. It evaluates service provision across different transportation modes and park tiers, revealing significant spatial disparities in accessibility. Particularly, it highlights a pronounced polarization, with some city core areas experiencing limited access. Additionally, this research establishes a clear correlation between housing prices and park accessibility, illustrating how urban greenery can influence real estate values. The findings underscore the need for policy interventions to promote equitable access to urban parks and to address potential green gentrification risks. The methodology proposed here offers valuable insights for urban planners and policymakers in scientifically assessing public resource accessibility and planning more equitable urban environments.

Interspecies-chimera machine vision with polarimetry for real-time navigation and anti-glare pattern recognition

Cutting-edge humanoid machine vision merely mimics human systems and lacks polarimetric functionalities that convey the information of navigation and authentic images. Interspecies-chimera vision reserving multiple hosts’ capacities will lead to advanced machine vision. However, implementing the visual functions of multiple species (human and non-human) in one optoelectronic device is still elusive. Here, we develop an optically-controlled polarimetry memtransistor based on a van der Waals heterostructure (ReS2/GeSe2). The device provides polarization sensitivity, nonvolatility, and positive/negative photoconductance simultaneously. The polarimetric measurement can identify celestial polarizations for real-time navigation like a honeybee. Meanwhile, cognitive tasks can be completed like a human by sensing, memory, and synaptic functions. Particularly, the anti-glare recognition with polarimetry saves an order of magnitude energy compared to the traditional humanoid counterpart. This technique promotes the concept of interspecies-chimera visual systems that will leverage the advances of autonomous vehicles, medical diagnoses, intelligent robotics, etc.

Adaptive Navigation Performance Evaluation Method for Civil Aircraft Navigation Systems with Unknown Time-Varying Sensor Noise.

During civil aviation flights, the aircraft needs to accurately monitor the real-time navigation capability and determine whether the onboard navigation system performance meets the required navigation performance (RNP). The airborne flight management system (FMS) uses actual navigation performance (ANP) to quantitatively calculate the uncertainty of aircraft position estimation, and its evaluation accuracy is highly dependent on the position estimation covariance matrix (PECM) provided by the airborne integrated navigation system. This paper proposed an adaptive PECM estimation method based on variational Bayes (VB) to solve the problem of ANP misevaluation, which is caused by the traditional simple ANP model failing to accurately estimate PECM under unknown time-varying noise. Combined with the 3D ANP model proposed in this paper, the accuracy of ANP evaluation can be significantly improved. This enhancement contributes to ensured navigation integrity and operational safety during civil flight.

Research on risk data analysis of hazardous chemical vessels based on stream computing

Abstract To effectively improve the safety supervision and information management level of vessels carrying hazardous chemicals, the maritime management department is promoting the construction of hazardous chemical vessel monitoring systems based on artificial intelligent technology for hazardous chemical terminals and shipping companies in their jurisdiction. It can identify specific risks and transmit the risk data in real-time to the data center. To analyze data in real-time, a risk data analysis platform for hazardous chemical vessels based on stream computing is designed and implemented. The platform acquires vessel automatic identification system (AIS), risk, basic information, and other data. Based on the stream computing of Flink, it evaluates the risk status of vessels in real time, counts the risk data, and displays the results in the form of charts. The analysis of risk data for hazardous chemical vessels helps maritime management departments to track the real-time navigation status of key vessels, timely identify potential safety hazards, and effectively improve the level and efficiency of safety supervision.

Application of Large Language Models and Assessment of Their Ship-Handling Theory Knowledge and Skills for Connected Maritime Autonomous Surface Ships

Maritime transport plays a critical role in global logistics. Compared to road transport, the pace of research and development is much slower for maritime transport. It faces many major challenges, such as busy ports, long journeys, significant accidents, and greenhouse gas emissions. The problems have been exacerbated by recent regional conflicts and increasing international shipping demands. Maritime Autonomous Surface Ships (MASSs) are widely regarded as a promising solution to addressing maritime transport problems with improved safety and efficiency. With advanced sensing and path-planning technologies, MASSs can autonomously understand environments and navigate without human intervention. However, the complex traffic and water conditions and the corner cases are large barriers in the way of MASSs being practically deployed. In this paper, to address the above issues, we investigated the application of Large Language Models (LLMs), which have demonstrated strong generalization abilities. Given the substantial computational demands of LLMs, we propose a framework for LLM-assisted navigation in connected MASSs. In this framework, LLMs are deployed onshore or in remote clouds, to facilitate navigation and provide guidance services for MASSs. Additionally, certain large oceangoing vessels can deploy LLMs locally, to obtain real-time navigation recommendations. To the best of our knowledge, this is the first attempt to apply LLMs to assist with ship navigation. Specifically, MASSs transmit assistance requests to LLMs, which then process these requests and return assistance guidance. A crucial aspect, which has not been investigated in the literature, of this safety-critical LLM-assisted guidance system is the knowledge and safety performance of the LLMs, in regard to ship handling, navigation rules, and skills. To assess LLMs’ knowledge of navigation rules and their qualifications for navigation assistance systems, we designed and conducted navigation theory tests for LLMs, which consisted of more than 1500 multiple-choice questions. These questions were similar to the official theory exams that are used to award the Officer Of the Watch (OOW) certificate based on the Standards of Training, Certification, and Watchkeeping (STCW) for Seafarers. A wide range of LLMs were tested, which included commercial ones from OpenAI and Baidu and an open-source one called ChatGLM, from Tsinghua. Our experimental results indicated that among all the tested LLMs, only GPT-4o passed the tests, with an accuracy of 86%. This suggests that, while the current LLMs possess significant potential in regard to navigation and guidance systems for connected MASSs, further improvements are needed.

USV formation navigation decision-making through hybrid deep reinforcement learning using self-attention mechanism

To address the challenging of balancing Unmanned Surface Vessel (USV) autonomous collision avoidance and formation maintenance in uncertain environments, a formation construction and navigation decision-making strategy based on Hybrid Deep Reinforcement Learning (HDRL) is proposed in this study. The novelty of this study is that: (1) A HDRL training approach is proposed, incorporating diverse DRL for virtual leader and followers, thereby significantly enhancing the decision-making adaptability of USVs formation. (2) The multi-head attention mechanism of decentralized Critic strategy is used to enhance the attentional focus of the HDRL algorithm towards different agents, thereby effectively improving convergence speed. (3) The method employs a meticulously designed hybrid reward function and incorporates the Optimal Reciprocal Collision Avoidance (ORCA) for speed selection, thereby providing optimal speed recommendations based on the current situation. It is worth emphasizing that the method exhibits exceptional performance in terms of success rate, navigation time, average reward value, and other relevant indicators within the simulation. Finally, the method is validated through the utilization of real-time navigation data in the Panama Canal, thereby substantiating the potential engineering applicability.

Clinical study of immediate implant placement in posterior maxillary region assisted by dynamic real-time navigation: A randomized controlled trial

ObjectiveTo evaluate the accuracy of the implant accuracy and clinical effect of navigation-assisted immediate implant placement (IIP) in the posterior maxillary tooth region. MethodsThis study included 60 patients with 96 implants undergoing IIP in the posterior maxillary region from January 2021 to December 2022, stratified into dynamic navigation and freehand implant groups. All clinical indicators, including initial stability, implant deviation (entry point deviation, end point deviation, depth deviation and angle deviation), marginal bone resorption and implant success rate, were systematically recorded. ResultsAll implants were successfully placed with an average torque of (24.38 ± 1.84)N.cm. The mean entry point deviation, apex point deviation and angular deviation in the navigation group were significantly smaller than that of the freehand group (P < 0.05). Marginal bone resorption was significantly less in the navigated group than in the freehand group (P < 0.05). All dental implants were considered an operational success and the mean follow-up time was (27.8 ± 8.4) months. ConclusionThe application of dynamic navigation-assisted immediate implant placement in the maxillary posterior region can achieve good implant accuracy and satisfactory clinical results.

Narrative review of patient-specific 3D visualization and reality technologies in skull base neurosurgery: enhancements in surgical training, planning, and navigation.

Recent advances in medical imaging, computer vision, 3-dimensional (3D) modeling, and artificial intelligence (AI) integrated technologies paved the way for generating patient-specific, realistic 3D visualization of pathological anatomy in neurosurgical conditions. Immersive surgical simulations through augmented reality (AR), virtual reality (VR), mixed reality (MxR), extended reality (XR), and 3D printing applications further increased their utilization in current surgical practice and training. This narrative review investigates state-of-the-art studies, the limitations of these technologies, and future directions for them in the field of skull base surgery. We begin with a methodology summary to create accurate 3D models customized for each patient by combining several imaging modalities. Then, we explore how these models are employed in surgical planning simulations and real-time navigation systems in surgical procedures involving the anterior, middle, and posterior cranial skull bases, including endoscopic and open microsurgical operations. We also evaluate their influence on surgical decision-making, performance, and education. Accumulating evidence demonstrates that these technologies can enhance the visibility of the neuroanatomical structures situated at the cranial base and assist surgeons in preoperative planning and intraoperative navigation, thus showing great potential to improve surgical results and reduce complications. Maximum effectiveness can be achieved in approach selection, patient positioning, craniotomy placement, anti-target avoidance, and comprehension of spatial interrelationships of neurovascular structures. Finally, we present the obstacles and possible future paths for the broader implementation of these groundbreaking methods in neurosurgery, highlighting the importance of ongoing technological advancements and interdisciplinary collaboration to improve the accuracy and usefulness of 3D visualization and reality technologies in skull base surgeries.

Needle tracking in low-resolution ultrasound volumes using deep learning

PurposeClinical needle insertion into tissue, commonly assisted by 2D ultrasound imaging for real-time navigation, faces the challenge of precise needle and probe alignment to reduce out-of-plane movement. Recent studies investigate 3D ultrasound imaging together with deep learning to overcome this problem, focusing on acquiring high-resolution images to create optimal conditions for needle tip detection. However, high-resolution also requires a lot of time for image acquisition and processing, which limits the real-time capability. Therefore, we aim to maximize the US volume rate with the trade-off of low image resolution. We propose a deep learning approach to directly extract the 3D needle tip position from sparsely sampled US volumes.MethodsWe design an experimental setup with a robot inserting a needle into water and chicken liver tissue. In contrast to manual annotation, we assess the needle tip position from the known robot pose. During insertion, we acquire a large data set of low-resolution volumes using a 16 ×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document} 16 element matrix transducer with a volume rate of 4 Hz. We compare the performance of our deep learning approach with conventional needle segmentation.ResultsOur experiments in water and liver show that deep learning outperforms the conventional approach while achieving sub-millimeter accuracy. We achieve mean position errors of 0.54 mm in water and 1.54 mm in liver for deep learning.ConclusionOur study underlines the strengths of deep learning to predict the 3D needle positions from low-resolution ultrasound volumes. This is an important milestone for real-time needle navigation, simplifying the alignment of needle and ultrasound probe and enabling a 3D motion analysis.

Integrating Augmented Reality in Spine Surgery: Redefining Precision with New Technologies.

The integration of augmented reality (AR) in spine surgery marks a significant advancement, enhancing surgical precision and patient outcomes. AR provides immersive, three-dimensional visualizations of anatomical structures, facilitating meticulous planning and execution of spine surgeries. This technology not only improves spatial understanding and real-time navigation during procedures but also aims to reduce surgical invasiveness and operative times. Despite its potential, challenges such as model accuracy, user interface design, and the learning curve for new technology must be addressed. AR's application extends beyond the operating room, offering valuable tools for medical education and improving patient communication and satisfaction. A literature review was conducted by searching PubMed and Scopus databases using keywords related to augmented reality in spine surgery, covering publications from January 2020 to January 2024. In total, 319 articles were identified through the initial search of the databases. After screening titles and abstracts, 11 articles in total were included in the qualitative synthesis. Augmented reality (AR) is becoming a transformative force in spine surgery, enhancing precision, education, and outcomes despite hurdles like technical limitations and integration challenges. AR's immersive visualizations and educational innovations, coupled with its potential synergy with AI and machine learning, indicate a bright future for surgical care. Despite the existing obstacles, AR's impact on improving surgical accuracy and safety marks a significant leap forward in patient treatment and care.

Utilization of artificial intelligence in minimally invasive right adrenalectomy: recognition of anatomical landmarks with deep learning

Background The primary surgical approach for removing adrenal masses is minimally invasive adrenalectomy. Recognition of anatomical landmarks during surgery is critical for minimizing complications. Artificial intelligence-based tools can be utilized to create real-time navigation systems during laparoscopic and robotic right adrenalectomy. In this study, we aimed to develop deep learning models that can identify critical anatomical structures during minimally invasive right adrenalectomy. Methods In this experimental feasibility study, intraoperative videos of 20 patients who underwent minimally invasive right adrenalectomy in a tertiary care center between 2011 and 2023 were analyzed and used to develop an artificial intelligence-based anatomical landmark recognition system. Semantic segmentation of the liver, the inferior vena cava (IVC), and the right adrenal gland were performed. Fifty random images per patient during the dissection phase were extracted from videos. The experiments on the annotated images were performed on two state-of-the-art segmentation models named SwinUNETR and MedNeXt, which are transformer and convolutional neural network (CNN)-based segmentation architectures, respectively. Two loss function combinations, Dice-Cross Entropy and Dice-Focal Loss were experimented with for both of the models. The dataset was split into training and validation subsets with an 80:20 distribution on a patient basis in a 5-fold cross-validation approach. To introduce a sample variability to the dataset, strong-augmentation techniques were performed using intensity modifications and perspective transformations to represent different surgery environment scenarios. The models were evaluated by Dice Similarity Coefficient (DSC) and Intersection over Union (IoU) which are widely used segmentation metrics. For pixelwise classification performance, accuracy, sensitivity and specificity metrics were calculated on the validation subset. Results Out of 20 videos, 1000 images were extracted, and the anatomical landmarks (liver, IVC, and right adrenal gland) were annotated. Randomly distributed 800 images and 200 images were selected for the training and validation subsets, respectively. Our benchmark results show that the utilization of Dice-Cross Entropy Loss with the transformer-based SwinUNETR model achieved 78.37%, whereas the CNN-based MedNeXt model reached a 77.09% mDSC score. Conversely, MedNeXt reaches a higher mIoU score of 63.71% than SwinUNETR by 62.10% on a three-region prediction task. Conclusion Artificial intelligence-based systems can predict anatomical landmarks with high performance in minimally invasive right adrenalectomy. Such tools can later be used to create real-time navigation systems during surgery in the near future.