Median Target Registration Error Research Articles

Immunohistochemistry annotations enhance AI identification of lymphocytes and neutrophils in digitized H&E slides from inflammatory bowel disease

Background and ObjectiveHistologic assessment of the immune infiltrate in H&E slides is vital in diagnosing and managing inflammatory bowel diseases, but these assessments are subjective and time-consuming even for those with expertise. The development of deep learning models to aid in these assessments has been limited by the paucity of image data with reliably annotated immune cells available for training. MethodsTo address these challenges, we developed a pipeline that automates the neutrophil and lymphocyte labeling in ROIs from digital H&E slides. The data included ROIs extracted from 19 digitized H&E slides and the same slides restained with immunohistochemistry. Our pipeline first delineates each nucleus in H&E ROIs. Using the colorimetric features of the immunohistochemical stains (red: neutrophils, green: lymphocytes) in the immunohistochemistry ROIs, each cell was labeled as a neutrophil, a lymphocyte, or another cell. The labels were then transferred to the corresponding H&E ROIs by image registration, and the ROI registration accuracy was assessed by the median target registration error resulting in a labeled dataset. The newly formed dataset (NeuLy-IHC) comprising 519 ROIs with 235,256 labeled cells(74,339 lymphocytes, 16,326 neutrophils and 144,591 other cells) was used to train theHoVer-Net(NeuLy) model. The performance of HoVer-Net(NeuLy) measured by DICE coefficient (segmentation accuracy) and F1-scores (classification accuracy), was compared to those achieved by HoVer-Net(MoNuSAC) and SMILE(MoNuSAC) publicly available models trained on cancer-containing ROIs from the MoNuSAC dataset with manual cell labeling and pathologists’ annotations. ResultsThe 1.0 μm median target registration error of ROIs observed was low demonstrating robust transferring of cellular labels from immunohistochemistry ROIs to H&E ROIs. In the test set comprising 76 NeuLy-IHC and 78 MoNuSAC ROIs, the HoVer-Net(NeuLy) achieved a DICE coefficient of 0.861 and F1-sores of 0.827, 0.838, and 0.828, for neutrophils, lymphocytes, and other cells, respectively, outperforming the HoVer-Net(MoNuSAC)'s and SMILE(MoNuSAC)’s DICE coefficient and F1 scores for each cell category. ConclusionsWe attribute the improved performance of HoVer-Net(NeuLy) to the larger number of immune cells in the NeuLy-IHC dataset (in total 5x more, including 21x more neutrophils) than in the MoNuSAC dataset. Despite being trained on data from inflammatory bowel disease specimens, our model maintained robust performance when tested on previously unseen data derived from cancer specimens. The NeuLy-IHC set provides opportunities for training accurate models to quantify the inflammatory infiltrate in digital histologic slides.

Multi-modal 3-Dimensional Visualization of Pediatric Neuroblastoma: Aiding Surgical Planning Beyond Anatomical Information

BackgroundPatient-specific 3D models of neuroblastoma and relevant anatomy are useful tools for surgical planning. However, these models do not represent the heterogenous biology of neuroblastoma. This heterogeneity is visualized with the ADC and 123I-MIGB-SPECT-CT imaging. Combining these multi-modal data into preoperative 3D heatmaps, may allow differentiation of the areas of vital and non-vital tumor tissue. We developed a workflow to create multi-modal preoperative 3D models for neuroblastoma surgery. MethodsWe included 7 patients who underwent neuroblastoma surgery between 2022 and 2023. We developed 3D models based on the contrast enhanced T1-weighted MRI scans. Subsequently, we aligned the corresponding ADC and 123I-MIBG-SPECT-CT images using rigid transformation. We estimated registration precision using the Dice score and the target registration error (TRE). 3D heatmaps were computed based on ADC and 123I-MIBG uptake. ResultsThe registration algorithm had a median Dice score of 0.81 (0.75–0.90) for ADC and 0.77 (0.65–0.91) for 123I-MIBG-SPECT. For the ADC registration, the median TRE of renal vessels was 4.90 mm (0.86–10.18) and of the aorta 4.67 mm (1.59–12.20). For the 123I -MIBG-SPECT imaging the TRE of the renal vessels was 5.52 mm (1.71–10.97) and 5.28 mm (3.33–16.77) for the aorta. ConclusionsWe successfully developed a registration workflow to create multi-modal 3D models which allows the surgeon to visualize the tumor and its biological behavior in relation to the surrounding tissue. Future research will include linking of pathological results to imaging data, to validate these multi-modal 3D models. Level of EvidenceLevel IV. Type of StudyClinical Research.

Endovascular navigation in patients: vessel-based registration of electromagnetic tracking to preoperative images

Electromagnetic tracking of instruments combined with preoperative images can supplement fluoroscopy for guiding endovascular aortic repair (EVAR). The aim of this study was to evaluate the in-vivo accuracy of a vessel-based registration algorithm for matching electromagnetically tracked positions of an endovascular instrument to preoperative computed tomography angiography. Five patients undergoing elective EVAR were included, and a clinically available semi-automatic 3D–3D registration algorithm, based on similarity measures computed over the entire image, was used for reference. Accuracy was reported as target registration error (TRE) evaluated in manually selected anatomic landmarks on bony structures, placed close to the volume-of-interest. The median TRE was 8.2 mm (range: 7.1 mm to 16.1 mm) for the vessel-based registration algorithm, compared to 2.2 mm (range: 1.8 mm to 3.7 mm) for the reference algorithm. This illustrates that registration based on intraoperative electromagnetic tracking is feasible, but the accuracy must be improved before clinical use.

A Fully Differentiable Framework for 2D/3D Registration and the Projective Spatial Transformers.

Image-based 2D/3D registration is a critical technique for fluoroscopic guided surgical interventions. Conventional intensity-based 2D/3D registration approa- ches suffer from a limited capture range due to the presence of local minima in hand-crafted image similarity functions. In this work, we aim to extend the 2D/3D registration capture range with a fully differentiable deep network framework that learns to approximate a convex-shape similarity function. The network uses a novel Projective Spatial Transformer (ProST) module that has unique differentiability with respect to 3D pose parameters, and is trained using an innovative double backward gradient-driven loss function. We compare the most popular learning-based pose regression methods in the literature and use the well-established CMAES intensity-based registration as a benchmark. We report registration pose error, target registration error (TRE) and success rate (SR) with a threshold of 10mm for mean TRE. For the pelvis anatomy, the median TRE of ProST followed by CMAES is 4.4mm with a SR of 65.6% in simulation, and 2.2mm with a SR of 73.2% in real data. The CMAES SRs without using ProST registration are 28.5% and 36.0% in simulation and real data, respectively. Our results suggest that the proposed ProST network learns a practical similarity function, which vastly extends the capture range of conventional intensity-based 2D/3D registration. We believe that the unique differentiable property of ProST has the potential to benefit related 3D medical imaging research applications. The source code is available at https://github.com/gaocong13/Projective-Spatial-Transformers.

Automatic registration with continuous pose updates for marker-less surgical navigation in spine surgery

Established surgical navigation systems for pedicle screw placement have been proven to be accurate, but still reveal limitations in registration or surgical guidance. Registration of preoperative data to the intraoperative anatomy remains a time-consuming, error-prone task that includes exposure to harmful radiation. Surgical guidance through conventional displays has well-known drawbacks, as information cannot be presented in-situ and from the surgeon’s perspective. Consequently, radiation-free and more automatic registration methods with subsequent surgeon-centric navigation feedback are desirable. In this work, we present a marker-less approach that automatically solves the registration problem for lumbar spinal fusion surgery in a radiation-free manner. A deep neural network was trained to segment the lumbar spine and simultaneously predict its orientation, yielding an initial pose for preoperative models, which then is refined for each vertebra individually and updated in real-time with GPU acceleration while handling surgeon occlusions. An intuitive surgical guidance is provided thanks to the integration into an augmented reality based navigation system. The registration method was verified on a public dataset with a median of 100% successful registrations, a median target registration error of 2.7 mm, a median screw trajectory error of 1.6°and a median screw entry point error of 2.3 mm. Additionally, the whole pipeline was validated in an ex-vivo surgery, yielding a 100% screw accuracy and a median target registration error of 1.0 mm. Our results meet clinical demands and emphasize the potential of RGB-D data for fully automatic registration approaches in combination with augmented reality guidance.

Ultrasound guidance in navigated liver surgery: toward deep-learning enhanced compensation of deformation and organ motion.

Accuracy of image-guided liver surgery is challenged by deformation of the liver during the procedure. This study aims at improving navigation accuracy by using intraoperative deep learning segmentation and nonrigid registration of hepatic vasculature from ultrasound (US) images to compensate for changes in liver position and deformation. This was a single-center prospective study of patients with liver metastases from any origin. Electromagnetic tracking was used to follow US and liver movement. A preoperative 3D model of the liver, including liver lesions, and hepatic and portal vasculature, was registered with the intraoperative organ position. Hepatic vasculature was segmented using a reduced 3D U-Net and registered to preoperative imaging after initial alignment followed by nonrigid registration. Accuracy was assessed as Euclidean distance between the tumor center imaged in the intraoperative US and the registered preoperative image. Median target registration error (TRE) after initial alignment was 11.6mm in 25 procedures and improved to 6.9mm after nonrigid registration (p = 0.0076). The number of TREs above 10mm halved from 16 to 8 after nonrigid registration. In 9 cases, registration was performed twice after failure of the first attempt. The first registration cycle was completed in median 11min (8:00-18:45min) and a second in 5min (2:30-10:20min). This novel registration workflow using automatic vascular detection and nonrigid registration allows to accurately localize liver lesions. Further automation in the workflow is required in initial alignment and classification accuracy.

Cross-Modality Image Registration Using a Training-Time Privileged Third Modality.

In this work, we consider the task of pairwise cross-modality image registration, which may benefit from exploiting additional images available only at training time from an additional modality that is different to those being registered. As an example, we focus on aligning intra-subject multiparametric Magnetic Resonance (mpMR) images, between T2-weighted (T2w) scans and diffusion-weighted scans with high b-value (DWI [Formula: see text]). For the application of localising tumours in mpMR images, diffusion scans with zero b-value (DWI [Formula: see text]) are considered easier to register to T2w due to the availability of corresponding features. We propose a learning from privileged modality algorithm, using a training-only imaging modality DWI [Formula: see text], to support the challenging multi-modality registration problems. We present experimental results based on 369 sets of 3D multiparametric MRI images from 356 prostate cancer patients and report, with statistical significance, a lowered median target registration error of 4.34 mm, when registering the holdout DWI [Formula: see text] and T2w image pairs, compared with that of 7.96 mm before registration. Results also show that the proposed learning-based registration networks enabled efficient registration with comparable or better accuracy, compared with a classical iterative algorithm and other tested learning-based methods with/without the additional modality. These compared algorithms also failed to produce any significantly improved alignment between DWI [Formula: see text] and T2w in this challenging application.

Assessing the accuracy of a new 3D2D registration algorithm based on a non-invasive skin marker model for navigated spine surgery

PurposeWe assessed the accuracy of a new 3D2D registration algorithm to be used for navigated spine surgery and explored anatomical and radiologic parameters affecting the registration accuracy. Compared to existing 3D2D registration algorithms, the algorithm does not need bone-mounted or table-mounted instruments for registration. Neither does the intraoperative imaging device have to be tracked or calibrated.MethodsThe rigid registration algorithm required imaging data (a pre-existing CT scan (3D) and two angulated fluoroscopic images (2D)) to register positions of vertebrae in 3D and is based on non-invasive skin markers. The algorithm registered five adjacent vertebrae and was tested in the thoracic and lumbar spine from three human cadaveric specimens. The registration accuracy was calculated for each registered vertebra and measured with the target registration error (TRE) in millimeters. We used multivariable analysis to identify parameters independently affecting the algorithm’s accuracy such as the angulation between the two fluoroscopic images (between 40° and 90°), the detector-skin distance, the number of skin markers applied, and waist circumference.ResultsThe algorithm registered 780 vertebrae with a median TRE of 0.51 mm [interquartile range 0.32–0.73 mm] and a maximum TRE of 2.06 mm. The TRE was most affected by the angulation between the two fluoroscopic images obtained (p < 0.001): larger angulations resulted in higher accuracy. The algorithm was more accurate in thoracic vertebrae (p = 0.004) and in the specimen with the smallest waist circumference (p = 0.003). The algorithm registered all five adjacent vertebrae with similar accuracy.ConclusionWe studied the accuracy of a new 3D2D registration algorithm based on non-invasive skin markers. The algorithm registered five adjacent vertebrae with similar accuracy in the thoracic and lumbar spine and showed a maximum target registration error of approximately 2 mm. To further evaluate its potential for navigated spine surgery, the algorithm may now be integrated into a complete navigation system.

Joint synthesis and registration network for deformable MR-CBCT image registration for neurosurgical guidance

Objective. The accuracy of navigation in minimally invasive neurosurgery is often challenged by deep brain deformations (up to 10 mm due to egress of cerebrospinal fluid during neuroendoscopic approach). We propose a deep learning-based deformable registration method to address such deformations between preoperative MR and intraoperative CBCT. Approach. The registration method uses a joint image synthesis and registration network (denoted JSR) to simultaneously synthesize MR and CBCT images to the CT domain and perform CT domain registration using a multi-resolution pyramid. JSR was first trained using a simulated dataset (simulated CBCT and simulated deformations) and then refined on real clinical images via transfer learning. The performance of the multi-resolution JSR was compared to a single-resolution architecture as well as a series of alternative registration methods (symmetric normalization (SyN), VoxelMorph, and image synthesis-based registration methods). Main results. JSR achieved median Dice coefficient (DSC) of 0.69 in deep brain structures and median target registration error (TRE) of 1.94 mm in the simulation dataset, with improvement from single-resolution architecture (median DSC = 0.68 and median TRE = 2.14 mm). Additionally, JSR achieved superior registration compared to alternative methods—e.g. SyN (median DSC = 0.54, median TRE = 2.77 mm), VoxelMorph (median DSC = 0.52, median TRE = 2.66 mm) and provided registration runtime of less than 3 s. Similarly in the clinical dataset, JSR achieved median DSC = 0.72 and median TRE = 2.05 mm. Significance. The multi-resolution JSR network resolved deep brain deformations between MR and CBCT images with performance superior to other state-of-the-art methods. The accuracy and runtime support translation of the method to further clinical studies in high-precision neurosurgery.

Pre-Clinical Development of Robot-Assisted Ventriculoscopy for 3D Image Reconstruction and Guidance of Deep Brain Neurosurgery.

Conventional neuro-navigation can be challenged in targeting deep brain structures via transventricular neuroendoscopy due to unresolved geometric error following soft-tissue deformation. Current robot-assisted endoscopy techniques are fairly limited, primarily serving to planned trajectories and provide a stable scope holder. We report the implementation of a robot-assisted ventriculoscopy (RAV) system for 3D reconstruction, registration, and augmentation of the neuroendoscopic scene with intraoperative imaging, enabling guidance even in the presence of tissue deformation and providing visualization of structures beyond the endoscopic field-of-view. Phantom studies were performed to quantitatively evaluate image sampling requirements, registration accuracy, and computational runtime for two reconstruction methods and a variety of clinically relevant ventriculoscope trajectories. A median target registration error of 1.2 mm was achieved with an update rate of 2.34 frames per second, validating the RAV concept and motivating translation to future clinical studies.

Improving deformable image registration with point metric and masking technique for postoperative breast cancer radiotherapy.

Deformable image registration (DIR) is increasingly used for target volume definition in radiotherapy. However, this method is challenging for postoperative breast cancer patients due to the large deformations and non-correspondence caused by tumor resection and clip insertion. In this study, an improved B-splines based DIR method was developed to address this issue for higher registration accuracy. The conventional B-splines based DIR method was improved with the introduction of point metric and masking technique. The point metric minimizes the distance between 2 point sets with known correspondence for regularization of intensity-based B-splines registration. The masking technique reduces the influence of non-corresponding regions in breast computed tomography (CT) images. Two sets of CT images before and after breast surgery were used for image registration. One set was the diagnostic CT image acquired before surgery, and another set was the planning CT image acquired after surgery for breast cancer radiotherapy. A total of 26 sets of CT images from 13 patients were collected retrospectively for the test. The improved DIR method's registration accuracy was evaluated by target registration error (TRE), the Jacobian determinant, and visual assessment. For soft tissue, the difference in the median TRE between the improved DIR method and the conventional DIR method was statistically significant (2.27 vs. 5.88, P<0.05). The Jacobian determinant of the deformation field was positive for all patients. For visual assessment, the improved DIR method with point metric achieved better matching for soft tissue. The improved DIR method's registration accuracy was higher than the conventional DIR method based on the preliminary results. With point metric and masking technique, the influence of large deformations and non-correspondence on registration between pre- and post-operative CT images can be effectively reduced. Therefore, this method provides a feasible way for target volume definition in postoperative breast cancer radiotherapy treatment planning.

Evaluation of an Ultrasound-Based Navigation System for Spine Neurosurgery: A Porcine Cadaver Study.

BackgroundWith the growing incidence of patients receiving surgical treatment for spinal metastatic tumours, there is a need for developing cost-efficient and radiation-free alternatives for spinal interventions. In this paper, we evaluate the capabilities and limitations of an image-guided neurosurgery (IGNS) system that uses intraoperative ultrasound (iUS) imaging for guidance.MethodsUsing a lumbosacral section of a porcine cadaver, we explored the impact of CT image resolution, ultrasound depth and ultrasound frequency on system accuracy, robustness and effectiveness. Preoperative CT images with an isotropic resolution of , and were acquired. During surgery, vertebrae L1 to L6 were exposed. For each vertebra, five iUS scans were acquired using two depth parameters (5 cm and 7 cm) and two frequencies (6 MHz and 12 MHz). A total of 120 acquisition trials were evaluated. Ultrasound-based registration performance is compared to the standard alignment procedure using intraoperative CT. We report target registration error (TRE) and computation time. In addition, the scans’ trajectories were analyzed to identify vertebral regions that provide the most relevant features for the alignment.ResultsFor all acquisitions, the median TRE ranged from 1.42 mm to 1.58 mm and the overall computation time was 9.04 s ± 1.58 s. Fourteen out of 120 iUS acquisitions (11.66%) yielded a level-to-level mismatch (and these are included in the accuracy measurements reported). No significant effect on accuracy was found with CT resolution (F (2,10) = 1.70, p = 0.232), depth (F (1,5) = 0.22, p= 0.659) nor frequency (F (1,5) = 1.02, p = 0.359). While misalignment increases linearly with the distance from the imaged vertebra, accuracy was satisfactory for directly adjacent levels. A significant relationship was found between iUS scan coverage of laminae and articular processes, and accuracy.ConclusionIntraoperative ultrasound can be used for spine surgery neuronavigation. We demonstrated that the IGNS system yield acceptable accuracy and high efficiency compared to the standard CT-based navigation procedure. The flexibility of the iUS acquisitions can have repercussions on the system performance, which are not fully identified. Further investigation is needed to understand the relationship between iUS acquisition and alignment performance.

Long-length tomosynthesis and 3D-2D registration for intraoperative assessment of spine instrumentation

Purpose. A system for long-length intraoperative imaging is reported based on longitudinal motion of an O-arm gantry featuring a multi-slot collimator. We assess the utility of long-length tomosynthesis and the geometric accuracy of 3D image registration for surgical guidance and evaluation of long spinal constructs. Methods. A multi-slot collimator with tilted apertures was integrated into an O-arm system for long-length imaging. The multi-slot projective geometry leads to slight view disparity in both long-length projection images (referred to as ‘line scans’) and tomosynthesis ‘slot reconstructions’ produced using a weighted-backprojection method. The radiation dose for long-length imaging was measured, and the utility of long-length, intraoperative tomosynthesis was evaluated in phantom and cadaver studies. Leveraging the depth resolution provided by parallax views, an algorithm for 3D-2D registration of the patient and surgical devices was adapted for registration with line scans and slot reconstructions. Registration performance using single-plane or dual-plane long-length images was evaluated and compared to registration accuracy achieved using standard dual-plane radiographs. Results. Longitudinal coverage of ∼50–64 cm was achieved with a single long-length slot scan, providing a field-of-view (FOV) up to (40 × 64) cm2, depending on patient positioning. The dose-area product (reference point air kerma × x-ray field area) for a slot scan ranged from ∼702–1757 mGy·cm2, equivalent to ∼2.5 s of fluoroscopy and comparable to other long-length imaging systems. Long-length scanning produced high-resolution tomosynthesis reconstructions, covering ∼12–16 vertebral levels. 3D image registration using dual-plane slot reconstructions achieved median target registration error (TRE) of 1.2 mm and 0.6° in cadaver studies, outperforming registration to dual-plane line scans (TRE = 2.8 mm and 2.2°) and radiographs (TRE = 2.5 mm and 1.1°). 3D registration using single-plane slot reconstructions leveraged the ∼7–14° angular separation between slots to achieve median TRE ∼2 mm and <2° from a single scan. Conclusion. The multi-slot configuration provided intraoperative visualization of long spine segments, facilitating target localization, assessment of global spinal alignment, and evaluation of long surgical constructs. 3D-2D registration to long-length tomosynthesis reconstructions yielded a promising means of guidance and verification with accuracy exceeding that of 3D-2D registration to conventional radiographs.

Drill-mounted video guidance for orthopaedic trauma surgery.

Purpose: Percutaneous fracture fixation is a challenging procedure that requires accurate interpretation of fluoroscopic images to insert guidewires through narrow bone corridors. We present a guidance system with a video camera mounted onboard the surgical drill to achieve real-time augmentation of the drill trajectory in fluoroscopy and/or CT. Approach: The camera was mounted on the drill and calibrated with respect to the drill axis. Markers identifiable in both video and fluoroscopy are placed about the surgical field and co-registered by feature correspondences. If available, a preoperative CT can also be co-registered by 3D-2D image registration. Real-time guidance is achieved by virtual overlay of the registered drill axis on fluoroscopy or in CT. Performance was evaluated in terms of target registration error (TRE), conformance within clinically relevant pelvic bone corridors, and runtime. Results: Registration of the drill axis to fluoroscopy demonstrated median TRE of 0.9mm and 2.0deg when solved with two views (e.g., anteroposterior and lateral) and five markers visible in both video and fluoroscopy-more than sufficient to provide Kirschner wire (K-wire) conformance within common pelvic bone corridors. Registration accuracy was reduced when solved with a single fluoroscopic view ( and 2.7deg) but was also sufficient for K-wire conformance within pelvic bone corridors. Registration was robust with as few as four markers visible within the field of view. Runtime of the initial implementation allowed fluoroscopy overlay and/or 3D CT navigation with freehand manipulation of the drill up to . Conclusions: A drill-mounted video guidance system was developed to assist with K-wire placement. Overall workflow is compatible with fluoroscopically guided orthopaedic trauma surgery and does not require markers to be placed in preoperative CT. The initial prototype demonstrates accuracy and runtime that could improve the accuracy of K-wire placement, motivating future work for translation to clinical studies.

Multistep, automatic and nonrigid image registration method for histology samples acquired using multiple stains

The use of multiple dyes during histological sample preparation can reveal distinct tissue properties. However, since the slide preparation differs for each dye, the tissue slides are being deformed and a nonrigid registration is required before further processing. The registration of histology images is complicated because of: (i) a high resolution of histology images, (ii) complex, large, nonrigid deformations, (iii) difference in the appearance and partially missing data due to the use of multiple dyes. In this work, we propose a multistep, automatic, nonrigid image registration method dedicated to histology samples acquired with multiple stains. The proposed method consists of a feature-based affine registration, an exhaustive rotation alignment, an iterative, intensity-based affine registration, and a nonrigid alignment based on modality independent neighbourhood descriptor coupled with the Demons algorithm. A dedicated failure detection mechanism is proposed to make the method fully automatic, without the necessity of any manual interaction. The described method was proposed by the AGH team during the Automatic Non-rigid Histological Image Registration (ANHIR) challenge. The ANHIR dataset consists of 481 image pairs annotated by histology experts. Moreover, the ANHIR challenge submissions were evaluated using an independent, server-side evaluation tool. The main evaluation criteria was the target registration error normalized by the image diagonal. The median of median target registration error is below 0.19%. The proposed method is currently the second-best in terms of the average ranking of median target registration error, without statistically significant differences compared to the top-ranked method. We provide an open access to the method software and used parameters, making the results fully reproducible.

Validation and Precision of Mixed Reality Technology in Baha Attract Implant Surgery

To test the feasibility of image-guided Baha Attract implant surgery with mixed reality (MR) in the form of the HoloLens to visualize critical structures and facilitate precise Baha implant placement. A cadaveric case study of bilateral Baha Attract implant approaches was conducted using Star Atlas MR three-dimensional (3D) medical interaction system guidance at the Otolaryngology Department of PUMCH, Beijing, China. The accuracy of visual surface registration was determined by the target registration error (TRE) between the predefined points on the preoperative 3D holographic Baha Attract implant model and the postoperatively reconstructed 3D model. Bilateral Baha Attract implantation was completed successfully for all four cadaveric heads using the Star Atlas MR 3D medical interaction system with the HoloLens. The preoperative 3D digital model characteristics (including bone quality and thickness and avoidance of cranial vessels, air cells, and cranial sutures) corresponded well with the 3D model of the actual implantation reconstructed postoperatively. The median TRE of our system was 2.97 mm (ranging from 1.98 to 4.58 mm) in terms of distance and 2.76 degrees (ranging from 0.59 to 6.4 degrees) in terms of angle. Applying MR technology in the form of the HoloLens in Baha Attract implant surgery is feasible and could improve the accuracy of the surgery. The described MR system for Baha Attract implantation has the potential to improve the surgeon's confidence, as well as the surgical safety, efficiency, and precision.

Open-source software for ultrasound-based guidance in spinal fusion surgery

Spinal instrumentation and surgical manipulations may cause loss of navigation accuracy requiring an efficient re-alignment of the patient anatomy with pre-operative images during surgery. While intra-operative ultrasound (iUS) guidance has shown clear potential to reduce surgery time, compared with clinical computed tomography (CT) guidance, rapid registration aiming to correct for patient misalignment has not been addressed. In this article, we present an open-source platform for pedicle screw navigation using iUS imaging. The alignment method is based on rigid registration of CT to iUS vertebral images and has been designed for fast and fully automatic patient re-alignment in the operating room. Two steps are involved: first, we use the iUS probe's trajectory to achieve an initial coarse registration; then, the registration transform is refined by simultaneously optimizing gradient orientation alignment and mean of iUS intensities passing through the CT-defined posterior surface of the vertebra. We evaluated our approach on a lumbosacral section of a porcine cadaver with seven vertebral levels. We achieved a median target registration error of 1.47 mm (100% success rate, defined by a target registration error <2 mm) when applying the probe's trajectory initial alignment. The approach exhibited high robustness to partial visibility of the vertebra with success rates of 89.86% and 88.57% when missing either the left or right part of the vertebra and robustness to initial misalignments with a success rate of 83.14% for random starts within ±20° rotation and ±20 mm translation. Our graphics processing unit implementation achieves an efficient registration time under 8 s, which makes the approach suitable for clinical application.

Structure guided deformable image registration for treatment planning CT and post stereotactic body radiation therapy (SBRT) Primovist® (Gd-EOB-DTPA) enhanced MRI.

The purpose of this study was to assess the performance of structure‐guided deformable image registration (SG‐DIR) relative to rigid registration and DIR using TG‐132 recommendations. This assessment was performed for image registration of treatment planning computed tomography (CT) and magnetic resonance imaging (MRI) scans with Primovist® contrast agent acquired post stereotactic body radiation therapy (SBRT). SBRT treatment planning CT scans and posttreatment Primovist® MRI scans were obtained for 14 patients. The liver was delineated on both sets of images and matching anatomical landmarks were chosen by a radiation oncologist. Rigid registration, DIR, and two types of SG‐DIR (using liver contours only; and using liver structures along with anatomical landmarks) were performed for each set of scans. TG‐132 recommended metrics were estimated which included Dice Similarity Coefficient (DSC), Mean Distance to Agreement (MDA), Target Registration Error (TRE), and Jacobian determinant. Statistical analysis was performed using Wilcoxon Signed Rank test. The median (range) DSC for rigid registration was 0.88 (0.77–0.89), 0.89 (0.81–0.93) for DIR, and 0.90 (0.86–0.94) for both types of SG‐DIR tested in this study. The median MDA was 4.8 mm (3.7–6.8 mm) for rigid registration, 3.4 mm (2.4–8.7 mm) for DIR, 3.2 mm (2.0–5.2 mm) for SG‐DIR where liver structures were used to guide the registration, and 2.8 mm (2.1–4.2 mm) for the SG‐DIR where liver structures and anatomical landmarks were used to guide the registration. The median TRE for rigid registration was 7.2 mm (0.5–23 mm), 6.8 mm (0.7–30.7 mm) for DIR, 6.1 mm (1.1–20.5 mm) for the SG‐DIR guided by only the liver structures, and 4.1 mm (0.8–19.7 mm) for SG‐DIR guided by liver contours and anatomical landmarks. The SG‐DIR shows higher liver conformality as per TG‐132 metrics and lowest TRE compared to rigid registration and DIR in Velocity AI software for the purpose of registering treatment planning CT and post‐SBRT MRI for the liver region. It was found that TRE decreases when liver contours and corresponding anatomical landmarks guide SG‐DIR.

A momentum-based diffeomorphic demons framework for deformable MR-CT image registration

Neuro-navigated procedures require a high degree of geometric accuracy but are subject to geometric error from complex deformation in the deep brain—e.g. regions about the ventricles due to egress of cerebrospinal fluid (CSF) upon neuroendoscopic approach or placement of a ventricular shunt. We report a multi-modality, diffeomorphic, deformable registration method using momentum-based acceleration of the Demons algorithm to solve the transformation relating preoperative MRI and intraoperative CT as a basis for high-precision guidance. The registration method (pMI-Demons) extends the mono-modality, diffeomorphic form of the Demons algorithm to multi-modality registration using pointwise mutual information (pMI) as a similarity metric. The method incorporates a preprocessing step to nonlinearly stretch CT image values and incorporates a momentum-based approach to accelerate convergence. Registration performance was evaluated in phantom and patient images: first, the sensitivity of performance to algorithm parameter selection (including update and displacement field smoothing, histogram stretch, and the momentum term) was analyzed in a phantom study over a range of simulated deformations; and second, the algorithm was applied to registration of MR and CT images for four patients undergoing minimally invasive neurosurgery. Performance was compared to two previously reported methods (free-form deformation using mutual information (MI-FFD) and symmetric normalization using mutual information (MI-SyN)) in terms of target registration error (TRE), Jacobian determinant (J), and runtime. The phantom study identified optimal or nominal settings of algorithm parameters for translation to clinical studies. In the phantom study, the pMI-Demons method achieved comparable registration accuracy to the reference methods and strongly reduced outliers in TRE (p 0.001 in Kolmogorov–Smirnov test). Similarly, in the clinical study: median TRE = 1.54 mm (0.83–1.66 mm interquartile range, IQR) for pMI-Demons compared to 1.40 mm (1.02–1.67 mm IQR) for MI-FFD and 1.64 mm (0.90–1.92 mm IQR) for MI-SyN. The pMI-Demons and MI-SyN methods yielded diffeomorphic transformations (J > 0) that preserved topology, whereas MI-FFD yielded unrealistic (J < 0) deformations subject to tissue folding and tearing. Momentum-based acceleration gave a ~35% speedup of the pMI-Demons method, providing registration runtime of 10.5 min (reduced to 2.2 min on GPU), compared to 15.5 min for MI-FFD and 34.7 min for MI-SyN. The pMI-Demons method achieved registration accuracy comparable to MI-FFD and MI-SyN, maintained diffeomorphic transformation similar to MI-SyN, and accelerated runtime in a manner that facilitates translation to image-guided neurosurgery.

Weakly-supervised convolutional neural networks for multimodal image registration.

One of the fundamental challenges in supervised learning for multimodal image registration is the lack of ground-truth for voxel-level spatial correspondence. This work describes a method to infer voxel-level transformation from higher-level correspondence information contained in anatomical labels. We argue that such labels are more reliable and practical to obtain for reference sets of image pairs than voxel-level correspondence. Typical anatomical labels of interest may include solid organs, vessels, ducts, structure boundaries and other subject-specific ad hoc landmarks. The proposed end-to-end convolutional neural network approach aims to predict displacement fields to align multiple labelled corresponding structures for individual image pairs during the training, while only unlabelled image pairs are used as the network input for inference. We highlight the versatility of the proposed strategy, for training, utilising diverse types of anatomical labels, which need not to be identifiable over all training image pairs. At inference, the resulting 3D deformable image registration algorithm runs in real-time and is fully-automated without requiring any anatomical labels or initialisation. Several network architecture variants are compared for registering T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients. A median target registration error of 3.6 mm on landmark centroids and a median Dice of 0.87 on prostate glands are achieved from cross-validation experiments, in which 108 pairs of multimodal images from 76 patients were tested with high-quality anatomical labels.