Image-based Registration Research Articles

A database of the healthy human spinal cord morphometry in the PAM50 template space

Abstract Measures of spinal cord morphometry computed from magnetic resonance images serve as relevant prognostic biomarkers for a range of spinal cord pathologies, including traumatic and non-traumatic spinal cord injury and neurodegenerative diseases. However, interpreting these imaging biomarkers is difficult due to considerable intra- and inter-subject variability. Yet, there is no clear consensus on a normalization method that would help reduce this variability and more insights into the distribution of these morphometrics are needed. In this study, we computed a database of normative values for six commonly used measures of spinal cord morphometry: cross-sectional area, anteroposterior diameter, transverse diameter, compression ratio, eccentricity, and solidity. Normative values were computed from a large open-access dataset of healthy adult volunteers (N = 203) and were brought to the common space of the PAM50 spinal cord template using a newly proposed normalization method based on linear interpolation. Compared to traditional image-based registration, the proposed normalization approach does not involve image transformations and, therefore, does not introduce distortions of spinal cord anatomy. This is a crucial consideration in preserving the integrity of the spinal cord anatomy in conditions such as spinal cord injury. This new morphometric database allows researchers to normalize based on sex and age, thereby minimizing inter-subject variability associated with demographic and biological factors. The proposed methodology is open-source and accessible through the Spinal Cord Toolbox (SCT) v6.0 and higher.

Artificial intelligence reveals features associated with breast cancer neoadjuvant chemotherapy responses from multi-stain histopathologic images

Advances in computational algorithms and tools have made the prediction of cancer patient outcomes using computational pathology feasible. However, predicting clinical outcomes from pre-treatment histopathologic images remains a challenging task, limited by the poor understanding of tumor immune micro-environments. In this study, an automatic, accurate, comprehensive, interpretable, and reproducible whole slide image (WSI) feature extraction pipeline known as, IMage-based Pathological REgistration and Segmentation Statistics (IMPRESS), is described. We used both H&E and multiplex IHC (PD-L1, CD8+, and CD163+) images, investigated whether artificial intelligence (AI)-based algorithms using automatic feature extraction methods can predict neoadjuvant chemotherapy (NAC) outcomes in HER2-positive (HER2+) and triple-negative breast cancer (TNBC) patients. Features are derived from tumor immune micro-environment and clinical data and used to train machine learning models to accurately predict the response to NAC in breast cancer patients (HER2+ AUC = 0.8975; TNBC AUC = 0.7674). The results demonstrate that this method outperforms the results trained from features that were manually generated by pathologists. The developed image features and algorithms were further externally validated by independent cohorts, yielding encouraging results, especially for the HER2+ subtype.

Supine magnetic resonance image registration for breast surgery: insights on material mechanics.

Breast conserving surgery (BCS) is a common procedure for early-stage breast cancer patients. Supine preoperative magnetic resonance (MR) breast imaging for visualizing tumor location and extent, while not standard for procedural guidance, is being explored since it more closely represents the surgical presentation compared to conventional diagnostic imaging positions. Despite this preoperative imaging position, deformation is still present between the supine imaging and surgical state. As a result, a fast and accurate image-to-physical registration approach is needed to realize image-guided breast surgery. In this study, three registration methods were investigated on healthy volunteers' breasts ( ) with the supine arm-down position simulating preoperative imaging and supine arm-up position simulating intraoperative presentation. The registration methods included (1)point-based rigid registration using synthetic fiducials, (2)nonrigid biomechanical model-based registration using sparse data, and (3)a data-dense three-dimensional diffeomorphic image-based registration from the Advanced Normalization Tools (ANTs) repository. Additionally, deformation metrics (volume change and anisotropy) were calculated from the ANTs deformation field to better understand breast material mechanics. The average target registration errors (TRE) were , , and (mean ± standard deviation) and the average fiducial registration errors (FRE) were , , and for the point-based rigid, nonrigid biomechanical, and ANTs registrations, respectively. The mechanics-based deformation metrics revealed an overall anisotropic tissue behavior and a statistically significant difference in volume change between glandular and adipose tissue, suggesting that nonrigid modeling methods may be improved by incorporating material heterogeneity and anisotropy. Overall, registration accuracy significantly improved with increasingly flexible and data-dense registration methods. Analysis of these outcomes may inform the future development of image guidance systems for lumpectomy procedures.

Quantitative evaluation of the influence of multiple MRI sequences and of pathological tissues on the registration of longitudinal data acquired during brain tumor treatment.

Registration methods facilitate the comparison of multiparametric magnetic resonance images acquired at different stages of brain tumor treatments. Image-based registration solutions are influenced by the sequences chosen to compute the distance measure, and the lack of image correspondences due to the resection cavities and pathological tissues. Nonetheless, an evaluation of the impact of these input parameters on the registration of longitudinal data is still missing. This work evaluates the influence of multiple sequences, namely T1-weighted (T1), T2-weighted (T2), contrast enhanced T1-weighted (T1-CE), and T2 Fluid Attenuated Inversion Recovery (FLAIR), and the exclusion of the pathological tissues on the non-rigid registration of pre- and post-operative images. We here investigate two types of registration methods, an iterative approach and a convolutional neural network solution based on a 3D U-Net. We employ two test sets to compute the mean target registration error (mTRE) based on corresponding landmarks. In the first set, markers are positioned exclusively in the surroundings of the pathology. The methods employing T1-CE achieves the lowest mTREs, with a improvement up to 0.8 mm for the iterative solution. The results are higher than the baseline when using the FLAIR sequence. When excluding the pathology, lower mTREs are observable for most of the methods. In the second test set, corresponding landmarks are located in the entire brain volumes. Both solutions employing T1-CE obtain the lowest mTREs, with a decrease up to 1.16 mm for the iterative method, whereas the results worsen using the FLAIR. When excluding the pathology, an improvement is observable for the CNN method using T1-CE. Both approaches utilizing the T1-CE sequence obtain the best mTREs, whereas the FLAIR is the least informative to guide the registration process. Besides, the exclusion of pathology from the distance measure computation improves the registration of the brain tissues surrounding the tumor. Thus, this work provides the first numerical evaluation of the influence of these parameters on the registration of longitudinal magnetic resonance images, and it can be helpful for developing future algorithms.

Integrating external beam and prostate seed implant dosimetry for intermediate and high-risk prostate cancer using biologically effective dose: Impact of image registration technique

Combining external beam radiation therapy (EBRT) and prostate seed implant (PSI) is efficacious in treating intermediate- and high-risk prostate cancer at the cost of increased genitourinary toxicity. Accurate combined dosimetry remains elusive due to lack of registration between treatment plans and different biological effect. The current work proposes a method to convert physical dose to biological effective dose (BED) and spatially register the dose distributions for more accurate combined dosimetry. A PSI phantom was CT scanned with and without seeds under rigid and deformed transformations. The resulting CTs were registered using image-based rigid registration (RI), fiducial-based rigid registration (RF), or b-spline deformable image registration (DIR) to determine which was most accurate. Physical EBRT and PSI dose distributions from a sample of 91 previously-treated combined-modality prostate cancer patients were converted to BED and registered using RI, RF, and DIR. Forty-eight (48) previously-treated patients whose PSI occurred before EBRT were included as a "control" group due to inherent registration. Dose-volume histogram (DVH) parameters were compared for RI, RF, DIR, DICOM, and scalar addition of DVH parameters using ANOVA or independent Student's t tests (α = 0.05). In the phantom study, DIR was the most accurate registration algorithm, especially in the case of deformation. In the patient study, dosimetry from RI was significantly different than the other registration algorithms, including the control group. Dosimetry from RF and DIR were not significantly different from the control group or each other. Combined dosimetry with BED and image registration is feasible. Future work will utilize this method to correlate dosimetry with clinical outcomes.

A novel bone suppression algorithm in intensity-based 2D/3D image registration for real-time tumor motion monitoring: Development and phantom-based validation.

BackgroundReal‐time tumor motion monitoring (TMM) is a crucial process for intra‐fractional respiration management in lung cancer radiotherapy. Since the tumor can be partly or fully located behind the ribs, the TMM is challenging.PurposeThe aim of this work was to develop a bone suppression (BS) algorithm designed for real‐time 2D/3D marker‐less TMM to increase the visibility of the tumor when overlapping with bony structures and consequently to improve the accuracy of TMM.MethodA BS method was implemented in the in‐house developed software for ultrafast intensity‐based 2D/3D tumor registration (Fast Image‐based Registration [FIRE]). The method operates on both, digitally reconstructed radiograph (DRR) and intra‐fractional X‐ray images. The bony structures are derived from computed tomography data by thresholding during ray‐casting, and the resulting bone DRR is subtracted from intra‐fractional X‐ray images to obtain a soft‐tissue‐only image for subsequent tumor registration. The accuracy of TMM utilizing BS was evaluated within a retrospective phantom study with nine different 3D‐printed tumor phantoms placed in the in‐house developed Advanced Radiation DOSimetry (ARDOS) breathing phantom. A 24 mm craniocaudal tumor motion, including rib eclipses, was simulated, and X‐ray images were acquired on the Elekta Versa HD Linac in the lateral and posterior–anterior directions. An error assessment for BS images was evaluated with respect to the ground truth tumor position.ResultsA total error (root mean square error) of 0.87 ± 0.23 mm and 1.03 ± 0.26 mm was found for posterior–anterior and lateral imaging; the mean time for BS was 8.03 ± 1.54 ms. Without utilizing BS, TMM failed in all X‐ray images since the registration algorithm focused on the rib position due to the predominant intensity of this tissue within DRR and X‐ray images.ConclusionThe BS algorithm developed and implemented improved the accuracy, robustness, and stability of real‐time TMM in lung cancer in a phantom study, even in the case of rib interlude where normal tumor registration fails.

Panorama-to-model registration through integration of image retrieval and semantic reprojection

Registering images to a building information model is an effective approach of associating as-built component status to as-designed information. However, registering a single still image to a digital twin is difficult due to pose ambiguities caused by the limited field of view in images. Several recent studies have started focusing on leveraging panoramic images to address problems, such as partial occlusions, repetitive facility components and textureless views, which could cause registration failures in existing image-based registration workflows. Even though having a bigger field of view helps in locating where the image was taken, registering a panoramic image to a building information model is still challenging due to inconsistent visual appearances, such as model vs. image visual differences caused by temporary objects in a scene, lighting condition changes, and different levels of details. In this paper, we present a novel method that registers panoramic images to a digital twin in a hierarchical way: the proposed method first performs rough registration through image retrieval using semantic segmentation, then localizes the image with a fine registration through minimizing semantic reprojection errors. Compared to existing methods, the developed method has the following contributions: 1) semantic-based image retrieval makes the rough registration process robust to lighting condition changes, texture differences, and temporary objects, 2) semantic-based image retrieval allows for bi-directional queries between a model and images, 3) reprojection-based fine registration further reduces the localization error due to the dimension reduction of features during image retrieval. Though the proposed method is developed for panoramic images, it can be generalized to monocular images at the cost of localization accuracy. The developed method was evaluated on a real-world academic building and a synthetic dataset, and the results showed that the proposed method can localize a panorama in less than a second and achieve sub-meter level localization error.

Reattachable fiducial skin marker for automatic multimodality registration

PurposeFusing image information has become increasingly important for optimal diagnosis and treatment of the patient. Despite intensive research towards markerless registration approaches, fiducial marker-based methods remain the default choice for a wide range of applications in clinical practice. However, as especially non-invasive markers cannot be positioned reproducibly in the same pose on the patient, pre-interventional imaging has to be performed immediately before the intervention for fiducial marker-based registrations.MethodsWe propose a new non-invasive, reattachable fiducial skin marker concept for multi-modal registration approaches including the use of electromagnetic or optical tracking technologies. We furthermore describe a robust, automatic fiducial marker localization algorithm for computed tomography (CT) and magnetic resonance imaging (MRI) images. Localization of the new fiducial marker has been assessed for different marker configurations using both CT and MRI. Furthermore, we applied the marker in an abdominal phantom study. For this, we attached the marker at three poses to the phantom, registered ten segmented targets of the phantom’s CT image to live ultrasound images and determined the target registration error (TRE) for each target and each marker pose.ResultsReattachment of the marker was possible with a mean precision of 0.02 mm ± 0.01 mm. Our algorithm successfully localized the marker automatically in all (n=201) evaluated CT/MRI images. Depending on the marker pose, the mean (n=10) TRE of the abdominal phantom study ranged from 1.51 ± 0.75 mm to 4.65 ± 1.22 mm.ConclusionsThe non-invasive, reattachable skin marker concept allows reproducible positioning of the marker and automatic localization in different imaging modalities. The low TREs indicate the potential applicability of the marker concept for clinical interventions, such as the puncture of abdominal lesions, where current image-based registration approaches still lack robustness and existing marker-based methods are often impractical.

Breast image registration for surgery: Insights on material mechanics modeling.

Breast conserving surgery (BCS) is a common procedure for early-stage breast cancer patients. Supine preoperative magnetic resonance (MR) breast imaging for visualizing tumor location and extent, while not standard for procedural guidance, more closely represents the surgical presentation compared to conventional diagnostic pendant positioning. Optimal utilization for surgical guidance, however, requires a fast and accurate image-to-physical registration from preoperative imaging to intraoperative surgical presentation. In this study, three registration methods were investigated on healthy volunteers' breasts (n=11) with the arm-down position simulating preoperative imaging and arm-up position simulating intraoperative data. The registration methods included: (1) point-based rigid registration using synthetic fiducials, (2) non-rigid biomechanical model-based registration using sparse data, and (3) a data-dense 3D diffeomorphic image-based registration from the Advanced Normalization Tools (ANTs) repository. The average target registration errors (TRE) were 10.4 ± 2.3, 6.4 ± 1.5, and 2.8 ± 1.3 mm (mean ± standard deviation) and the average fiducial registration errors (FRE) were 7.8 ± 1.7, 2.5 ± 1.1, and 3.1 ± 1.1 mm (mean ± standard deviation) for the point-based rigid, nonrigid biomechanical, and ANTs registrations, respectively. Additionally, common mechanics-based deformation metrics (volume change and anisotropy) were calculated from the ANTs deformation field. The average metrics revealed anisotropic tissue behavior and a statistical difference in volume change between glandular and adipose tissue, suggesting that nonrigid modeling methods may be improved by incorporating material heterogeneity and anisotropy. Overall, registration accuracy significantly improved with increasingly flexible registration methods, which may inform future development of image guidance systems for lumpectomy procedures.

3D Image-Guided Robotic System for Bone Fracture Reduction

To overcome possible drawbacks of bone fracture reduction such as risks of malrotation, radiation exposure, as well as laborious traction, image-guided robotic surgery systems have been proposed. However, the use of optical tracking systems (OTS) creates inherent line-of-sight problems that cause frequent interruptions during surgery. We propose an OTS-free image-guided bone fracture reduction system utilizing a Stewart robot platform to solve the problem of conventional OTS-based robotic systems. The system applies inverse kinematics to compute the relative positions between the broken bone fragments. Each fragment is pre-operatively registered in the robotic system with a semi-automatic image-based registration method using an attachable jig designed for the proposed platform. This approach is particularly effective when the image features are not clearly detected in the fluoroscopic images. The accuracy of the proposed system was evaluated via pre- and post-operative computed tomography (CT) scans of femoral phantoms. Ex-vivo experiments were also performed on caprine legs to assess the clinical feasibility. In the phantom and ex-vivo experiments, the mean rotational errors of the reduction were 1.79° and 1.76°, respectively. The mean time for the reduction was approximately 3 min. This letter proposes a new method to compute the relative positions between the bone fragments using inverse kinematics and semi-automatic robot-patient registration without requiring a C-arm and an OTS. The OTS-free robotic surgery system has the potential advantages to enhance the accuracy of fracture reduction and reduce the surgery time and radiation exposure.

A novel bone registration method using impression molding and structured-light 3D scanning technology.

Accurate bone registration is critical for computer navigation and robotic surgery. Existing registration systems are expensive, cumbersome, limited in accuracy and/or require intraoperative radiation. We recently reported a novel method of registration utilizing an inexpensive, compact, and X-ray-free structured-light 3D scanner. However, this technique is not always practical in a real surgical setting where soft tissue and blood can obstruct the continuous line-of-sight required for structured-light technology. We sought to remedy these limitations using a novel technique using rapid-setting impression molding to capture bone surface features and scan the undersurface of the mold with a structured-light scanner. The photonegative of this mold is compared to the preoperative computed tomography (CT)-scan to register the bone. A registration accuracy study was conducted on 36 CT-scanned femur sawbones, simulating typical exposure in hip/knee arthroplasty and bone tumor surgery. A cadaver experiment was also conducted to evaluate the feasibility of using the impression molding in a more realistic operating room setting. The registration accuracy of the proposed technique was 0.50 ± 0.19 mm. This was close to the reported accuracy of 0.43 ± 0.18 mm using a structured-light scanner without impression molding (p = 0.085). In comparison, historical values for "paired-point" and intraoperative CT image-based registration methods currently used in modern robotic/computer-navigation systems were 0.68 ± 0.14 mm (p = 0.004) and 0.86 ± 0.38 mm, respectively. The registration accuracy of the cadaver experiment was consistent with that of sawbone experiments. Although future studies are needed to extend to human subjects, this study shows that the impression molding method can produce comparable or better registration accuracy than the existing techniques.

Evaluation of Data-Driven Rigid Motion Correction in Clinical Brain PET Imaging.

Head motion during brain PET imaging can significantly degrade the quality of the reconstructed image, leading to reduced diagnostic value and inaccurate quantitation. A fully data-driven motion correction approach was recently demonstrated to produce highly accurate motion estimates (<1 mm) with high temporal resolution (≥1 Hz), which can then be used for a motion-corrected reconstruction. This can be applied retrospectively with no impact on the clinical image acquisition protocol. We present a reader-based evaluation and an atlas-based quantitative analysis of this motion correction approach within a clinical cohort. Methods: Clinical patient data were collected over 2019-2020 and processed retrospectively. Motion was estimated using image-based registration on reconstructions of ultrashort frames (0.6-1.8 s), after which list-mode reconstructions that were fully motion-corrected were performed. Two readers graded the motion-corrected and uncorrected reconstructions. An atlas-based quantitative analysis was performed. Paired Wilcoxon tests were used to test for significant differences in reader scores and SUVs between reconstructions. The Levene test was used to determine whether motion correction had a greater impact on quantitation in the presence of motion than when motion was low. Results: Fifty standard clinical 18F-FDG brain PET datasets (age range, 13-83 y; mean ± SD, 59 ± 20 y; 27 women) from 3 scanners were collected. The reader study showed a significantly different, diagnostically relevant improvement by motion correction when motion was present (P = 0.02) and no impact in low-motion cases. Eight percent of all datasets improved from diagnostically unacceptable to acceptable. The atlas-based analysis demonstrated a significant difference between the motion-corrected and uncorrected reconstructions in cases of high motion for 7 of 8 regions of interest (P < 0.05). Conclusion: The proposed approach to data-driven motion estimation and correction demonstrated a clinically significant impact on brain PET image reconstruction.

Segmentation-guided multi-modal registration of liver images for dose estimation in SIRT

PurposeSelective internal radiation therapy (SIRT) requires a good liver registration of multi-modality images to obtain precise dose prediction and measurement. This study investigated the feasibility of liver registration of CT and MR images, guided by segmentation of the liver and its landmarks. The influence of the resulting lesion registration on dose estimation was evaluated.MethodsThe liver segmentation was done with a convolutional neural network (CNN), and the landmarks were segmented manually. Our image-based registration software and its liver-segmentation-guided extension (CNN-guided) were tuned and evaluated with 49 CT and 26 MR images from 20 SIRT patients. Each liver registration was evaluated by the root mean square distance (RMSD) of mean surface distance between manually delineated liver contours and mass center distance between manually delineated landmarks (lesions, clips, etc.). The root mean square of RMSDs (RRMSD) was used to evaluate all liver registrations. The CNN-guided registration was further extended by incorporating landmark segmentations (CNN&LM-guided) to assess the value of additional landmark guidance. To evaluate the influence of segmentation-guided registration on dose estimation, mean dose and volume percentages receiving at least 70 Gy (V70) estimated on the 99mTc-labeled macro-aggregated albumin (99mTc-MAA) SPECT were computed, either based on lesions from the reference 99mTc-MAA CT (reference lesions) or from the registered floating CT or MR images (registered lesions) using the CNN- or CNN&LM-guided algorithms.ResultsThe RRMSD decreased for the floating CTs and MRs by 1.0 mm (11%) and 3.4 mm (34%) using CNN guidance for the image-based registration and by 2.1 mm (26%) and 1.4 mm (21%) using landmark guidance for the CNN-guided registration. The quartiles for the relative mean dose difference (the V70 difference) between the reference and registered lesions and their correlations [25th, 75th; r] are as follows: [− 5.5% (− 1.3%), 5.6% (3.4%); 0.97 (0.95)] and [− 12.3% (− 2.1%), 14.8% (2.9%); 0.96 (0.97)] for the CNN&LM- and CNN-guided CT to CT registrations, [− 7.7% (− 6.6%), 7.0% (3.1%); 0.97 (0.90)] and [− 15.1% (− 11.3%), 2.4% (2.5%); 0.91 (0.78)] for the CNN&LM- and CNN-guided MR to CT registrations.ConclusionGuidance by CNN liver segmentations and landmarks markedly improves the performance of the image-based registration. The small mean dose change between the reference and registered lesions demonstrates the feasibility of applying the CNN&LM- or CNN-guided registration to volume-level dose prediction. The CNN&LM- and CNN-guided registrations for CTs can be applied to voxel-level dose prediction according to their small V70 change for most lesions. The CNN-guided MR to CT registration still needs to incorporate landmark guidance for smaller change of voxel-level dose estimation.

LAPNet: Non-Rigid Registration Derived in k-Space for Magnetic Resonance Imaging.

Physiological motion, such as cardiac and respiratory motion, during Magnetic Resonance (MR) image acquisition can cause image artifacts. Motion correction techniques have been proposed to compensate for these types of motion during thoracic scans, relying on accurate motion estimation from undersampled motion-resolved reconstruction. A particular interest and challenge lie in the derivation of reliable non-rigid motion fields from the undersampled motion-resolved data. Motion estimation is usually formulated in image space via diffusion, parametric-spline, or optical flow methods. However, image-based registration can be impaired by remaining aliasing artifacts due to the undersampled motion-resolved reconstruction. In this work, we describe a formalism to perform non-rigid registration directly in the sampled Fourier space, i.e. k-space. We propose a deep-learning based approach to perform fast and accurate non-rigid registration from the undersampled k-space data. The basic working principle originates from the Local All-Pass (LAP) technique, a recently introduced optical flow-based registration. The proposed LAPNet is compared against traditional and deep learning image-based registrations and tested on fully-sampled and highly-accelerated (with two undersampling strategies) 3D respiratory motion-resolved MR images in a cohort of 40 patients with suspected liver or lung metastases and 25 healthy subjects. The proposed LAPNet provided consistent and superior performance to image-based approaches throughout different sampling trajectories and acceleration factors.

Consistency Check of Deformable Image Registration-Based Dose Summation for Off-Line Adaptive Re-Planning

<h3>Purpose/Objective(s)</h3> Uncertainty of deformable image registration (DIR) based dose accumulation is a concern due to the uncertainty with the DIR, in particular for a large deformation. In this work, we introduce a practical workflow of using a DVH overlay technique to check consistency in DIR-based dose accumulation during off-line adaptive replanning for large deformations observed during radiation therapy for abdominal tumors. <h3>Materials/Methods</h3> Required inputs to the DVH overlay workflow are deformably registered primary (original plan) and secondary (re-plan) images, primary regions-of-interest (ROIs), and a secondary dose distribution. Primary ROIs are forward warped to the secondary image, secondary dose is inversely warped to the primary image, and the ROI DVHs for each image are overlaid. Differences between the DVHs at a given dose point (e.g., V_45Gy) quantify the dose uncertainty due to inverse consistency error (ICE) inherent in the DIR. We applied the workflow to a pancreas case treated using modulated arc therapy (mARC) on a Linac equipped with integrated kVCT-on-rails IGRT. For the original plan, 65.25-Gy and 50.75-Gy PTVs were concurrently treated over 29 fractions. After fraction 21, the patient was observed to have lost sufficient weight to justify an adaptive mARC plan based on an updated kVCT image. A composite dose distribution was compiled on the original plan CT by summing 21 fractions of the original planned dose and 8 fractions of a dose distribution warped from the re-plan CT to the original plan CT. Warping was performed using a hybrid DIR algorithm combining contour-based and image-based registration. For each organ-at-risk (OAR) DVH dose point, the workflow-derived ICE (scaled to 8 fractions) was additively applied as an uncertainty to the summation DVH. <h3>Results</h3> For the re-planned pancreas case, ICE based on the DVH overlay workflow was greatest for the small bowel V_45Gy (approximately 5%) and for the stomach V_45Gy and V_56Gy (approximately 2% for both). ICE was also approximately 4% at V_15Gy for the duodenum and colon. For all other ROIs, ICE was 1% or less from the DVH overlay, indicating a favorable DIR for those regions. When applying the DVH overlay-based ICE the resulting accumulated dose remained compliant with the original plan objectives for all OARs except for the stomach V_56Gy, which equaled 7.9 cm³ with ICE included (plan objective < 5 cm³). <h3>Conclusion</h3> A DVH overlay technique was used to estimate ICE attributable to the uncertainty in DIR-based dose accumulation. The method was applied to a pancreas case for which dose accumulation was performed due to off-line adaptive re-planning for a large anatomic change during the treatment course. In instances where the ICE is over a pre-defined threshold or the summation dose exceeds plan objectives if ICE is included, it is prudent to review the DIR dose transfer and accumulation before using it to make clinical decisions.

Y-Net: Learning Domain Robust Feature Representation for ground camera image and large-scale image-based point cloud registration

Registering the 2D images (2D space) with the 3D model of the environment (3D space) provides a promising solution to outdoor Augmented Reality (AR) virtual-real registration. In this work, we use the position and orientation of the ground camera image to synthesize a corresponding rendered image from the outdoor large-scale 3D image-based point cloud. To achieve the virtual-real registration, we indirectly establish the spatial relationship between 2D and 3D space by matching the above two kinds (2D/3D space) of cross-domain images. However, matching cross-domain images goes beyond the capability of handcrafted descriptors and existing deep neural networks. To address this issue, we propose an end-to-end network, Y-Net, to learn Domain Robust Feature Representations (DRFRs) for the cross-domain images. Besides, we introduce a cross-domain-constrained loss function that balances the loss in image content and cross-domain consistency of the feature representations. Experimental results show that the DRFRs simultaneously preserve the representation of image content and suppress the influence of independent domains. Furthermore, Y-Net outperforms the existing algorithms on extracting feature representations and achieves state-of-the-art performance in cross-domain image retrieval. Finally, we validate the Y-Net-based registration approach on campus to demonstrate its possible applicability.

Optimizing the frame duration for data-driven rigid motion estimation in brain PET imaging.

Data-driven rigid motion estimation for PET brain imaging is usually performed using data frames sampled at low temporal resolution to reduce the overall computation time and to provide adequate signal-to-noise ratio in the frames. In recent work it has been demonstrated that list-mode reconstructions of ultrashort frames are sufficient for motion estimation and can be performed very quickly. In this work we take the approach of using image-based registration of reconstructions of very short frames for data-driven motion estimation, and optimize a number of reconstruction and registration parameters (frame duration, MLEM iterations, image pixel size, post-smoothing filter, reference image creation, and registration metric) to ensure accurate registrations while maximizing temporal resolution and minimizing total computation time. Data from 18 F-fluorodeoxyglucose (FDG) and 18 F-florbetaben (FBB) tracer studies with varying count rates are analyzed, for PET/MR and PET/CT scanners. For framed reconstructions using various parameter combinations interframe motion is simulated and image-based registrations are performed to estimate that motion. For FDG and FBB tracers using 4×105 true and scattered coincidence events per frame ensures that 95% of the registrations will be accurate to within 1mm of the ground truth. This corresponds to a frame duration of 0.5-1sec for typical clinical PET activity levels. Using four MLEM iterations with no subsets, a transaxial pixel size of 4mm, a post-smoothing filter with 4-6mm full width at half maximum, and averaging two or more frames to create the reference image provides an optimal set of parameters to produce accurate registrations while keeping the reconstruction and processing time low. It is shown that very short frames (≤1sec) can be used to provide accurate and quick data-driven rigid motion estimates for use in an event-by-event motion corrected reconstruction.

A hybrid, image-based and biomechanics-based registration approach to markerless intraoperative nodule localization during video-assisted thoracoscopic surgery.

The resection of small, low-dense or deep lung nodules during video-assisted thoracoscopic surgery (VATS) is surgically challenging. Nodule localization methods in clinical practice typically rely on the preoperative placement of markers, which may lead to clinical complications. We propose a markerless lung nodule localization framework for VATS based on a hybrid method combining intraoperative cone-beam CT (CBCT) imaging, free-form deformation image registration, and a poroelastic lung model with allowance for air evacuation. The difficult problem of estimating intraoperative lung deformations is decomposed into two more tractable sub-problems: (i) estimating the deformation due the change of patient pose from preoperative CT (supine) to intraoperative CBCT (lateral decubitus); and (ii) estimating the pneumothorax deformation, i.e.a collapse of the lung within the thoracic cage. We were able to demonstrate the feasibility of our localization framework with a retrospective validation study on 5 VATS clinical cases. Average initial errors in the range of 22 to 38mm were reduced to the range of 4 to 14mm, corresponding to an error correction in the range of 63 to 85%. To our knowledge, this is the first markerless lung deformation compensation method dedicated to VATS and validated on actual clinical data.

Fully convolutional network-based registration for augmented assembly systems

Image-based registration methods have been widely used in augmented assembly systems. However, many challenges remain to be solved, such as low robustness and poor timeliness during registration. This paper presents a deep learning approach for registration. To reduce the workload of data collection, an automatic picture generation method is offered for deep learning algorithm, and a dataset is built for detecting the keypoints of assembly objects. We propose a fully convolutional network (FCN) model for detecting keypoints from a single RGB image. The FCN model, which consists of 13 layers, can accurately detect the location of keypoints with a low computational resource consumption. The detected keypoints are used to solve the camera position and orientation for augmented assembly registration. The experiments demonstrate that our method is robust against different rotation angles of the assembly object and against background interference. The detection accuracy is high under different camera motion blurs; to be specific, the pixel error in different directions of 640 × 480 images was only 0.9 pixels. The FCN-based registration approach was shown to be fast during augmented assembly experiments, achieving up to 30 FPS.

Cost-function testing methodology for image-based registration of endoscopy to CT images in the head and neck

One of the largest geometric uncertainties in designing radiotherapy treatment plans for squamous cell cancers of the head and neck is contouring the gross tumor volume. We have previously described a method of projecting mucosal disease contours, visible on endoscopy, to volumetrically reconstructed planning computed tomography (CT) datasets, using electromagnetic (EM) tracking of a flexible endoscope, enabling rigid registration between endoscopic and CT images.However, to achieve better accuracy for radiotherapy planning, we propose refining this initial registration with image-based registration methods. In this paper, several types of cost functions are evaluated based on accuracy and robustness. Three phantoms and eight clinical cases are used to test each cost function, with initial registration of endoscopy to CT provided by the pose of the flexible endoscope recovered from EM tracking. Cost function classes include: cross correlation, mutual information and gradient methods. For each test case, a ground truth virtual camera pose was first defined by manual registration of anatomical features visible in both real and virtual endoscope images. A new set of evenly spaced fiducial points and a sample contour were created and projected onto the CT image to be used in assessing image registration quality. A new set of 5000 displaced poses was generated by random sampling displacements along each translational and rotational dimension. At each pose, fiducial and contour points in the real image were again projected on the CT image. The cost function, fiducial registration error and contouring error values were then calculated.While all cost functions performed well in select cases, only the normalized gradient field function consistently had registration errors less than 2 mm, which is the accuracy needed if this application of registering mucosal disease identified on optical image to CT images is to be used in the clinical practice of radiation treatment planning.(Registration: ClinicalTrials.gov NCT02704169)