Deformable Image Registration Research Articles

A unified framework for focal intensity change detection and deformable image registration. Application to the monitoring of multiple sclerosis lesions in longitudinal 3D brain MRI.

Registration is a crucial step in the design of automatic change detection methods dedicated to longitudinal brain MRI. Even small registration inaccuracies can significantly deteriorate the detection performance by introducing numerous spurious detections. Rigid or affine registration are usually considered to align baseline and follow-up scans, as a pre-processing step before applying a change detection method. In the context of multiple sclerosis, using deformable registration can be required to capture the complex deformations due to brain atrophy. However, non-rigid registration can alter the shape of appearing and evolving lesions while minimizing the dissimilarity between the two images. To overcome this issue, we consider registration and change detection as intertwined problems that should be solved jointly. To this end, we formulate these two separate tasks as a single optimization problem involving a unique energy that models their coupling. We focus on intensity-based change detection and registration, but the approach is versatile and could be extended to other modeling choices. We show experimentally on synthetic and real data that the proposed joint approach overcomes the limitations of the sequential scheme.

Detailed dosimetric evaluation of inter-fraction and respiratory motion in lung stereotactic body radiation therapy based on daily 4D cone beam CT images

Objective. Periodic respiratory motion and inter-fraction variations are sources of geometric uncertainty in stereotactic body radiation therapy (SBRT) of pulmonary lesions. This study extensively evaluates and validates the separate and combined dosimetric effect of both factors using 4D-CT and daily 4D-cone beam CT (CBCT) dose accumulation scenarios. Approach. A first cohort of twenty early stage or metastatic disease lung cancer patients were retrospectively selected to evaluate each scenario. The planned-dose (3DRef) was optimized on a 3D mid-position CT. To estimate the dosimetric impact of respiratory motion (4DRef), inter-fractional variations (3DAcc) and the combined effect of both factors (4DAcc), three dose accumulation scenarios based on 4D-CT, daily mid-cone beam CT (CBCT) position and 4D-CBCT were implemented via CT-CT/CT-CBCT deformable image registration (DIR) techniques. Each scenario was compared to 3DRef. A separate cohort of ten lung SBRT patients was selected to validate DIR techniques. The distance discordance metric (DDM) was implemented per voxel and per patient for tumor and organs at risk (OARs), and the dosimetric impact for CT-CBCT DIR geometric errors was calculated. Main results. Median and interquartile range (IQR) of the dose difference per voxel were 0.05/2.69 Gy and −0.12/2.68 Gy for and For the IQR was considerably smaller −0.15/0.78 Gy. These findings were confirmed by dose volume histogram parameters calculated in tumor and OARs. For CT-CT/CT-CBCT DIR validation, DDM (95th percentile) was highest for heart (6.26 mm)/spinal cord (8.00 mm), and below 3 mm for tumor and the rest of OARs. The dosimetric impact of CT-CBCT DIR errors was below 2 Gy for tumor and OARs. Significance. The dosimetric impact of inter-fraction variations were shown to dominate those of periodic respiration in SBRT for pulmonary lesions. Therefore, treatment evaluation and dose-effect studies would benefit more from dose accumulation focusing on day-to-day changes then those that focus on respiratory motion.

Relationship between absorbed dose and changes in liver volume after chemoradiotherapy for esophageal cancer.

The liver is the largest organ in the abdomen and is often irradiated in radiotherapy for non-hepatic malignancies. As most of the studies on changes in liver volume are on hepatocellular carcinoma based on liver dysfunction, there are few studies on healthy liver. In this study, we investigated the relationship between absorbed dose and changes in liver volume after chemoradiotherapy for esophageal cancer in patients without apparent pre-treatment liver dysfunction. Liver volume was compared between pre-treatment, acute (< 4months) and late post-treatment (≥ 4 and < 13months) phases in 12 patients using abdominal plain CT images. Volume changes were evaluated separately for the right and left lobes. We investigated the relationship between the volume change and VxGy (percentage of volume received x Gy or more dose). In addition, volume change for each absorbed dose was investigated using deformable image registration. The volume of the left lobe showed a significant decrease between pre-treatment and acute post-treatment phases (p < 0.001), while the volume of right lobe and between acute and late post-treatment phase of left lobe did not. The mean value of the volume reduction rate of the left lobe was 51.1% and equivalent to the mean value of V30Gy. As a result of the volume change for each absorbed dose, the volume reduction rate increased as the absorbed dose increased, and a significant volume loss was observed at doses above 11Gy. Volume of the liver significantly decreased only in the acute phase after chemoradiotherapy for esophageal cancer. The tolerable dose for a healthy liver is generally considered to be 30Gy, but attention should be paid to lower doses to avoid radiation-induced liver injury.

Part-and-whole: A novel framework for deformable medical image registration

Part-and-whole: A novel framework for deformable medical image registration

How the Rigid and Deformable Image Registration Approaches Affect the Absorbed Dose Estimation Using Images Collected before and after Transarterial Radioembolization with 90Y Resin Microspheres in a Clinical Setting

Background: Transarterial radioembolization (TARE) relies on directly injected 90Y- or 166Ho-loaded microspheres in the hepatic arteries. The activity to be injected is generally based on pre-TARE 99mTc-macro-aggregated-albumin (MAA) imaging, while the actual dose distribution is based on post-treatment images. The volume of interest (VOIs) propagation methods (i.e., rigid and deformable) from pre- to post-TARE imaging might affect the estimation of the mean absorbed dose in the tumor and non-tumoral liver (NTL), i.e., DT and DNTL, respectively. Methods: In 101 consecutive patients, liver and tumor were delineated on pre-TARE images and semi-automatically transferred on 90Y-PET/CT images with a rigid or deformable registration approach. Pre- and post-TARE volumes and DT/DNTL/DL were compared using correlation coefficient (CC) indexes, such as intra-class (ICC), Pearson’s (PCC), concordance (CCCo) and Bland–Altman analyses. The Kaplan–Meier curves of overall survival (OS) were calculated according to DT. Results: All computed CCs indicated very good (&gt;0.92) agreement for volume comparison, while they suggested good (ICC ≥ 0.869, PCC ≥ 0.876 and CCCo ≥ 0.790) and moderate agreement in the intra- and inter-modality DT/DNTL/DL comparisons, respectively. Bland–Altman analyses showed percentage differences between the manual and deformable approaches of up to about −31%, 9% and 62% for tumoral volumes, DT and DNTL, respectively. The overall survival analysis showed statistically significant differences using DT cutoffs of 110, 90 and 85 Gy for the manual, rigid and deformable approaches, respectively. Conclusions: The semi-automatic transfer of VOIs from pre- and post-TARE imaging is feasible, but the selected method might affect prognostic DT/DNTL constraints.

Development of a multi-layer quality assurance program to evaluate the uncertainty of deformable dose accumulation in adaptive radiotherapy.

Deformable dose accumulation (DDA) has uncertainties which impede the implementation of DDA-based adaptive radiotherapy (ART) in clinic. The purpose of this study is to develop a multi-layer quality assurance (MLQA) program to evaluate uncertainties in DDA. A computer program is developed to generate a pseudo-inverse displacement vector field (DVF) for each deformable image registration (DIR) performed in Accuray's PreciseART. The pseudo-inverse DVF is first used to calculate a pseudo-inverse consistency error (PICE) and then implemented in an energy and mass congruent mapping (EMCM) method to reconstruct a deformed dose. The PICE is taken as a metric to estimate DIR uncertainties. A pseudo-inverse dose agreement rate (PIDAR) is used to evaluate the consequence of the DIR uncertainties in DDA and the principle of energy conservation is used to validate the integrity of dose mappings. The developed MLQA program was tested using the data collected from five representative cancer patients treated with tomotherapy. DIRs were performed in PreciseART to generate primary DVFs for the five patients. The fidelity index and PICE of these DVFs on average are equal to 0.028mm and 0.169mm, respectively. With the criteria of 3mm/3% and 5mm/5%, the PIDARs of the PreciseART-reconstructed doses are 73.9±4.4% and 87.2±3.3%, respectively. The PreciseART and EMCM-based dose reconstructions have their deposited energy changed by 5.6±3.9% and 2.6±1.5% in five GTVs, and by 9.2±7.8% and 4.7±3.6% in 30 OARs, respectively. A pseudo-inverse map-based EMCM program has been developed to evaluate DIR and dose mapping uncertainties. This program could also be used as a sanity check tool for DDA-based ART.

A review of deep learning-based deformable medical image registration.

The alignment of images through deformable image registration is vital to clinical applications (e.g., atlas creation, image fusion, and tumor targeting in image-guided navigation systems) and is still a challenging problem. Recent progress in the field of deep learning has significantly advanced the performance of medical image registration. In this review, we present a comprehensive survey on deep learning-based deformable medical image registration methods. These methods are classified into five categories: Deep Iterative Methods, Supervised Methods, Unsupervised Methods, Weakly Supervised Methods, and Latest Methods. A detailed review of each category is provided with discussions about contributions, tasks, and inadequacies. We also provide statistical analysis for the selected papers from the point of view of image modality, the region of interest (ROI), evaluation metrics, and method categories. In addition, we summarize 33 publicly available datasets that are used for benchmarking the registration algorithms. Finally, the remaining challenges, future directions, and potential trends are discussed in our review.

Context-driven pyramid registration network for estimating large topology-preserved deformation

Deep learning-based deformable image registration methods have become attractive alternatives to traditional methods because of their great performance and fast run time. However, it is still challenging for these methods to estimate large topology-preserved deformation, and the contextual information that is important for large deformation is also under-mined. To address these issues, we propose a novel unsupervised context-driven pyramid registration network for estimating large topology-preserved deformation named CPRNet. Specifically, based on the multi-resolution feature pyramids, we first design multi-receptive-field guidance modules, aiming at exploiting the multi-scale spatial correlation between features of two pyramids. Then we devise multi-view context fusion modules to dynamically fuse deep contextual information containing high-level semantic information from different views of feature maps. Further, we develop a residual estimation strategy to estimate the deformation in a coarse-to-fine manner. Moreover, a deformation field regularization module is proposed to address the challenge of balancing the registration performance and topology preservation. The experiments both on liver computed tomography (CT) images and brain magnetic resonance (MR) images demonstrate that our proposed method provides effective and accurate registration on various datasets with a fast run time. Compared with existing learning-based registration methods, our proposed method exceeds the performance in most trials while maintaining desirable topology preservation capability and can potentially fit various image registration tasks.

Correlation between CT Value on Lung Subtraction CT and Radioactive Count on Perfusion Lung Single Photon Emission CT in Chronic Thromboembolic Pulmonary Hypertension.

Lung subtraction CT (LSCT), the subtraction of noncontrast CT from CT pulmonary angiography (CTPA) without spatial misregistration, is easily applicable by utilizing a software-based deformable image registration technique without additional hardware and permits the evaluation of lung perfusion as iodine accumulation, similar to that observed in perfusion lung single photon emission CT (PL-SPECT). The aim of this study was to use LSCT to newly assess the quantitative correlation between the CT value on LSCT and radioactive count on PL-SPECT as a reference and validate the quantification of lung perfusion by measuring the CT value in chronic thromboembolic pulmonary hypertension (CTEPH). We prospectively enrolled 47 consecutive patients with CTEPH undergoing both LSCT and PL-SPECT; we used noncontrast CT, CTPA, and LSCT to measure CT values and PL-SPECT to measure radioactive counts in areas representing three different perfusion classes-no perfusion defect, subsegmental perfusion defect, and segmental perfusion defect; we compared CT values on noncontrast CT, CTPA, and LSCT and radioactive counts on PL-SPECT among the three classes, then assessed the correlation between them. Both the CT values and radioactive counts differed significantly among the three classes (p &lt; 0.01 for all) and showed weak correlation (ρ = 0.38) by noncontrast CT, moderate correlation (ρ = 0.61) by CTPA, and strong correlation (ρ = 0.76) by LSCT. The CT value measurement on LSCT is a novel quantitative approach to assess lung perfusion in CTEPH and only correlates strongly with radioactive count measurement on PL-SPECT.

Multiscale Approach for Bounded Deformation Image Registration

Deformable image registration is a very important topic in the field of image processing. It is widely used in image fusion and shape analysis. Generally speaking, image registration models can be divided into two categories: smooth registration and non-smooth registration. During the last decades, many smooth registration models (i.e., diffeomorphic registration) were proposed. However, image with strong noise may lead to discontinuous deformation, which cannot be modelled by smooth registration. To simulate this kind of deformation, some non-smooth registration models were also proposed. However, numerical algorithms for these models are easily trapped into a local minimum because of the nonconvexity of the object functional. To overcome the local minimum of the object functional, we propose a multiscale approach for a non-smooth registration model: the bounded deformation (BD) model. The convergence of the approach is shown, and numerical tests are also performed to show the good performance of the proposed multiscale approach.

Detailed analysis of failure patterns using deformable image registration in hypopharyngeal cancer patients treated with sequential boost intensity-modulated radiotherapy.

Sequential boost intensity-modulated radiotherapy (SQB-IMRT) uses two different planning CTs (pCTs) and treatment plans. SQB-IMRT is a form of adaptive radiotherapy that allows for responses to changes in the shape of the tumour and organs at risk (OAR). On the other hand, dose accumulation with the two plans can be difficult to evaluate. The purpose of this study was to analyse patterns of loco-regional failure using deformable image registration (DIR) in hypopharyngeal cancer patients treated with SQB-IMRT. Between 2013 and 2019, 102 patients with hypopharyngeal cancer were treated with definitive SQB-IMRT at our institution. Dose accumulation with the 1st and 2nd plans was performed, and the dose to the loco-regional recurrent tumour volume was calculated using the DIR workflow. Failure was classified as follows: (i) in-field (≥95% of the recurrent tumour volume received 95% of the prescribed dose); (ii) marginal (20-95%); or (iii) out-of-field (<20%). After a median follow-up period of 25months, loco-regional failure occurred in 34 patients. Dose-volume histogram analysis showed that all loco-regional failures occurred in the field within 95% of the prescribed dose, with no marginal or out-of-field recurrences observed. The dosimetric analysis using DIR showed that all loco-regional failures were within the high-dose region. More aggressive treatment may be required for gross tumours.

Spatial normalisation for novel MR-image based data mining in children with brain tumours

<h3>Objectives</h3> Image-based data mining (IBDM) is a novel technique to identify dose-response relationships (DRRs) in radiotherapy. IBDM comprises two steps: spatial normalisation of patient data to a reference anatomy using deformable image registration, and ‘per-voxel' dose comparison. The main advantage of IBDM is the possibility of identifying DRRs at the sub-organ level and in structures that are not routinely contoured, depending on the image registration accuracy. Computed Tomography (CT)-based IBDM has been successfully applied in adults; however, for children with brain tumours, Magnetic Resonance (MR) imaging is indicated due to superior anatomical definition. We evaluate the accuracy of spatial normalisation using MR images to determine the feasibility of MR-based IBDM in children with brain tumours. <h3>Methods</h3> We used MR images and radiotherapy contours from 129 children previously treated on the SJMB03 protocol for medulloblastoma. The patient with the median brain volume was selected as the reference anatomy for deformable image registration (NiftyReg software). We analysed registration accuracy, comparing each patient's deformed contours to the reference contours, in 5 brain structures (Table 1) via 95%-Hausdorff (HD95%), average-contour (MeanDTA) and centre-of-mass (CoM) distances. Based on previous results in CT data, meanDTA accuracy <3mm was considered acceptable. <h3>Results</h3> 16/129 cases required manual adjustments to fix registration errors, commonly arising due to enlarged ventricles or large rotations compared to the reference anatomy. MeanDTA for all structures was <2.1mm and all metrics <6mm (Table 1). <h3>Conclusion</h3> Accurate spatial normalisation was achieved using MR images from paediatric brain tumour patients, suggesting that MR-based IBDM is feasible in this patient group.

Patient-specific daily updated deep learning auto-segmentation for MRI-guided adaptive radiotherapy

Background and purposeDeep Learning (DL) technique has shown great potential but still has limited success in online contouring for MR-guided adaptive radiotherapy (MRgART). This study proposed a patient-specific DL auto-segmentation (DLAS) strategy using the patient’s previous images and contours to update the model and improve segmentation accuracy and efficiency for MRgART. Methods and materialsA prototype model was trained for each patient using the first set of MRI and corresponding contours as inputs. The patient-specific model was updated after each fraction with all the available fractional MRIs/contours, and then used to predict the segmentation for the next fraction. During model training, a variant was fitted under consistency constraints, limiting the differences in the volume, length and centroid between the predictions for the latest MRI within a reasonable range. The model performance was evaluated for both organ-at-risks and tumors auto-segmentation for a total of 6 abdominal/pelvic cases (each with at least 8 sets of MRIs/contours) underwent MRgART through Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance (HD95), and was compared with deformable image registration (DIR) and frozen DL model (no updating after pre-training). The contouring time was also recorded and analyzed. ResultsThe proposed model achieved superior performance with higher mean DSC (0.90, 95 % CI: 0.88–0.95), as compared to DIR (0.63, 95 %CI: 0.59–0.68) and frozen DL models (0.74, 95 % CI: 0.71–0.79). As for tumors, the proposed method yielded a median DSC of 0.95, 95 % CI: 0.94–0.97, and a median HD95 of 1.63 mm, 95 % CI: 1.22 mm-2.06 mm. The contouring time was reduced significantly (p < 0.05) using the proposed method (73.4 ± 6.5 secs) compared to the manual process (12 ∼ 22 mins). The online ART time was reduced to 1650 ± 274 seconds with the proposed method, as compared to 3251.8 ± 447 seconds using the original workflow. ConclusionThe proposed patient-specific DLAS method can significantly improve the segmentation accuracy and efficiency for longitudinal MRIs, thereby facilitating the routine practice of MRgART.

Segmentation by test-time optimization for CBCT-based adaptive radiation therapy.

Online adaptive radiotherapy (ART) requires accurate and efficient auto-segmentation of target volumes and organs-at-risk (OARs) in mostly cone-beam computed tomography (CBCT) images, which often have severe artifacts and lack soft-tissue contrast, making direct segmentation very challenging. Propagating expert-drawn contours from the pretreatment planning CT through traditional or deep learning (DL)-based deformable image registration (DIR) can achieve improved results in many situations. Typical DL-based DIR models are population based, that is, trained with a dataset for a population of patients, and so they may be affected by the generalizability problem. In this paper, we propose a method called test-time optimization (TTO) to refine a pretrained DL-based DIR population model, first for each individual test patient, and then progressively for each fraction of online ART treatment. Our proposed method is less susceptible to the generalizability problem and thus can improve overall performance of different DL-based DIR models by improving model accuracy, especially for outliers. Our experiments used data from 239 patients with head-and-neck squamous cell carcinoma to test the proposed method. First, we trained a population model with 200 patients and then applied TTO to the remaining 39 test patients by refining the trained population model to obtain 39 individualized models. We compared each of the individualized models with the population model in terms of segmentation accuracy. The average improvement of the Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD95) of segmentation can be up to 0.04 (5%) and 0.98mm (25%), respectively, with the individualized models compared to the population model over 17 selected OARs and a target of 39 patients. Although the average improvement may seem mild, we found that the improvement for outlier patients with structures of large anatomical changes is significant. The number of patients with at least 0.05 DSC improvement or 2mm HD95 improvement by TTO averaged over the 17 selected structures for the state-of-the-art architecture VoxelMorph is 10 out of 39 test patients. By deriving the individualized model using TTO from the pretrained population model, TTO models can be ready in about 1min. We also generated the adapted fractional models for each of the 39 test patients by progressively refining the individualized models using TTO to CBCT images acquired at later fractions of online ART treatment. When adapting the individualized model to a later fraction of the same patient, the model can be ready in less than a minute with slightly improved accuracy. The proposed TTO method is well suited for online ART and can boost segmentation accuracy for DL-based DIR models, especially for outlier patients where the pretrained models fail.

Seq2Morph: A deep learning deformable image registration algorithm for longitudinal imaging studies and adaptive radiotherapy.

To simultaneously register all the longitudinal images acquired in a radiotherapy course for analyzing patients' anatomy changes for adaptive radiotherapy (ART). To address the unique needs of ART, we designed Seq2Morph, a novel deep learning-based deformable image registration (DIR) network. Seq2Morph was built upon VoxelMorph which is a general-purpose framework for learning-based image registration. The major upgrades are (1) expansion of inputs to all weekly cone-beam computed tomography (CBCTs) acquired for monitoring treatment responses throughout a radiotherapy course, for registration to their planning CT; (2) incorporation of 3D convolutional long short-term memory between the encoder and decoder of VoxelMorph, to parse the temporal patterns of anatomical changes; and (3) addition of bidirectional pathways to calculate and minimize inverse consistency errors (ICEs). Longitudinal image sets from 50 patients, including a planning CT and 6 weekly CBCTs per patient, were utilized for network training and cross-validation. The outputs were deformation vector fields for all the registration pairs. The loss function was composed of a normalized cross-correlation for image intensity similarity, a DICE for contour similarity, an ICE, and a deformation regularization term. For performance evaluation, DICE and Hausdorff distance (HD) for the manual versus predicted contours of tumor and esophagus on weekly basis were quantified and further compared with other state-of-the-art algorithms, including conventional VoxelMorph and large deformation diffeomorphic metric mapping (LDDMM). Visualization of the hidden states of Seq2Morph revealed distinct spatiotemporal anatomy change patterns. Quantitatively, Seq2Morph performed similarly to LDDMM, but significantly outperformed VoxelMorph as measured by GTV DICE: (0.799±0.078, 0.798±0.081, and 0.773±0.078), and 50% HD (mm): (0.80±0.57, 0.88±0.66, and 0.95±0.60). The per-patient inference of Seq2Morph took 22 s, much less than LDDMM (∼30 min). Seq2Morph can provide accurate and fast DIR for longitudinal image studies by exploiting spatial-temporal patterns. It closely matches the clinical workflow and has the potential to serve both online and offline ART.

Evaluating the accumulated dose distribution of organs at risk in combined radiotherapy for cervical carcinoma based on deformable image registration

ObjectiveTo evaluate the feasibility and value of deformable image registration (DIR) in calculating the cumulative doses of organs at risk (OARs) in the combined radiotherapy of cervical cancer. Patients and methodsThirty cervical cancer patients treated with external beam radiotherapy (EBRT) combined with intracavitary brachytherapy (ICBT) were reviewed. The simulation CT images of EBRT and ICBT were imported into Varian Velocity 4.1 for the DIR-based dose accumulation. Cumulative dose-volume parameters of D2cc for rectum and bladder were compared between the direct addition (DA) and DIR methods. The quantitative parameters were measured to evaluate the accuracy of DIR. ResultsThe three-dimensional cumulative dose distribution of the tumor and OARs were graphically well illustrated by composite isodose lines. In combined EBRT and ICBT, the mean cumulative bladder D2cc calculated by DIR and DA was 86.13 Gy and 86.27 Gy, respectively. The mean cumulative rectal D2cc calculated by DIR and DA was 72.97 Gy and 73.90 Gy, respectively. No significant differences were noted between these two methods (p > 0.05). As to the parameters used to evaluate the DIR accuracy, the mean DSC, Jacobian, MDA (mm) and Hausdorff distance (mm) were 0.79, 1.0, 3.84, and 22.01 respectively for the bladder and 0.53, 1.2, 7.31, and 29.58 respectively for the rectum. In this study, the DSC seemed to be slightly lower compared with previous studies. ConclusionDose accumulation based on DIR might be an alternative method to illustrate and evaluate the cumulative doses of the OARs in combined radiotherapy for cervical cancer. However, DIR should be used with caution before overcoming the relevant limitations.

Leveraging deep learning-based segmentation and contours-driven deformable registration for dose accumulation in abdominal structures

Discrepancies between planned and delivered dose to GI structures during radiation therapy (RT) of liver cancer may hamper the prediction of treatment outcomes. The purpose of this study is to develop a streamlined workflow for dose accumulation in a treatment planning system (TPS) during liver image-guided RT and to assess its accuracy when using different deformable image registration (DIR) algorithms. Fifty-six patients with primary and metastatic liver cancer treated with external beam radiotherapy guided by daily CT-on-rails (CTOR) were retrospectively analyzed. The liver, stomach and duodenum contours were auto-segmented on all planning CTs and daily CTORs using deep-learning methods. Dose accumulation was performed for each patient using scripting functionalities of the TPS and considering three available DIR algorithms based on: (i) image intensities only; (ii) intensities + contours; (iii) a biomechanical model (contours only). Planned and accumulated doses were converted to equivalent dose in 2Gy (EQD2) and normal tissue complication probabilities (NTCP) were calculated for the stomach and duodenum. Dosimetric indexes for the normal liver, GTV, stomach and duodenum and the NTCP values were exported from the TPS for analysis of the discrepancies between planned and the different accumulated doses. Deep learning segmentation of the stomach and duodenum enabled considerable acceleration of the dose accumulation process for the 56 patients. Differences between accumulated and planned doses were analyzed considering the 3 DIR methods. For the normal liver, stomach and duodenum, the distribution of the 56 differences in maximum doses (D2%) presented a significantly higher variance when a contour-driven DIR method was used instead of the intensity only-based method. Comparing the two contour-driven DIR methods, differences in accumulated minimum doses (D98%) in the GTV were >2Gy for 15 (27%) of the patients. Considering accumulated dose instead of planned dose in standard NTCP models of the duodenum demonstrated a high sensitivity of the duodenum toxicity risk to these dose discrepancies, whereas smaller variations were observed for the stomach. This study demonstrated a successful implementation of an automatic workflow for dose accumulation during liver cancer RT in a commercial TPS. The use of contour-driven DIR methods led to larger discrepancies between planned and accumulated doses in comparison to using an intensity only based DIR method, suggesting a better capability of these approaches in estimating complex deformations of the GI organs.

Learning Deformable Image Registration From Optimization: Perspective, Modules, Bilevel Training and Beyond.

Conventional deformable registration methods aim at solving an optimization model carefully designed on image pairs and their computational costs are exceptionally high. In contrast, recent deep learning-based approaches can provide fast deformation estimation. These heuristic network architectures are fully data-driven and thus lack explicit geometric constraints which are indispensable to generate plausible deformations, e.g., topology-preserving. Moreover, these learning-based approaches typically pose hyper-parameter learning as a black-box problem and require considerable computational and human effort to perform many training runs. To tackle the aforementioned problems, we propose a new learning-based framework to optimize a diffeomorphic model via multi-scale propagation. Specifically, we introduce a generic optimization model to formulate diffeomorphic registration and develop a series of learnable architectures to obtain propagative updating in the coarse-to-fine feature space. Further, we propose a new bilevel self-tuned training strategy, allowing efficient search of task-specific hyper-parameters. This training strategy increases the flexibility to various types of data while reduces computational and human burdens. We conduct two groups of image registration experiments on 3D volume datasets including image-to-atlas registration on brain MRI data and image-to-image registration on liver CT data. Extensive results demonstrate the state-of-the-art performance of the proposed method with diffeomorphic guarantee and extreme efficiency. We also apply our framework to challenging multi-modal image registration, and investigate how our registration to support the down-streaming tasks for medical image analysis including multi-modal fusion and image segmentation.

MR-based synthetic CT image for intensity-modulated proton treatment planning of nasopharyngeal carcinoma patients

Purpose To develop an advanced deep convolutional neural network (DCNN) architecture to generate synthetic CT (SCT) images from MR images for intensity-modulated proton therapy (IMPT) treatment planning of nasopharyngeal cancer (NPC) patients. Methods T1-weighted MR images and paired CT (PCT) images were obtained from 206 NPC patients. For each patient, deformable image registration was performed between MR and PCT images to create an MR-CT image pair. Thirty pairs were randomly chosen as the independent test set and the remaining 176 pairs (14 for validation and 162 for training) were used to build two conditional generative adversarial networks (GANs): 1) GAN3D: using a 3D U-net enhanced with residual connections and attentional mechanism as the generator and 2) GAN2D: using a 2D U-net as the generator. For each test patient, SCT images were generated using the generators with the MR images as input and were compared with respect to the corresponding PCT image. A clinical IMPT plan was created and optimized on the PCT image. The dose was recalculated on the SCT images and compared with the one calculated on the PCT image. Results The mean absolute errors (MAEs) between the PCT and SCT, within the body, were (64.89 ± 5.31) HU and (64.31 ± 4.61) HU for the GAN2D and GAN3D. Within the high-density bone (HU > 600), the GAN3D achieved a smaller MAE compared with the GAN2D (p < 0.001). Within the body, the absolute point dose deviation was reduced from (0.58 ± 1.61) Gy for the GAN2D to (0.47 ± 0.94) Gy for the GAN3D. The (3 mm/3%) gamma passing rates were above 97.32% for all SCT images. Conclusions The SCT images generated using GANs achieved clinical acceptable dosimetric accuracy for IMPT of NPC patients. Using advanced DCNN architecture design, such as residual connections and attention mechanism, SCT image quality was further improved and resulted in a small dosimetric improvement.

Fast Deformable Image Registration for Real-Time Target Tracking During Radiation Therapy Using Cine MRI and Deep Learning

We developed a deep learning (DL) model for fast deformable image registration using 2-dimensional sagittal cine magnetic resonance imaging (MRI) acquired during radiation therapy and evaluated its potential for real-time target tracking compared with conventional image registration methods. Our DL model uses a pair of cine MRI images as input and provides a motion vector field (MVF) as output. The MVF is then applied to align the input images. A retrospective study was conducted to train and evaluate our model using cine MRI data from patients undergoing treatment for abdominal and thoracic tumors. For each treatment fraction, MR-linear accelerator delivery log files, tracking videos, and cine image files were analyzed. Individual MRI frames were temporally sampled to construct a large set of image registration pairs used to evaluate multiple methods. The DL model was optimized using 5-fold cross validation, and model outputs (transformed images and MVFs) using test set images were saved for comparison with 3 conventional registration methods (affine, b-spline, and demons). Evaluation metrics were 3-fold: (1) registration error, (2) MVF stability (both spatial and temporal), and (3) average computation time. We analyzed >21 hours of cine MRI (>629,000 frames) acquired during 86 treatment fractions from 21 patients. In a test set of 10,320 image registration pairs, DL registration outperformed conventional methods in both registration error (affine, b-spline, demons, DL; root mean square error: 0.067, 0.040, 0.036, 0.032; paired t test demons vs DL: t[20]=4.2, P < .001) and computation time per frame (51, 1150, 4583, 8 ms). Among deformable methods, spatial stability of resulting MVFs was comparable; however, the DL model had significantly improved temporal consistency. DL-based image registration can leverage large-scale MR cine data sets to outperform conventional registration methods and is a promising solution for real-time deformable motion estimation in radiation therapy.