Registration Of Image Pairs Research Articles

A deep learning approach to remove contrast from contrast-enhanced CT for proton dose calculation.

Non-Contrast Enhanced CT (NCECT) is normally required for proton dose calculation while Contrast Enhanced CT (CECT) is often scanned for tumor and organ delineation. Possible tissue motion between these two CTs raises dosimetry uncertainties, especially for moving tumors in the thorax and abdomen. Here we report a deep-learning approach to generate NCECT directly from CECT. This method could be useful to avoid the NCECT scan, reduce CT simulation time and imaging dose, and decrease the uncertainties caused by tissue motion between otherwise two different CT scans. A deep network was developed to convert CECT to NCECT. The network receives a 3D image from CECT images as input and generates a corresponding contrast-removed NCECT image patch. Abdominal CECT and NCECT image pairs of 20 patients were deformably registered and 8000 image patch pairs extracted from the registered image pairs were utilized to train and test the model. CTs of clinical proton patients and their treatment plans were employed to evaluate the dosimetric impact of using the generated NCECT for proton dose calculation. Our approach achieved a Cosine Similarity score of 0.988 and an MSE value of 0.002. A quantitative comparison of clinical proton dose plans computed on the CECT and the generated NCECT for five proton patients revealed significant dose differences at the distal of beam paths. V100% of PTV and GTV changed by 3.5% and 5.5%, respectively. The mean HU difference for all five patients between the generated and the scanned NCECTs was ∼4.72, whereas the difference between CECT and the scanned NCECT was ∼64.52, indicating a ∼93% reduction in mean HU difference. A deep learning approach was developed to generate NCECTs from CECTs. This approach could be useful for the proton dose calculation to reduce uncertainties caused by tissue motion between CECT and NCECT.

Attention-Aware Non-Rigid Image Registration for Accelerated MR Imaging.

Accurate motion estimation at high acceleration factors enables rapid motion-compensated reconstruction in Magnetic Resonance Imaging (MRI) without compromising the diagnostic image quality. In this work, we introduce an attention-aware deep learning-based framework that can perform non-rigid pairwise registration for fully sampled and accelerated MRI. We extract local visual representations to build similarity maps between the registered image pairs at multiple resolution levels and additionally leverage long-range contextual information using a transformer-based module to alleviate ambiguities in the presence of artifacts caused by undersampling. We combine local and global dependencies to perform simultaneous coarse and fine motion estimation. The proposed method was evaluated on in-house acquired fully sampled and accelerated data of 101 patients and 62 healthy subjects undergoing cardiac and thoracic MRI. The impact of motion estimation accuracy on the downstream task of motion-compensated reconstruction was analyzed. We demonstrate that our model derives reliable and consistent motion fields across different sampling trajectories (Cartesian and radial) and acceleration factors of up to 16x for cardiac motion and 30x for respiratory motion and achieves superior image quality in motion-compensated reconstruction qualitatively and quantitatively compared to conventional and recent deep learning-based approaches. The code is publicly available at https://github.com/lab-midas/GMARAFT.

Dynamic Fusion Network for RGBT Tracking

For both visible and infrared images have their own advantages and disadvantages, RGBT tracking has attracted more and more attention. The key points of RGBT tracking lie in feature extraction and feature fusion of visible and infrared images. Current RGBT tracking methods mostly pay attention to both individual features (features extracted from images of a single camera) and common features (features extracted and fused from an RGB camera and a thermal camera), while pay less attention to the different and dynamic contributions of individual features and common features for different sequences of registered image pairs. This paper proposes a novel RGBT tracking method, called Dynamic Fusion Network (DFNet), which adopts a two-stream structure, in which two non-shared convolution kernels are employed in each layer to extract individual features. Besides, DFNet has shared convolution kernels for each layer to extract common features. Non-shared convolution kernels and shared convolution kernels are adaptively weighted and summed according to different image pairs, so that DFNet can deal with different contributions for different sequences. DFNet has a fast speed, which is 28.658 FPS. The experimental results show that when DFNet only increases the Mult-Adds of 0.02% than the non-shared-convolution-kernel-based fusion method, Precision Rate (PR) and Success Rate (SR) reach 88.1% and 71.9% respectively.

SYNTHETIC CT GENERATION FROM T1-WEIGHTED KNEE MRIs USING A UNET

Despite recent evolution in magnetic resonance imaging (MRI) techniques for musculoskeletal applications, computed tomography (CT) remains the reference modality for the assessment of bone structure. The use of generative deep learning models, such as U-Nets, was shown to enable the synthesis of CT-like contrasts from MRI images. However, the development and validation of such tools have been hindered by the need for large datasets with paired CT and MRI acquisitions. In this preliminary work, we propose to train a U-Net, a supervised deep learning technique, to generate synthetic CT (sCT) knee images from three-dimensional T1-weighted MRI scans by leveraging a large knee dataset with paired acquisitions. The synthetic CT images were then assessed quantitatively and qualitatively. A cohort consisting of 249 patients (39.7±16.0 years old, 133 females) received both a knee MR examination (3T MAGNETOM Prismafit, Siemens Healthcare, Erlangen, Germany) and a CT scan (Revolution, General Electronics Healthcare). T1-weighted MR images (TR 700ms, TE 11ms, 0.5mm isotropic) were spatially registered to the down-sampled CT data (originally 0.3mm isotropic). To ensure a good voxel-to-voxel correspondence, the 99 best registered image pairs were selected and split between training (80%), testing (10%) and validation sets (10%). During training, 100 central slices were extracted for each orientation (axial, coronal and sagittal) and fed to a 2.5D network as stacks of three consecutive MRI slices. At inference, sCT slices were generated for each orientation, and the voxel-wise median across orientations was computed. Qualitatively, our method successfully generated images with a CT-like contrast exhibiting satisfactory levels of anatomical details, including bone contours, and the femoral and tibial physes. However, the sCT images looked generally oversmoothed compared to the original CT data, hindering the visualization of some of the bone trabeculae, especially in the epiphyses. Some anatomical details such as vascular canals were not depicted accurately. In terms of quantitative evaluation, our model achieved a mean average error of 167±23.1 and 49.5±6.76 Hounsfield Units (HU) in bone and soft tissue, respectively. In this preliminary work, we showed the feasibility of generating sCT images from T1-weighted MR data with a good level of anatomical details and quantitative HU estimation. Future work will focus on reducing the impact of registration errors to further improve the model accuracy.

Unsupervised deformable image registration network for 3D medical images

Image registration aims to establish an active correspondence between a pair of images. Such correspondence is critical for many significant applications, such as image fusion, tumor growth monitoring, and atlas generation. In this study, we propose an unsupervised deformable image registration network (UDIR-Net) for 3D medical images. The proposed UDIR-Net is designed in an encoder-decoder architecture and directly estimates the complex deformation field between input pairwise images without any supervised information. In particular, we recalibrate the feature slice of each feature map that is propagated between the encoder and the decoder in accordance with the importance of each feature slice and the correlation between feature slices. This method enhances the representational power of feature maps. To achieve efficient and robust training, we design a novel hierarchical loss function that evaluates multiscale similarity loss between registered image pairs. The proposed UDIR-Net is tested on different public magnetic resonance image datasets of the human brain. Experimental results show that UDIR-Net exhibits competitive performance against several state-of-the-art methods.

Mode-enhanced space-time DIC: applications to ultra-high-speed imaging

Digital image correlation (DIC), which consists in registering image pairs to measure displacement fields, can be tailored to analyze image sequences from videos taking advantage of a common reference image. In the present study it is no longer the first image of the sequence but rather a computed image from the entire video. The sought kinematics, which is separated in space and time, offers the opportunity to extract ‘modes’, each of which is a spatial displacement field multiplied by a scalar function of time only. These modes are not chosen a priori but rather computed from a specific formulation of DIC so that they capture the displacements at best. The exploited mathematical technique to achieve this modal representation is a form of proper generalized decomposition that makes use of the DIC variational formulation, where both spatial and temporal regularizations can be included. Two image videos acquired with an ultra-high speed camera at 5 and 10 million frames per second are analyzed to illustrate the proposed technique. Very large computation time gains are obtained with no noticeable differences in the kinematic measurements. Moreover, it is shown that motions have a lower complexity, i.e. require less modes, than the direct proper orthogonal decomposition (or principal component analysis) performed on the entire video.

Segmentation of parotid glands from registered CT and MR images

PurposeTo develop an automatic multimodal method for segmentation of parotid glands (PGs) from pre-registered computed tomography (CT) and magnetic resonance (MR) images and compare its results to the results of an existing state-of-the-art algorithm that segments PGs from CT images only. MethodsMagnetic resonance images of head and neck were registered to the accompanying CT images using two different state-of-the-art registration procedures. The reference domains of registered image pairs were divided on the complementary PG regions and backgrounds according to the manual delineation of PGs on CT images, provided by a physician. Patches of intensity values from both image modalities, centered around randomly sampled voxels from the reference domain, served as positive or negative samples in the training of the convolutional neural network (CNN) classifier. The trained CNN accepted a previously unseen (registered) image pair and classified its voxels according to the resemblance of its patches to the patches used for training. The final segmentation was refined using a graph-cut algorithm, followed by the dilate-erode operations. ResultsUsing the same image dataset, segmentation of PGs was performed using the proposed multimodal algorithm and an existing monomodal algorithm, which segments PGs from CT images only. The mean value of the achieved Dice overlapping coefficient for the proposed algorithm was 78.8%, while the corresponding mean value for the monomodal algorithm was 76.5%. ConclusionsAutomatic PG segmentation on the planning CT image can be augmented with the MR image modality, leading to an improved RT planning of head and neck cancer.

Street-view change detection with deconvolutional networks

We propose a system for performing structural change detection in street-view videos captured by a vehicle-mounted monocular camera over time. Our approach is motivated by the need for more frequent and efficient updates in the large-scale maps used in autonomous vehicle navigation. Our method chains a multi-sensor fusion SLAM and fast dense 3D reconstruction pipeline, which provide coarsely registered image pairs to a deep Deconvolutional Network (DN) for pixel-wise change detection. We investigate two DN architectures for change detection, the first one is based on the idea of stacking contraction and expansion blocks while the second one is based on the idea of Fully Convolutional Networks. To train and evaluate our networks we introduce a new urban change detection dataset which is an order of magnitude larger than existing datasets and contains challenging changes due to seasonal and lighting variations. Our method outperforms existing literature on this dataset, which we make available to the community, and an existing panoramic change detection dataset, demonstrating its wide applicability.

Multimodal registration of remotely sensed images based on Jeffrey’s divergence

Entropy-based measures (e.g., mutual information, also known as Kullback-Leiber divergence), which quantify the similarity between two signals, are widely used as similarity measures for image registration. Although they are proven superior to many classical statistical measures, entropy-based measures, such as mutual information, may fail to yield the optimum registration if the multimodal image pair has insufficient scene overlap region. To overcome this challenge, we proposed using the symmetric form of Kullback-Leiber divergence, namely Jeffrey’s divergence, as the similarity measure in practical multimodal image registration tasks. Mathematical analysis was performed to investigate the causes accounting for the limitation of mutual information when dealing with insufficient scene overlap image pairs. Experimental registrations of SPOT image, Landsat TM image, ALOS PalSAR image, and DEM data were carried out to compare the performance of Jeffrey’s divergence and mutual information. Results indicate that Jeffrey’s divergence is capable of providing larger feasible search space, which is favorable for exploring optimum transformation parameters in a larger range. This superiority of Jeffrey’s divergence was further confirmed by a series of paradigms. Thus, the proposed model is more applicable for registering image pairs that are greatly misaligned or have an insufficient scene overlap region.

Confidence Estimation for Medical Image Registration Based On Stereo Confidences.

In this paper, we propose a novel method to estimate the confidence of a registration that does not require any ground truth, is independent from the registration algorithm and the resulting confidence is correlated with the amount of registration error. We first apply a local search to match patterns between the registered image pairs. Local search induces a cost space per voxel which we explore further to estimate the confidence of the registration similar to confidence estimation algorithms for stereo matching. We test our method on both synthetically generated registration errors and on real registrations with ground truth. The experimental results show that our confidence measure can estimate registration errors and it is correlated with local errors.

An Ensemble Approach to Image Matching Using Contextual Features.

We propose a contextual framework for 2D image matching and registration using an ensemble feature. Our system is beneficial for registering image pairs that have captured the same scene but have large visual discrepancies between them. It is common to encounter challenging visual variations in image sets with artistic rendering differences or in those collected over a period of time during which the lighting conditions and scene content may have changed. Differences between images may also be caused using a variety of cameras with different sensors, focal lengths, and exposure values. Local feature matching techniques cannot always handle these difficulties, so we have developed an approach that builds on traditional methods to consider linear and histogram of gradient information over a larger, more stable region. We also present a technique for using linear features to estimate corner keypoints, or pseudo corners, that can be used for matching. Our pipeline follows this unique matching stage with homography refinement methods using edge and gradient information. Our goal is to increase the size of accurate keypoint match sets and align photographs containing a combination of man-made and natural imagery. We show that incorporating contextual information can provide complimentary information for scale invariant feature transform and boost local keypoint matching performance, as well as be used to describe corner feature points.

A hybrid framework for registration of carotid ultrasound images combining iconic and geometric features

Stroke is the third major cause of death worldwide behind heart disease and cancer. Carotid atherosclerosis is the most frequent cause of ischemic stroke. Early diagnosis of carotid plaque and serial monitoring of its size with the help of imaging modalities can help to prevent the atherosclerotic complications. The main difficulty is inevitable variability of patient's head positions during image acquisitions. The time series registration of carotid images helps to improve the monitoring, characterization, and quantification of the disease by suppressing the global movements of the patient. In this work, a novel hybrid registration technique has been proposed and evaluated for registration of carotid ultrasound images taken at different times. The proposed hybrid method bridges the gap between the feature-based and intensity-based registration methods. The feature-based iterative closest point algorithm is used to provide a coarse registration which is subsequently refined by the intensity-based algorithm. The proposed framework uses rigid transformation model coupled with mutual information (MI) similarity measure and Powell optimizer. For quantitative evaluation, different registration approaches have been compared using four error metrics: visual information fidelity, structural similarity index, correlation coefficient, and MI. Qualitative evaluation has also been done using the visual examination of the registered image pairs.

Reproducibility of intensity‐based estimates of lung ventilation

Lung function depends on lung expansion and contraction during the respiratory cycle. Respiratory-gated CT imaging and image registration can be used to estimate the regional lung volume change by observing CT voxel density changes during inspiration or expiration. In this study, the authors examine the reproducibility of intensity-based estimates of lung tissue expansion and contraction in three mechanically ventilated sheep and ten spontaneously breathing humans. The intensity-based estimates are compared to the estimates of lung function derived from image registration deformation field. 4DCT data set was acquired for a cohort of spontaneously breathing humans and anesthetized and mechanically ventilated sheep. For each subject, two 4DCT scans were performed with a short time interval between acquisitions. From each 4DCT data set, an image pair consisting of a volume reconstructed near end inspiration and a volume reconstructed near end exhalation was selected. The end inspiration and end exhalation images were registered using a tissue volume preserving deformable registration algorithm. The CT density change in the registered image pair was used to compute intensity-based specific air volume change (SAC) and the intensity-based Jacobian (IJAC), while the transformation-based Jacobian (TJAC) was computed directly from the image registration deformation field. IJAC is introduced to make the intensity-based and transformation-based methods comparable since SAC and Jacobian may not be associated with the same physiological phenomenon and have different units. Scan-to-scan variations in respiratory effort were corrected using a global scaling factor for normalization. A gamma index metric was introduced to quantify voxel-by-voxel reproducibility considering both differences in ventilation and distance between matching voxels. The authors also tested how different CT prefiltering levels affected intensity-based ventilation reproducibility. Higher reproducibility was found for anesthetized mechanically ventilated animals than for the humans for both the intensity-based (IJAC) and transformation-based (TJAC) ventilation estimates. The human IJAC maps had scan-to-scan correlation coefficients of 0.45 ± 0.14, a gamma pass rate 70 ± 8 without normalization and 75 ± 5 with normalization. The human TJAC maps had correlation coefficients 0.81 ± 0.10, a gamma pass rate 86 ± 11 without normalization and 93 ± 4 with normalization. The gamma pass rate and correlation coefficient of the IJAC maps gradually increased with increased smoothing, but were still much lower than those of the TJAC maps. The transformation-based ventilation maps show better reproducibility than the intensity-based maps, especially in human subjects. Reproducibility was also found to depend on variations in respiratory effort; all techniques were better when applied to images from mechanically ventilated sheep compared to spontaneously breathing human subjects. Nevertheless, intensity-based techniques applied to mechanically ventilated sheep were less reproducible than the transformation-based applied to spontaneously breathing humans, suggesting the method used to determine ventilation maps is important. Prefiltering of the CT images may help to improve the reproducibility of the intensity-based ventilation estimates, but even with filtering the reproducibility of the intensity-based ventilation estimates is not as good as that of transformation-based ventilation estimates.

AccPbFRET: An ImageJ plugin for semi-automatic, fully corrected analysis of acceptor photobleaching FRET images

BackgroundThe acceptor photobleaching fluorescence resonance energy transfer (FRET) method is widely used for monitoring molecular interactions in cells. This method of FRET, while among those with the simplest mathematics, is robust, self-controlled and independent of fluorophore amounts and ratios.ResultsAccPbFRET is a user-friendly, efficient ImageJ plugin which allows fully corrected, pixel-wise calculation and detailed, ROI (region of interest)-based analysis of FRET efficiencies in microscopic images. Furthermore, automatic registration and semi-automatic analysis of large image sets is provided, which are not available in any existing FRET evaluation software.ConclusionDespite of the widespread applicability of the acceptor photobleaching FRET technique, this is the first paper where all possible sources of major errors of the measurement and analysis are considered, and AccPbFRET is the only program which provides the complete suite of corrections – for registering image pairs, for unwanted photobleaching of the donor, for cross-talk of the acceptor and/or its photoproduct to the donor channel and for partial photobleaching of the acceptor. The program efficiently speeds up the analysis of large image sets even for novice users and is freely available.

Vector-valued local frequency representation for robust multimodal image registration.

Automatic registration of multi-modal images involves algorithmically estimating the coordinate transformation required to align the data sets. Most existing methods in literature are unable to cope with registration of image pairs with large non-overlapping field of view (FOV). We propose a robust algorithm, based on matching vector-valued local frequency (corresponding to the dominant local frequency magnitude) image representations, which can cope with image pairs with large non-overlapping FOV. The work reported here is an extension of our earlier work which used dominant local frequency magnitude based representations of the input data as input to the matching algorithm. The local frequency representation naturally allows for processing the data at different scales/resolutions, a very desirable property from a computational efficiency view point. Our algorithm involves minimizing - over all affine transformations - the integral of the squared error (ISE or L2E) between a Gaussian model of the residual and its true density function. The residual here refers to the difference between the local frequency representations of the transformed (by an unknown transformation) source and target data. We present implementation results for misalignments between MR brain scans obtained using different image acquisition protocols.

Simultaneous homographic and comparametric alignment of multiple exposure-adjusted pictures of the same scene

An approach is presented that can simultaneously align multiple exposure-adjusted pictures of the same scene both in their spatial coordinates as well as in their pixel values. The approach is featureless and produces an image mosaic at a common spatial and exposure reference and also addresses the misalignment problem common to methods that compose mosaics from only pair-wise registered image pairs. The objective function considered minimizes the sum of the collective variance over pixels of a global coordinate grid on which to create the final image. The models employed relate images spatially by homographic transformations and tonally by comparametric functions. The importance of performing joint spatial and tonal registration on exposure-adjusted images is emphasized by providing two examples in which spatial-only registration fails. A discussion describing the performance between pair-wise and simultaneous registration under both spatial-only and joint registration procedures is provided.

Jointly registering images in domain and range by piecewise linear comparametric analysis

This paper describes an approach whereby comparametric analysis is used in jointly registering image pairs in their domain and range, i.e., in their spatial coordinates and pixel values, respectively. This is accomplished by approximating a camera's nonlinear comparametric function with a constrained piecewise linear one. The optimal fitting of this approximation to comparagram data is then used in a re-parameterized version of the camera's comparametric function to estimate the exposure difference between images. Doing this allows the inherently nonlinear problem of joint domain and range registration to be performed using a computationally attractive least squares formalism. The paper first presents the range registration process and then describes the strategy for performing the joint registration. The models used allow for the pair-wise registration of images taken from a camera that can automatically adjust its exposure as well as tilt, pan, rotate and zoom about its optical center. Results concerning the joint registration as well as range-only registration are provided to demonstrate the method's effectiveness.

Blind cross-spectral image registration using prefiltering and Fourier-based translation detection

This paper describes a fast technique for registering image pairs from visible and infrared spectra that differ by translation, small rotations, and small changes of scale. The main result of this paper is a nonlinear prefiltering and thresholding technique that substantially enhances the cross-spectral correlation, provided that the image pairs have many features in common. We show the use of the technique in conjunction with a Fourier-based normalized correlation algorithm to perform fast cross-spectral registrations. In the absence of such prefiltering, local reversals of contrast from image to image tend, to impair the quality of correlation-based registrations. The algorithm computes the translation that maximizes the overall normalized correlation of the filtered images without examining individual image features. Small rotations and scale changes can be recovered by computing the translation displacement in several different regions of the image pairs. In an experiment on a moderate-sized database, the algorithm produced a correct registration rate of over 90% at a false positive rate of less than 10%. For a particularly difficult subset of images in the database, the correct registration rate fell to approximately 85% at a false positive rate of less than 10%. This retrieval quality is comparable to that of a mutual-information-based algorithm.

A feature-based technique for joint, linear estimation of high-order image-to-mosaic transformations: mosaicing the curved human retina

An algorithm for constructing image mosaics from multiple, uncalibrated, weak-perspective views of the human retina is presented and analyzed. It builds on an algorithm for registering pairs of retinal images using a noninvertible, 12-parameter, quadratic image transformation model and hierarchical, robust estimation. The major innovation presented is a linear, feature-based, noniterative method for jointly estimating consistent transformations of all images onto the mosaic image. Constraints for this estimation are derived from pairwise registration both directly with the anchor image and indirectly between pairs of nonanchor images. An incremental, graph-based technique constructs the set of registered image pairs used in the solution. The estimation technique allows images that do not overlap the anchor frame to be successfully mosaiced, a valuable capability for mosaicing images of the retinal periphery. Experimental analysis on data sets from 16 eyes shows the average overall median transformation error in final mosaic to be 0.76 pixels. The technique is simpler, more accurate, and offers broader coverage than previously published methods.