Multiresolution Image Registration Research Articles

Uncertainty estimation and evaluation of deformation image registration based convolutional neural networks

Objective. Fast and accurate deformable image registration (DIR), including DIR uncertainty estimation, is essential for safe and reliable clinical deployment. While recent deep learning models have shown promise in predicting DIR with its uncertainty, challenges persist in proper uncertainty evaluation and hyperparameter optimization for these methods. This work aims to develop and evaluate a model that can perform fast DIR and predict its uncertainty in seconds. Approach. This study introduces a novel probabilistic multi-resolution image registration model utilizing convolutional neural networks to estimate a multivariate normal distributed dense displacement field (DDF) in a multimodal image registration problem. To assess the quality of the DDF distribution predicted by the model, we propose a new metric based on the Kullback–Leibler divergence. The performance of our approach was evaluated against three other DIR algorithms (VoxelMorph, Monte Carlo dropout, and Monte Carlo B-spline) capable of predicting uncertainty. The evaluation of the models included not only the quality of the deformation but also the reliability of the estimated uncertainty. Our application investigated the registration of a treatment planning computed tomography (CT) to follow-up cone beam CT for daily adaptive radiotherapy. Main results. The hyperparameter tuning of the models showed a trade-off between the estimated uncertainty’s reliability and the deformation’s accuracy. In the optimal trade-off, our model excelled in contour propagation and uncertainty estimation (p <0.05) compared to existing uncertainty estimation models. We obtained an average dice similarity coefficient of 0.89 and a KL-divergence of 0.15. Significance. By addressing challenges in DIR uncertainty estimation and evaluation, our work showed that both the DIR and its uncertainty can be reliably predicted, paving the way for safe deployment in a clinical environment.

Unsupervised registration for liver CT-MR images based on the multiscale integrated spatial-weight module and dual similarity guidance.

Multimodal registration is a key task in medical image analysis. Due to the large differences of multimodal images in intensity scale and texture pattern, it is a great challenge to design distinctive similarity metrics to guide deep learning-based multimodal image registration. Besides, since the limitation of the small receptive field, existing deep learning-based methods are mainly suitable for small deformation, but helpless for large deformation. To address the above issues, we present an unsupervised multimodal image registration method based on the multiscale integrated spatial-weight module and dual similarity guidance. In this method, a U-shape network with our multiscale integrated spatial-weight module is embedded into a multi-resolution image registration architecture to achieve end-to-end large deformation registration, where the spatial-weight module can effectively highlight the regions with large deformation and aggregate discriminative features, and the multi-resolution architecture further helps to solve the optimization problem of the network in a coarse-to-fine pattern. Furthermore, we introduce a special loss function based on dual similarity, which represents both global gray-scale similarity and local feature similarity, to optimize the unsupervised multimodal registration network. We verified the effectiveness of the proposed method on liver CT-MR images. Experimental results indicate that the proposed method achieves the optimal DSC value and TRE value of 92.70±1.75(%) and 6.52±2.94(mm), compared with other state-of-the-art registration algorithms. The proposed method can accurately estimate the large deformation field by aggregating multiscale features, and achieve higher registration accuracy and fast registration speed. Comparative experiments also demonstrate the effectiveness and generalization ability of the algorithm.

Multi-resolution transmission image registration based on “Terrace Compression Method” and normalized mutual information

Multispectral transmission images provide the possibility for the early diagnosis of breast cancer, and promote the study of family self-screening of breast tumors. However, the scattering characteristic of biological tissue leads to transmission images with fuzzy boundaries and poor contrast. And there are different offsets in the captured image sequences because of the instability of the human body such as breathing and slight jitter, which will affect the data precision. In view of the above problems, a multi-resolution image registration method combining “Terrace Compression Method” and normalized mutual information (NMI) is proposed, and the effectiveness of the method is verified by taking the transmission image of wavelength at 435 ​nm as an example. First of all, the gray-scale values of the selected boundary region of heterogeneity are sorted. According to the “Terrace Gradient” of gray-scale values, multiple gray-scale intervals are divided to enhance the gradient information of the image. Then, the image of the terraced shape of each gray-scale interval is extracted respectively, and the edge detection result is obtained by the Sobel operator. Finally, the Gaussian pyramid model is used for image down-sampling to achieve multi-resolution image registration by combining with NMI similarity measure. Compared with the other registration methods, the method proposed in this paper can detect the transformation relationship between images more accurately and effectively, and the similarity of the registered image with the reference image is also higher. The combination of the “Terrace Compression Method” and NMI effectively improves the registration accuracy of transmission images, and provides favorable conditions for achieving heterogeneity detection in multispectral transmission images.

Robust Registration of Medical Images in the Presence of Spatially-Varying Noise

Spatially-varying intensity noise is a common source of distortion in medical images and is often associated with reduced accuracy in medical image registration. In this paper, we propose two multi-resolution image registration algorithms based on Empirical Mode Decomposition (EMD) that are robust against additive spatially-varying noise. EMD is a multi-resolution tool that decomposes a signal into several principle patterns and residual components. Our first proposed algorithm (LR-EMD) is based on the registration of EMD feature maps from both floating and reference images in various resolutions. In the second algorithm (AFR-EMD), we first extract a single average feature map based on EMD and then use a simple hierarchical multi-resolution algorithm to register the average feature maps. We then showcase the superior performance of both algorithms in the registration of brain MRIs as well as retina images. For the registration of brain MR images, using mutual information as the similarity measure, both AFR-EMD and LR-EMD achieve a lower error rate in intensity (42% and 32%, respectively) and lower error rate in transformation (52% and 41%, respectively) compared to intensity-based hierarchical registration. Our results suggest that the two proposed algorithms offer robust registration solutions in the presence of spatially-varying noise.

MDReg-Net: Multi-resolution diffeomorphic image registration using fully convolutional networks with deep self-supervision.

We present a diffeomorphic image registration algorithm to learn spatial transformations between pairs of images to be registered using fully convolutional networks (FCNs) under a self‐supervised learning setting. Particularly, a deep neural network is trained to estimate diffeomorphic spatial transformations between pairs of images by maximizing an image‐wise similarity metric between fixed and warped moving images, similar to those adopted in conventional image registration algorithms. The network is implemented in a multi‐resolution image registration framework to optimize and learn spatial transformations at different image resolutions jointly and incrementally with deep self‐supervision in order to better handle large deformation between images. A spatial Gaussian smoothing kernel is integrated with the FCNs to yield sufficiently smooth deformation fields for diffeomorphic image registration. The spatial transformations learned at coarser resolutions are utilized to warp the moving image, which is subsequently used as input to the network for learning incremental transformations at finer resolutions. This procedure proceeds recursively to the full image resolution and the accumulated transformations serve as the final transformation to warp the moving image at the finest resolution. Experimental results for registering high‐resolution 3D structural brain magnetic resonance (MR) images have demonstrated that image registration networks trained by our method obtain robust, diffeomorphic image registration results within seconds with improved accuracy compared with state‐of‐the‐art image registration algorithms.

NON-RIGID IMAGE REGISTRATION USING SELF-SUPERVISED FULLY CONVOLUTIONAL NETWORKS WITHOUT TRAINING DATA.

A novel non-rigid image registration algorithm is built upon fully convolutional networks (FCNs) to optimize and learn spatial transformations between pairs of images to be registered in a self-supervised learning framework. Different from most existing deep learning based image registration methods that learn spatial transformations from training data with known corresponding spatial transformations, our method directly estimates spatial transformations between pairs of images by maximizing an image-wise similarity metric between fixed and deformed moving images, similar to conventional image registration algorithms. The image registration is implemented in a multi-resolution image registration framework to jointly optimize and learn spatial transformations and FCNs at different spatial resolutions with deep self-supervision through typical feedforward and backpropagation computation. The proposed method has been evaluated for registering 3D structural brain magnetic resonance (MR) images and obtained better performance than state-of-the-art image registration algorithms.

Multiresolution image registration for multimodal brain images and fusion for better neurosurgical planning

BackgroundImaging modalities in medicine gives complementary information. Inadequacy in clinical information made single imaging modality insufficient. There is a need for computer-based system that permits rapid acquisition of digital medical images and performs multi-modality registration, segmentation and three-dimensional planning of minimally invasive neurosurgical procedures. In this regard proposed article presents multimodal brain image registration and fusion for better neurosurgical planning. MethodsIn proposed work brain data is acquired from Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) modalities. CT and MRI images are pre-processed and given for image registration. BSpline deformable registration and multiresolution image registration is performed on the CT and MRI sequence. CT is fixed image and MRI is moving image for registration. Later end result is fusion of CT and registered MRI sequences. ResultsBSpline deformable registration is performed on the slices gave promising results but on the sequences noise have been introduced in the resultant image because of multimodal and multiresolution input images. Then multiresolution registration technique is performed on the CT and MRI sequence of the brain which gave promising results. ConclusionThe end resultant fused images are validated by the radiologists and mutual information measure is used to validate registration results. It is found that CT and MRI sequence with more number of slices gave promising results. Few cases with deformation during misregistrations recorded with low mutual information of about 0.3 and which is not acceptable and few recorded with 0.6 and above mutual information during registration gives promising results.

Multiresolution Search of the Rigid Motion Space for Intensity-Based Registration.

We study the relation between the correlation-based target functions of low-resolution and high-resolution intensity-based registration for the class of rigid transformations. Our results show that low-resolution target values can tightly bound the high-resolution target function in natural images. This can help with analyzing and better understanding the process of multiresolution image registration. It also gives a guideline for designing multiresolution algorithms in which the search space in higher resolution registration is restricted given the fitness values for lower resolution image pairs. To demonstrate this, we incorporate our multiresolution technique into a Lipschitz global optimization framework. We show that using the multiresolution scheme can result in large gains in the efficiency of such algorithms. The method is evaluated by applying to the problems of 2D registration, 3D rotation search, and the detection of reflective symmetry in 2D and 3D images.

Making People's Life Much Easier by Effective Face Image Retrieval from the Popular Online Social Network such as Facebook

The primary objective of this paper is to make people's life much easier by retrieving images containing faces of the same person appearing in the query image, from the popular online social network like Facebook (web-scale image database) containing millions of face images effectively and accurately. In this work face image of a person will be used as a query image to retrieve images of the same person with various poses, expressions, and illumination. Such face retrieval system has many applications, including name-based face image search, face tagging in images and videos, copyright enforcement, labeling faces in photos from the popular online social network such as Facebook etc. Also, this paper proposes a method for Automatic Image Registration in which pair of images with differences in rotation, translation, and spectral content is registered for remote sensing applications using multisensors. This work identifies that an image edge feature extraction using perceptional organization methods based on the rules of human perception are allowed us to reduce noise and over segmentation problem in image feature extraction. Generic edge tokens are perceptually significant edge features which are descriptive in the representation and therefore can be manipulated qualitatively in supporting perceptional grouping. This paper provided a new technique that can speed up registration (by reducing data) and improve accuracy by selecting a subimage for the registration process based on its ability to provide accurate results and determined a faster method for implementing multi-resolution image registration. The proposed work not only reduces the computation cost of future work, but it also greatly reduces the complexity and confusion caused by over-segmentation.

Multiresolution Image Registration in Digital X-Ray Angiography with Intensity Variation Modeling

Digital subtraction angiography (DSA) is a widely used technique for visualization of vessel anatomy in diagnosis and treatment. However, due to unavoidable patient motions, both externally and internally, the subtracted angiography images often suffer from motion artifacts that adversely affect the quality of the medical diagnosis. To cope with this problem and improve the quality of DSA images, registration algorithms are often employed before subtraction. In this paper, a novel elastic registration algorithm for registration of digital X-ray angiography images, particularly for the coronary location, is proposed. This algorithm includes a multiresolution search strategy in which a global transformation is calculated iteratively based on local search in coarse and fine sub-image blocks. The local searches are accomplished in a differential multiscale framework which allows us to capture both large and small scale transformations. The local registration transformation also explicitly accounts for local variations in the image intensities which incorporated into our model as a change of local contrast and brightness. These local transformations are then smoothly interpolated using thin-plate spline interpolation function to obtain the global model. Experimental results with several clinical datasets demonstrate the effectiveness of our algorithm in motion artifact reduction.

Using three-dimensional multigrid-based snake and multiresolution image registration for reconstruction of cranial defect

In cranioplasty, neurosurgeons use bone grafts to repair skull defects. To ensure the protection of intracranial tissues and recover the original head shape for aesthetic purposes, a custom-made pre-fabricated prosthesis must match the cranial incision as closely as possible. In our previous study (Liao et al. in Med Biol Eng Comput 49:203-211, 2011), we proposed an algorithm consisting of the 2D snake and image registration using the patient's own diagnostic low-resolution and defective high-resolution computed tomography (CT) images to repair the impaired skull. In this study, we developed a 3D multigrid snake and employed multiresolution image registration to improve the computational efficiency. After extracting the defect portion images, we designed an image-trimming process to remove the bumped inner margin that can facilitate the placement of skull implants without manual trimming during surgery. To evaluate the performance of the proposed algorithm, a set of skull phantoms were manufactured to simulate six different conditions of cranial defects, namely, unilateral, bilateral, and cross-midline defects with 20 or 40% skull defects. The overall image processing time in reconstructing the defect portion images can be reduced from 3 h to 20 min, as compared with our previous method. Furthermore, the reconstruction accuracies using the 3D multigrid snake were superior to those using the 2D snake.

Error analysis of corner and center points for image registration

Image registration is an active research topic in the fields of remote sensing, photogrammetry, computer vision, pattern recognition, and medical image processing. Both corner and geometric center points are used for image registration, but not much attention has been paid to date to the differences in the outcome of image registration when these different types of points are used. Such differences have the potential to directly affect the success of multiresolution or multimodal image registration. The goal of this paper is to compare the characteristics of corner and center points in image registration and to quantify the differences in the registration accuracy when different features are used. Corners and geometric centers were compared in two cases: registration of images with the same resolution and registration of images with different resolutions. The results show that geometric centers provide more accurate results for image registration in both cases. Quantitative analysis revealed that the position error of the centers is only about 47%–63% of the position error of the corners. Experiments confirmed this finding.

Multiresolution image registration based on tree data structures

This paper presents a new approach for image registration based on the partitioning of the two source images in binary-space and quadtree structures, obtained with a maximum mutual information gain algorithm. Two different implementation approaches that differ in the level at which information is considered have been studied. The first works at pixel level using the simplified images directly, while the second works at node level dealing with the tree data structure. The obtained results show an outstanding accuracy and robustness of the proposed methods. In particular, the use of binary-space partitioned images drastically reduces the grid effects in comparison with regular downsampled images. An important advantage of our approach comes from the reduced size of the data structures corresponding to the simplified images, which makes this method appropriate to be applied in a multiresolution framework and telemedicine applications.

SU-FF-I-97: Comparison of End of Exhalation (EE) 4DCT Phase and Breadth-Hold Scans

Purpose: 1)compare congruency of phase 50 with EE breath‐hold scan 2)demonstrate improvement of the demons image registration algorithm with hierarchical regularization component. Method and Materials: We consider 10 cases with motion exceeding 0.5 cm. The tumor was contoured on phase 50. Image registration provided the transfer of a contour to EE scan. Contour comparison is afforded by the same DICOM‐coordinates. Dice coefficient and centroids' shift between scans gauge tumor congruence. Multiresolution image registration is based on the fluid version of the demons algorithm. Algorithm is improved by the hierarchical regularization component. SSD is used to compare image fusions. Deformation maps are compared and described with maximal and mean magnitude of the deformation vector and its standard deviation. Number of iteration reveals the speed up of performance. Results: The tumor volumes ranged from 2.27cc to 37.21cc on phase 50. Transferred to EE differed on average by 7%. Tumor centroids shifts ranged upto 5 mm along the lateral, upto 7mm along ventro‐dorsal and upto 7 mm along the superior‐inferior axis. The ipsilateral lung volume between the scans differed upto 15%. The Dice coefficient did not exceed 0.7. Adaptive regularization accelerates convergence and improves on all scoring criteria. Conclusion: Data indicate incongruency between phase 50 and EE breath‐hold scan, which precludes the latter in any emulation or substitution for phase 50 scan. Possible reasons are: relatively small tumor sizes, motion above 0.5cm and different air intake in both scans. Hierarchical application of the deformation field regularization provides more robust performance and heuristically can be seen as an adaptation to scale‐size changes during iterations.

Motion-compensated MR valve imaging with COMB tag tracking and super-resolution enhancement

Understanding the morphology and function of heart valves is important to the study of underlying causes of heart failure. Existing techniques such as those based on echocardiography are limited by the relatively low signal-to-noise ratio (SNR), attenuation artefacts, and restricted access. The alternative of cardiovascular MR imaging offers versatility and accuracy in 3D localisation, but is hampered by large movements of the valves throughout the cardiac cycle. This paper presents a motion-compensated adaptive imaging approach for MR valve imaging. To illustrate its clinical potential, 3D motion of the aortic valve plane is first captured through a single breath-hold COMB tag pre-scan and then tracked in real-time with an automatic method based on multi-resolution image registration. Motion-compensated coverage of the aortic valve is then acquired prospectively, thus allowing its clear 3D reconstruction and visualisation. To provide isotropic voxel coverage of the imaging volume, retrospective projection onto convex sets (POCS) super-resolution enhancement is applied to the slice-select direction. In vivo results demonstrate the effectiveness of the proposed motion-compensation and super-resolution schemes for depicting the structure of the valve leaflets throughout the cardiac cycle. The proposed method fundamentally changes the way MR imaging is performed by transforming it from a spatially to materially localised imaging method. This also has important implications for quantifying blood flow and myocardial perfusion, as well as tracking anatomy and function of the heart.

Examination of 2‐DE in the Human Proteome Organisation Brain Proteome Project pilot studies with the new RAIN gel matching technique

The Human Proteome Organisation (HUPO) Brain Proteome Project (BPP) pilot studies have generated over 200 2-D gels from eight participating laboratories. This data includes 67 single-channel and 60 DIGE gels comparing 30 whole frozen C57/BL6 female mouse brains, ten each at embryonic day 16, postnatal day 7 (juvenile) and postnatal day 54-56 (adult); and ten single-channel and three DIGE gels comparing human epilepsy surgery of the temporal front lobe with a corresponding post-mortem specimen. The samples were generated centrally and distributed to the participating laboratories, but otherwise no restrictions were placed on sample preparation, running and staining protocols, nor on the 2-D gel analysis packages used. Spots were characterised by MS and the annotated gel images published on a ProteinScape web server. In order to examine the resultant differential expression and protein identifications, we have reprocessed a large subset of the gels using the newly developed RAIN (Robust Automated Image Normalisation) 2-D gel matching algorithm. Traditional approaches use symbolic representation of spots at the very early stages of the analysis, which introduces persistent errors due to inaccuracies in spot modelling and matching. With RAIN, image intensity distributions, rather than selected features, are used, where smooth geometric deformation and expression bias are modelled using multi-resolution image registration and bias-field correction. The method includes a new approach of volume-invariant warping which ensures the volume of protein expression under transformation is preserved. An image-based statistical expression analysis phase is then proposed, where small insignificant expression changes over one gel pair can be revealed when reinforced by the same consistent changes in others. Results of the proposed method as applied to the HUPO BPP data show significant intra-laboratory improvements in matching accuracy over a previous state-of-the-art technique, Multi-resolution Image Registration (MIR), and the commercial Progenesis PG240 package.

Navigation system based on multiresolution image registration for bronchofiberoscopic procedures

We present a fully adaptive numerical scheme for evolutionary PDEs in Cartesian geometry based on a second-order finite volume discretization. A multiresolution strategy allows local grid refinement while controlling the approximation error in space. For time discretization we use an explicit Runge–Kutta scheme of second-order with a scale-dependent time step. On the finest scale the size of the time step is imposed by the stability condition of the explicit scheme. On larger scales, the time step can be increased without violating the stability requirement of the explicit scheme. The implementation uses a dynamic tree data structure. Numerical validations for test problems in one space dimension demonstrate the efficiency and accuracy of the local time-stepping scheme with respect to both multiresolution scheme with global time stepping and finite volume scheme on a regular grid. Fully adaptive three-dimensional computations for reaction–diffusion equations illustrate the additional speed-up of the local time stepping for a thermo-diffusive flame instability.

Multi-resolution image registration using multi-class Hausdorff fraction

Recently, a new image registration method, based on the Hausdorff fraction and a multi-resolution search of the transformation space, has been developed in the literature. This method has been applied to problems involving translations, translation and scale, and affine transformations. In this paper, we adapt the above method to the set of similarity transformations. We also introduce a new variant of the Hausdorff fraction similarity measure based on a multi-class approach, which we call the multi-class Hausdorff fraction (MCHF). The multi-class approach is more efficient because it matches feature points only if they are from the same class. To validate our approach, we segment edge maps into two classes which are the class of straight lines and the class of curves, and we apply the new multi-class approach to two image registration examples, using synthetic and real images, respectively. Experimental results show that the multi-class approach speeds up the multi-resolution search algorithm.

Multiresolution image registration for two-dimensional gel electrophoresis

Multiresolution image registration for two-dimensional gel electrophoresis

Techniques for multiresolution image registration in the presence of occlusions

This paper describes and compares image registration techniques for situations in which one or both candidate images are partially occluded. Both fractional masks (introduced) and binary masks improve registration accuracy. Efficient mask-based algorithms operate at low resolution on low-pass subbands of wavelets and correlate images in the frequency domain.