Alignment Of Multimodal Images Research Articles

A multi-level image alignment method for aerial image and road-based geo-parcel data

Image alignment between aerial image and geo-parcel data is a meaningful and challenging task in remote sensing field. In this article, a deep learning framework based on multi-level progressive architecture focusing on aerial image and road based geo-parcel data alignment is proposed. Firstly, an image segmentation with U-Net model as a preprocessing work is applied to obtain the road binary image of aerial image, which benefits following image alignment by turning multi-modal image alignment into mono-modal image alignment. Afterwards, multi-scale deep features are extracted to take part in the proposed image alignment network. The proposed image alignment network consists of a global multiple homographies prediction module and a local flow map estimation module, forming a coarse-to-fine and global-to-local multi-level paradigm. Finally, synthetic image datasets are generated to test and verify the performance of proposed method. The experiments on the synthetic aerial image and road based geo-parcel data including Mnih dataset and Deep Globe road dataset demonstrate that the proposed method can align image pairs effectively, and the proposed method achieves a certain increase in matching performance by comparing with recent existing alignment methods qualitatively and quantitatively.

Semi-Automatic Multiparametric MR Imaging Classification Using Novel Image Input Sequences and 3D Convolutional Neural Networks

The role of multi-parametric magnetic resonance imaging (mp-MRI) is becoming increasingly important in the diagnosis of the clinical severity of prostate cancer (PCa). However, mp-MRI images usually contain several unaligned 3D sequences, such as DWI image sequences and T2-weighted image sequences, and there are many images among the entirety of 3D sequence images that do not contain cancerous tissue, which affects the accuracy of large-scale prostate cancer detection. Therefore, there is a great need for a method that uses accurate computer-aided detection of mp-MRI images and minimizes the influence of useless features. Our proposed PCa detection method is divided into three stages: (i) multimodal image alignment, (ii) automatic cropping of the sequence images to the entire prostate region, and, finally, (iii) combining multiple modal images of each patient into novel 3D sequences and using 3D convolutional neural networks to learn the newly composed 3D sequences with different modal alignments. We arrange the different modal methods to make the model fully learn the cancerous tissue features; then, we predict the clinical severity of PCa and generate a 3D cancer response map for the 3D sequence images from the last convolution layer of the network. The prediction results and 3D response map help to understand the features that the model focuses on during the process of 3D-CNN feature learning. We applied our method to Toho hospital prostate cancer patient data; the AUC (=0.85) results were significantly higher than those of other methods.

Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI that typically works well only for small displacements; this points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while asymptotically superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g. rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, of aerial images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at \url{github.com/MIDA-group/globalign}.

A two-step registration-classification approach to automated segmentation of multimodal images for high-throughput greenhouse plant phenotyping

BackgroundAutomated segmentation of large amount of image data is one of the major bottlenecks in high-throughput plant phenotyping. Dynamic optical appearance of developing plants, inhomogeneous scene illumination, shadows and reflections in plant and background regions complicate automated segmentation of unimodal plant images. To overcome the problem of ambiguous color information in unimodal data, images of different modalities can be combined to a virtual multispectral cube. However, due to motion artefacts caused by the relocation of plants between photochambers the alignment of multimodal images is often compromised by blurring artifacts.ResultsHere, we present an approach to automated segmentation of greenhouse plant images which is based on co-registration of fluorescence (FLU) and of visible light (VIS) camera images followed by subsequent separation of plant and marginal background regions using different species- and camera view-tailored classification models. Our experimental results including a direct comparison with manually segmented ground truth data show that images of different plant types acquired at different developmental stages from different camera views can be automatically segmented with the average accuracy of 93% (SD=5%) using our two-step registration-classification approach.ConclusionAutomated segmentation of arbitrary greenhouse images exhibiting highly variable optical plant and background appearance represents a challenging task to data classification techniques that rely on detection of invariances. To overcome the limitation of unimodal image analysis, a two-step registration-classification approach to combined analysis of fluorescent and visible light images was developed. Our experimental results show that this algorithmic approach enables accurate segmentation of different FLU/VIS plant images suitable for application in fully automated high-throughput manner.

Multi-Modal Medical Image Registration with Full or Partial Data: A Manifold Learning Approach.

Multi-modal image registration is the primary step in integrating information stored in two or more images, which are captured using multiple imaging modalities. In addition to intensity variations and structural differences between images, they may have partial or full overlap, which adds an extra hurdle to the success of registration process. In this contribution, we propose a multi-modal to mono-modal transformation method that facilitates direct application of well-founded mono-modal registration methods in order to obtain accurate alignment of multi-modal images in both cases, with complete (full) and incomplete (partial) overlap. The proposed transformation facilitates recovering strong scales, rotations, and translations. We explain the method thoroughly and discuss the choice of parameters. For evaluation purposes, the effectiveness of the proposed method is examined and compared with widely used information theory-based techniques using simulated and clinical human brain images with full data. Using RIRE dataset, mean absolute error of 1.37, 1.00, and 1.41 mm are obtained for registering CT images with PD-, T1-, and T2-MRIs, respectively. In the end, we empirically investigate the efficacy of the proposed transformation in registering multi-modal partially overlapped images.

Multimodal Image Alignment via Linear Mapping between Feature Modalities.

We propose a novel landmark matching based method for aligning multimodal images, which is accomplished uniquely by resolving a linear mapping between different feature modalities. This linear mapping results in a new measurement on similarity of images captured from different modalities. In addition, our method simultaneously solves this linear mapping and the landmark correspondences by minimizing a convex quadratic function. Our method can estimate complex image relationship between different modalities and nonlinear nonrigid spatial transformations even in the presence of heavy noise, as shown in our experiments carried out by using a variety of image modalities.

Groupwise Registration of Multimodal Images by an Efficient Joint Entropy Minimization Scheme

Groupwise registration is concerned with bringing a group of images into the best spatial alignment. If images in the group are from different modalities, then the intensity correspondences across the images can be modeled by the joint density function (JDF) of the cooccurring image intensities. We propose a so-called treecode registration method for groupwise alignment of multimodal images that uses a hierarchical intensity-space subdivision scheme through which an efficient yet sufficiently accurate estimation of the (high-dimensional) JDF based on the Parzen kernel method is computed. To simultaneously align a group of images, a gradient-based joint entropy minimization was employed that also uses the same hierarchical intensity-space subdivision scheme. If the Hilbert kernel is used for the JDF estimation, then the treecode method requires no data-dependent bandwidth selection and is thus fully automatic. The treecode method was compared with the ensemble clustering (EC) method on four different publicly available multimodal image data sets and on a synthetic monomodal image data set. The obtained results indicate that the treecode method has similar and, for two data sets, even superior performances compared to the EC method in terms of registration error and success rate. The obtained good registration performances can be mostly attributed to the sufficiently accurate estimation of the JDF, which is computed through the hierarchical intensity-space subdivision scheme, that captures all the important features needed to detect the correct intensity correspondences across a multimodal group of images undergoing registration.

Phase mutual information as a similarity measure for registration

This article describes a new method for non-rigid alignment of multimodal images. Multimodal image registration is most often accomplished by modelling, in some sense, an intensity mapping between the images. Here, the alternative strategy of modelling a relationship between local image phase is introduced. This method is intrinsically image feature based, and searches for relationships between feature appearances, rather than tissue class intensity. This enables registration of modalities for which image intensity is not a simple function of tissue class, for example ultrasound. It is also demonstrated that this method performs comparably to an intensity method even when the images are related by a simple intensity transform, but that the phase method is significantly more robust to image artifacts which corrupt the ideal intensity mapping.