Multimodal Image Pairs Research Articles

Image fusion with structural saliency measure and content adaptive consistency verification

Image fusion obtains a desired image by integrating the useful information of multiple input images. Most traditional fusion strategy is usually guided by image local contrast or variance, which cannot well represent visual discernable features of source images. And the undesirable seam effects or artifacts produced due to the inconsistency between fusion weight map and image content may severely degrade the visual quality of the fused images. An efficient image fusion method with structural saliency measure and content adaptive consistency verification was proposed. The fusion is implemented under the nonsubsampled contourlet transform (NSCT)-based image fusion framework. The low-frequency NSCT decomposition coefficients are fused with the weight map constructed by considering both structural saliency and visual uniqueness features and refined by spatial consistency with guide filter. The high-frequency NSCT decomposition coefficients are fused with structural saliency. The performances of the proposed method have been verified on several pairs of multifocus images, infrared-visible images, and multimodal medical images. Experimental results clearly demonstrate the superiority of the proposed algorithm compared with several existing state-of-the-art algorithms in terms of both visual and quantitative comparison.

Multimodal 3D medical image registration guided by shape encoder-decoder networks.

Nonlinear multimodal image registration, for example, the fusion of computed tomography (CT) and magnetic resonance imaging (MRI), fundamentally depends on a definition of image similarity. Previous methods that derived modality-invariant representations focused on either global statistical grayscale relations or local structural similarity, both of which are prone to local optima. In contrast to most learning-based methods that rely on strong supervision of aligned multimodal image pairs, we aim to overcome this limitation for further practical use cases. We propose a new concept that exploits anatomical shape information and requires only segmentation labels for both modalities individually. First, a shape-constrained encoder-decoder segmentation network without skip connections is jointly trained on labeled CT and MRI inputs. Second, an iterative energy-based minimization scheme is introduced that relies on the capability of the network to generate intermediate nonlinear shape representations. This further eases the multimodal alignment in the case of large deformations. Our novel approach robustly and accurately aligns 3D scans from the multimodal whole-heart segmentation dataset, outperforming classical unsupervised frameworks. Since both parts of our method rely on (stochastic) gradient optimization, it can be easily integrated in deep learning frameworks and executed on GPUs. We present an integrated approach for weakly supervised multimodal image registration. Achieving promising results due to the exploration of intermediate shape features as registration guidance encourages further research in this direction.

Performance of Change Detection Algorithms Using Heterogeneous Images and Extended Multi-attribute Profiles (EMAPs)

We present detection performance of ten change detection algorithms with and without the use of Extended Multi-Attribute Profiles (EMAPs). Heterogeneous image pairs (also known as multimodal image pairs), which are acquired by different imagers, are used as the pre-event and post-event images in the investigations. The objective of this work is to examine if the use of EMAP, which generates synthetic bands, can improve the detection performances of these change detection algorithms. Extensive experiments using five heterogeneous image pairs and ten change detection algorithms were carried out. It was observed that in 34 out of 50 cases, change detection performance was improved with EMAP. A consistent detection performance boost in all five datasets was observed with EMAP for Homogeneous Pixel Transformation (HPT), Chronochrome (CC), and Covariance Equalization (CE) change detection algorithms.

Multimodal image seamless fusion

An iterative joint bilateral filter is used to obtain a natural weight map. Images from different modalities are merged by a weighted-sum rule in the spatial domain. Saliency maps are determined by the gradient of the pairwise raw images. Comparing the pairwise values of saliency maps, a coarse weight map is attained to determine which pixel is preferred. Since such a coarse weight map obtained by pairwise comparison is not a natural weight map subjectively, i.e., it is inconsistent with human visual system, the weight map is modified by using an iterative joint bilateral filter. With the iterative joint bilateral filter, the weight map becomes natural. We use the refined weight map to obtain the fused image and we seamlessly merge images from different modalities effectively. Experiments were conducted on several pairs of multimodal images to verify the effectiveness and superiority of the proposed image fusion algorithm compared to the state-of-the-art methods. (C) 2019 SPIE and IS&T

A novel extended phase correlation algorithm based on Log-Gabor filtering for multimodal remote sensing image registration

ABSTRACT Automatic registration of multimodal remote sensing images, which is a critical prerequisite in a range of applications (e.g. image fusion, image mosaic, and image analysis), continues to be a fundamental and challenging problem. In this paper, we propose a novel extended phase correlation algorithm based on Log-Gabor filtering (LGEPC) for the registration of images with nonlinear radiometric differences and geometric differences (e.g. rotation, scale, and translation). Our algorithm focuses on two problems that the traditional extended phase correlation algorithms cannot well handle: 1) significant nonlinear radiometric differences and 2) large-scale differences between image pairs. After an over-complete multi-scale atlas space of the original image is built based on the filtered magnitudes obtained by using Log-Gabor filters with different central frequencies, the phase correlation of the single scale images is extended by LGEPC to atlases phase correlation, which is conducive to solving the problem of large scale and rotation differences between the image pairs. Subsequently, LGEPC eliminates the interface of the significant nonlinear radiometric differences by superimposing multi-scale geometric structural spectra and carrying out the phase correlation module, so that the translation can be well determined. Our experiments on synthetic images demonstrated the rationality and effectiveness of LGEPC, and the experiments on a variety of multimodal images confirmed that LGEPC can ideally achieve pixel-wise registration accuracy for multimodal image pairs that conform to the similarity transformation model.

Multimodal medical image fusion based on Yager’s intuitionistic fuzzy sets

The objective of image fusion for medical images is to combine multiple images obtained from various sources into a single image suitable for better diagnosis. Most of the state-of-the-art image fusing technique is based on nonfuzzy sets, and the fused image so obtained lags with complementary information. Intuitionistic fuzzy sets (IFS) are determined to be more suitable for civilian, and medical image processing as more uncertainties are considered compared with fuzzy set theory. In this paper, an algorithm for effectively fusing multimodal medical images is presented. In the proposed method, images are initially converted into Yager’s intuitionistic fuzzy complement images (YIFCIs), and a new objective function called intuitionistic fuzzy entropy (IFE) is employed to obtain the optimum value of the parameter in membership and non-membership functions. Next, the YIFCIs are compared using contrast visibility (CV) to construct a decision map (DM). DM is refined with consistency verification to create a fused image. Simulations on several pairs of multimodal medical images are performed and compared with the existing fusion methods, such as simple average, discrete cosine transform (DCT), redundant wavelet transform (RWT), intuitionistic fuzzy set, fuzzy transform and interval-valued intuitionistic fuzzy set (IVIFS). The superiority of the proposed method is presented and is justified. Fused image quality is also verified with various quality metrics, such as spatial frequency (SF), average gradient (AG), fusion symmetry (FS), edge information preservation (QAB/F), entropy (E) and computation time (CoT).

A Block-Based Multifeature Extraction Scheme for SAR Image Registration

In this letter, a block-based multifeature extraction scheme is proposed to register the synthetic aperture radar (SAR) images. With appropriate modifications, the scale-invariant feature transform (SIFT) and the SAR-SIFT operators are used to extract two types of features including texture points and corner points from the SAR images. The input images are divided into a certain number of blocks and the two types of features are extracted from each of the blocks for the uniform distribution of the features. A novel scheme is presented to obtain these features in the same proportion from the input images. The proposed method has the advantages of proper controllability of the number of extracted features and the uniform distribution of the features. A correct match identification by local searching algorithm is proposed to significantly increase the number of correct matches between the SAR images. Experiments on three pairs of multimodal and multitemporal SAR images demonstrate the effectiveness of the proposed method.

Weakly-supervised convolutional neural networks for multimodal image registration.

One of the fundamental challenges in supervised learning for multimodal image registration is the lack of ground-truth for voxel-level spatial correspondence. This work describes a method to infer voxel-level transformation from higher-level correspondence information contained in anatomical labels. We argue that such labels are more reliable and practical to obtain for reference sets of image pairs than voxel-level correspondence. Typical anatomical labels of interest may include solid organs, vessels, ducts, structure boundaries and other subject-specific ad hoc landmarks. The proposed end-to-end convolutional neural network approach aims to predict displacement fields to align multiple labelled corresponding structures for individual image pairs during the training, while only unlabelled image pairs are used as the network input for inference. We highlight the versatility of the proposed strategy, for training, utilising diverse types of anatomical labels, which need not to be identifiable over all training image pairs. At inference, the resulting 3D deformable image registration algorithm runs in real-time and is fully-automated without requiring any anatomical labels or initialisation. Several network architecture variants are compared for registering T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients. A median target registration error of 3.6 mm on landmark centroids and a median Dice of 0.87 on prostate glands are achieved from cross-validation experiments, in which 108 pairs of multimodal images from 76 patients were tested with high-quality anatomical labels.

LPPCO: A Novel Multimodal Medical Image Registration Using New Feature Descriptor Based on the Local Phase and Phase Congruency of Different Orientations

At present, many feature descriptors-based registration methods have been proven to be robust in case of complex intensity distortions. However, most of these feature descriptors are only related to intensity information in a patch of neighboring pixels and ignoring the neighbor orientation information, which make the registration performance for medical images with low-resolution appear to be weak robustness and low accuracy. To improve the robustness and accuracy, a novel feature descriptor, named Local-Phase mean and Phase-Congruency values of different Orientations (LPPCO), is developed using filter-bank of Log-Gabor filters at different orientations and frequencies. Next, a similarity measure named LPPCOncc is developed using the normalized cross correlation (NCC) of the LPPCO descriptors, followed by a fast template matching techniques for detecting correspondences between the different images. Additionally, a more sensitivity of phase deviation function is presented for the calculation of phase congruency. The main steps of constructing the similarity measure LPPCOncc include: firstly, we extract the local phase mean and phase congruency values for each pixel in each orientation; secondly, local phase mean orientation histograms and phase congruency values orientation histograms over all the pixels are computed respectively, where the maximum response values are chosen to vote for the corresponding bin; thirdly, combining the two resulting histograms obtains the feature descriptor LPPCO; finally, the LPPCO descriptor is compared across images using NCC. Experimental results show that LPPCOncc is robust to complex intensity distortions between multimodal medical images and outperforms other similarity metrics such as NCC and DLSC. Furthermore, LPPCOncc-based registration algorithm preformed on various types of multimodal medical image pairs shows that it outperforms the NMI-based and DLSC-based registration methods in the registration robustness and accuracy.

Adopting Quaternion Wavelet Transform to Fuse Multi-Modal Medical Images

Medical image fusion plays an important role in clinical applications such as image-guided surgery, image-guided radiotherapy, noninvasive diagnosis, and treatment planning. In this paper, we propose a novel multi-modal medical image fusion method based on simplified pulse-coupled neural network and quaternion wavelet transform. The proposed fusion algorithm is capable of combining not only pairs of computed tomography (CT) and magnetic resonance (MR) images, but also pairs of CT and proton-density-weighted MR images, and multi-spectral MR images such as T1 and T2. Experiments on six pairs of multi-modal medical images are conducted to compare the proposed scheme with four existing methods. The performances of various methods are investigated using mutual information metrics and comprehensive fusion performance characterization (total fusion performance, fusion loss, and modified fusion artifacts criteria). The experimental results show that the proposed algorithm not only extracts more important visual information from source images, but also effectively avoids introducing artificial information into fused medical images. It significantly outperforms existing medical image fusion methods in terms of subjective performance and objective evaluation metrics.

Multimodal Remote Sensing Image Registration With Accuracy Estimation at Local and Global Scales

This paper focuses on potential accuracy of remote sensing images registration. We investigate how this accuracy can be estimated without ground truth available and used to improve registration quality of mono- and multi-modal pair of images. At the local scale of image fragments, the Cramer-Rao lower bound (CRLB) on registration error is estimated for each local correspondence between coarsely registered pair of images. This CRLB is defined by local image texture and noise properties. Opposite to the standard approach, where registration accuracy is only evaluated at the output of the registration process, such valuable information is used by us as an additional input knowledge. It greatly helps detecting and discarding outliers and refining the estimation of geometrical transformation model parameters. Based on these ideas, a new area-based registration method called RAE (Registration with Accuracy Estimation) is proposed. In addition to its ability to automatically register very complex multimodal image pairs with high accuracy, the RAE method provides registration accuracy at the global scale as covariance matrix of estimation error of geometrical transformation model parameters or as point-wise registration Standard Deviation. This accuracy does not depend on any ground truth availability and characterizes each pair of registered images individually. Thus, the RAE method can identify image areas for which a predefined registration accuracy is guaranteed. The RAE method is proved successful with reaching subpixel accuracy while registering eight complex mono/multimodal and multitemporal image pairs including optical to optical, optical to radar, optical to Digital Elevation Model (DEM) images and DEM to radar cases. Other methods employed in comparisons fail to provide in a stable manner accurate results on the same test cases.

Performance evaluation of similarity measures for dense multimodal stereovision

Multimodal imaging systems have recently been drawing attention in fields such as medical imaging, remote sensing, and video surveillance systems. In such systems, estimating depth has become possible due to the promising progress of multimodal matching techniques. We perform a systematic performance evaluation of similarity measures frequently used in the literature for dense multimodal stereovision. The evaluated measures include mutual information (MI), sum of squared distances, normalized cross-correlation, census transform, local self-similarity (LSS) as well as descriptors adopted to multimodal settings, like scale invariant feature transform (SIFT), speeded-up robust features (SURF), histogram of oriented gradients (HOG), binary robust independent elementary features, and fast retina keypoint (FREAK). We evaluate the measures over datasets we generated, compiled, and provided as a benchmark and compare the performances using the Winner Takes All method. The datasets are (1) synthetically modified four popular pairs from the Middlebury Stereo Dataset (namely, Tsukuba, Venus, Cones, and Teddy) and (2) our own multimodal image pairs acquired using the infrared and the electro-optical cameras of a Kinect device. The results show that MI and HOG provide promising results for multimodal imagery, and FREAK, SURF, SIFT, and LSS can be considered as alternatives depending on the multimodality level and the computational complexity requirements of the intended application.

Multimodal medical image fusion using PCNN optimized by the QPSO algorithm

This paper proposed a method to fuse multimodal medical images using the adaptive pulse-coupled neural networks (PCNN), which was optimized by the quantum-behaved particle swarm optimization (QPSO) algorithm. In this fusion model, two source images, A and B, were first processed by the QPSO-PCNN model, respectively. Through the QPSO algorithm, the PCNN model could find the optimal parameters for the source images, A and B. To improve the efficiency and quality of QPSO, three evaluation criteria, image entropy (EN), average gradient (AG) and spatial frequency (SF) were selected as the hybrid fitness function. Then, the output of the fusion model was obtained by the judgment factor according to the firing maps of two source images, which maybe was the pixel value of the image A, or that of the image B, or the tradeoff value of them. Based on the output of the fusion model, the fused image was gained. Finally, we used five pairs of multimodal medical images as experimental data to test and verify the proposed method. Furthermore, the mutual information (MI), structural similarity (SSIM), image entropy (EN), etc. were used to judge the performances of different methods. The experimental results illustrated that the proposed method exhibited better performances.

Multispectral Stereo Odometry

In this paper, we investigate the problem of visual odometry for ground vehicles based on the simultaneous utilization of multispectral cameras. It encompasses a stereo rig composed of an optical (visible) and thermal sensors. The novelty resides in the localization of the cameras as a stereo setup rather than two monocular cameras of different spectrums. To the best of our knowledge, this is the first time such task is attempted. Log-Gabor wavelets at different orientations and scales are used to extract interest points from both images. These are then described using a combination of frequency and spatial information within the local neighborhood. Matches between the pairs of multimodal images are computed using the cosine similarity function based on the descriptors. Pyramidal Lucas–Kanade tracker is also introduced to tackle temporal feature matching within challenging sequences of the data sets. The vehicle egomotion is computed from the triangulated 3-D points corresponding to the matched features. A windowed version of bundle adjustment incorporating Gauss–Newton optimization is utilized for motion estimation. An outlier removal scheme is also included within the framework to deal with outliers. Multispectral data sets were generated and used as test bed. They correspond to real outdoor scenarios captured using our multimodal setup. Finally, detailed results validating the proposed strategy are illustrated.

Multimodal image registration with joint structure tensor and local entropy.

Nonrigid registration of multimodal medical images remains a challenge in image-guided interventions. A common approach is to use mutual information (MI), which is robust to the intensity variations across modalities. However, primarily based on intensity distribution, MI does not take into account of underlying spatial and structural information of the images, which might lead to local optimization. To address such a challenge, this paper proposes a two-stage multimodal nonrigid registration scheme with joint structural information and local entropy. In our two-stage multimodal nonrigid registration scheme, both the reference image and floating image are firstly converted to a common space. A unified representation in the common space for the images is constructed by fusing the structure tensor (ST) trace with the local entropy (LE). Through the representation that reflects its geometry uniformly across modalities, the complicated deformation field is estimated using L(1) or L(2) distance. We compared our approach to four other methods: (1) the method using LE, (2) the method using ST, (3) the method using spatially weighted LE and (4) the conventional MI-based method. Quantitative evaluations on 80 multimodal image pairs of different organs including 50 pairs of MR images with artificial deformations, 20 pairs of medical brain MR images and 10 pairs of breast images showed that our proposed method outperformed the comparison methods. Student's t test demonstrated that our method achieved statistically significant improvement on registration accuracy. The two-stage registration with joint ST and LE outperformed the conventional MI-based method for multimodal images. Both the ST and the LE contributed to the improved registration accuracy.

Robust point matching method for multimodal retinal image registration

In this paper, motivated by the problem of multimodal retinal image registration, we introduce and improve the robust registration framework based on partial intensity invariant feature descriptor (PIIFD), then present a registration framework based on speed up robust feature (SURF) detector, PIIFD and robust point matching, called SURF–PIIFD–RPM. Existing retinal image registration algorithms are unadaptable to any case, such as complex multimodal images, poor quality, and nonvascular images. Harris-PIIFD framework usually fails in correctly aligning color retinal images with other modalities when faced large content changes. Our proposed registration framework mainly solves the problem robustly. Firstly, SURF detector is useful to extract more repeatable and scale-invariant interest points than Harris. Secondly, a single Gaussian robust point matching model is based on the kernel method of reproducing kernel Hilbert space to estimate mapping function in the presence of outliers. Most importantly, our improved registration framework performs well even when confronted a large number of outliers in the initial correspondence set. Finally, multiple experiments on our 142 multimodal retinal image pairs demonstrate that our SURF–PIIFD–RPM outperforms existing algorithms, and it is quite robust to outliers.

Multimodal Segmentation of Optic Disc and Cup From SD-OCT and Color Fundus Photographs Using a Machine-Learning Graph-Based Approach.

In this work, a multimodal approach is proposed to use the complementary information from fundus photographs and spectral domain optical coherence tomography (SD-OCT) volumes in order to segment the optic disc and cup boundaries. The problem is formulated as an optimization problem where the optimal solution is obtained using a machine-learning theoretical graph-based method. In particular, first the fundus photograph is registered to the 2D projection of the SD-OCT volume. Three in-region cost functions are designed using a random forest classifier corresponding to three regions of cup, rim, and background. Next, the volumes are resampled to create radial scans in which the Bruch's Membrane Opening (BMO) endpoints are easier to detect. Similar to in-region cost function design, the disc-boundary cost function is designed using a random forest classifier for which the features are created by applying the Haar Stationary Wavelet Transform (SWT) to the radial projection image. A multisurface graph-based approach utilizes the in-region and disc-boundary cost images to segment the boundaries of optic disc and cup under feasibility constraints. The approach is evaluated on 25 multimodal image pairs from 25 subjects in a leave-one-out fashion (by subject). The performances of the graph-theoretic approach using three sets of cost functions are compared: 1) using unimodal (OCT only) in-region costs, 2) using multimodal in-region costs, and 3) using multimodal in-region and disc-boundary costs. Results show that the multimodal approaches outperform the unimodal approach in segmenting the optic disc and cup.

Biorthogonal Wavelet Transform Using Bilateral Filter and Adaptive Histogram Equalization

Image fusion is a process of combining data from multiple sources to achieve refined or improved information for making decisions. It has many applications. When we use images with a similar acquisition time, the expected result is to obtain a fused image that retains the spatial resolution from the panchromatic image and color content from the multi-spectral image. In recent time different methods have been developed. These methods are both in spatial domain and in wavelet domain. Out of these two the wavelet domain based methods are more suitable as they are capable to handle the spatial distortion produced by the spatial domain. In this paper the proposed method is compared with principle component analysis, discrete cosine transform and also with biorthogonal wavelet transform in which bilateral filter and adaptive histogram is not present. This comparison is on the bases of different parameters. Biorthogonal wavelet transform is capable to preserve edge information and hence reducing the distortions in the fused image. It has two important properties wavelet symmetry and linear phase which are not present in spatial domain. The performance of the proposed method has been extensively tested on several pairs of multi-focus and multimodal images. Experimental results show that the proposed method improves fusion quality by reducing loss of significant information available in individual images.

An efficient approach for robust multimodal retinal image registration based on UR-SIFT features and PIIFD descriptors

Existing algorithms based on scale invariant feature transform (SIFT) and Harris corners such as edge-driven dual-bootstrap iterative closest point and Harris-partial intensity invariant feature descriptor (PIIFD) respectivley have been shown to be robust in registering multimodal retinal images. However, they fail to register color retinal images with other modalities in the presence of large content or scale changes. Moreover, the approaches need preprocessing operations such as image resizing to do well. This restricts the application of image registration for further analysis such as change detection and image fusion. Motivated by the need for efficient registration of multimodal retinal image pairs, this paper introduces a novel integrated approach which exploits features of uniform robust scale invariant feature transform (UR-SIFT) and PIIFD. The approach is robust against low content contrast of color images and large content, appearance, and scale changes between color and other retinal image modalities like the fluorescein angiography. Due to low efficiency of standard SIFT detector for multimodal images, the UR-SIFT algorithm extracts high stable and distinctive features in the full distribution of location and scale in images. Then, feature points are adequate and repeatable. Moreover, the PIIFD descriptor is symmetric to contrast, which makes it suitable for robust multimodal image registration. After the UR-SIFT feature extraction and the PIIFD descriptor generation in images, an initial cross-matching process is performed and followed by a mismatch elimination algorithm. Our dataset consists of 120 pairs of multimodal retinal images. Experiment results show the outperformance of the UR-SIFT-PIIFD over the Harris-PIIFD and similar algorithms in terms of efficiency and positional accuracy.

Accurate M-hausdorff distance similarity combining distance orientation for matching multi-modal sensor images

Although Hausdorff distance (HD) has been widely used in an object identification between same modality images, the object identification between different modality images are challenging because of the poor edge correspondence coming from heterogeneous image characteristics. This paper proposes a robust Hausdorff distance similarity (accurate M-HD: AMHD) between multi-modal sensor data. To improve robustness against the outliers when comparing the pairs of multi-modal images, the AMHD utilizes the orientation information of each point in addition to the distance transform (DT) map as a similarity criterion. In the AMHD scheme, the DT map is generated by applying dead-reckoning signed DT, and the distance orientation (DO) map is constructed by employing the Kirsch compass kernel to the DT map, respectively. Using the additional information on the DO, the proposed similarity can precisely examine the outliers including non-correspondent edges and noises, and discard false correspondent distances efficiently. The computer simulations show that the proposed AMHD yields superior performance at aligning multi-modal sensor data (visible-thermal IR face images) over those achieved by the conventional robust schemes in terms of the position error between the ground truth and the computed position.