Multi-sensor Image Fusion Research Articles

Novel approach on fusion of multisensor images based on neutrosophic domain in consideration of regional relevance

The presented work describes the fusion of multisensor (Multispectral and Panchromatic) images in remote sensing domain. The fusion of multisensor remote sensing images which utilizes neutrosophic set is introduced. The proposed approach preserves the intrinsic structural information as well as bringing the complimentary information to the supplementary information in a right proportion. Novel regional based fusion rule is introduced in the proposed work. The indeterminacy in the input data is handled to provide a comprehensive fusion result, where the compared methods fails. The proposed approach is tested against the Multi-resolution Analysis(MRA) and Multi-Geometric Analysis based techniques quantitatively and qualitatively based on the metrics. The proposed approach provides promoting results against the compared methods in terms of quality and quantity is observed through experimental results.

Multisensor image fusion approach utilizing hybrid pre-enhancement and double nonsubsampled contourlet transform

A multisensor image fusion approach established on the hybrid-domain image enhancement and double nonsubsampled contourlet transform (NSCT) is proposed. The hybrid-domain pre-enhancement algorithm can promote the contrast of the visible color image. Different fusion rules are, respectively, selected and applied to obtain fusion results. The double NSCT framework is introduced to obtain better fusion performance than the general single NSCT framework. Experimental outcomes in fused images and performance results demonstrate that the presented approach is apparently more advantageous.

A Novel Multi-Focus Image Fusion Method Based on Stationary Wavelet Transform and Local Features of Fuzzy Sets

The key issue of multi-sensor image fusion is how to accurately extract and fuse the high-quality pixels or coefficients of source images. Nevertheless, the so-called high-quality is an uncertain or fuzzy definition, which is very suitable for fuzzy theory to address this problem. By the integration of stationary wavelet transform (SWT) and fuzzy sets, this paper proposes a new multi-focus image fusion scheme, which can merge the important features of different source images into a fused image. First, the source images are decomposed by SWT to get a set of sub-images with different detailed features. Second, the Gaussian membership function (GMF) is utilized to get the fuzzy sets of sub-images data. Third, the local spatial frequency (LSF) is employed to extract the local features of the sub-images by the generated fuzzy sets. At last, the fusion rule is designed based on consistency verification to fuse the sub-images according to the LSF of fuzzy sets, and then inverse SWT (ISWT) is implemented to reconstruct the fused image. The experimental and contrastive results of different image sets show that the proposed method is an effective multi-focus image fusion scheme which can achieve better fusion effect than other methods.

Image Fusion for Mars Data Using Mix of Robust PCA

Multi-sensor image fusion is the process of combining relevant information from high spatial resolution image and high spectral resolution image. This paper proposes a pansharpening method for the fusion of Mars images obtained by the Mars Reconnaissance Orbiter satellite (PAN) and Mars Odyssey satellite (THEMIS MS). The method is based on some mix of Intensity, Hue, Saturation (IHS) and robust principal component analysis (RPCA) combined with the discrete wavelet transformation (DWT). Meanwhile, the results obtained with that of the other fusion techniques are compared, and a relatively objective comprehensive evaluation for fused image is used. Experiments show that the proposed algorithm has a significant suppression of artificial textures and less spectral distortion, and its running time is acceptable.

Multi-Sensor Image Fusion with Adaptive Guided Image Filtering in Nonsubsampled Contourlet Transform Domain

Multi-Sensor Image Fusion with Adaptive Guided Image Filtering in Nonsubsampled Contourlet Transform Domain

Multisensor image fusion guidelines in remote sensing

Remote sensing delivers multimodal and -temporal data from the Earth's surface. In order to cope with these multidimensional data sources and to make the most of them, image fusion is a valuable tool. It has developed over the past few decades into a usable image processing technique for extracting information of higher quality and reliability. As more sensors and advanced image fusion techniques have become available, researchers have conducted a vast amount of successful studies using image fusion. However, the definition of an appropriate workflow prior to processing the imagery requires knowledge in all related fields - i.e. remote sensing, image fusion and the desired image exploitation processing. From the findings of this research it can be seen that the choice of the appropriate technique, as well as the fine-tuning of the individual parameters of this technique, is crucial. There is still a lack of strategic guidelines due to the complexity and variability of data selection, processing techniques and applications. This paper gives an overview on the state-of-the-art in remote sensing image fusion including sensors and applications. Putting research results in image fusion from the past 15 years into a context provides a new view on the subject and helps other researchers to build their innovation on these findings. Recommendations of experts help to understand further needs to achieve feasible strategies in remote sensing image fusion.

Pixel-level multisensor image fusion based on matrix completion and robust principal component analysis

Acquired digital images are often corrupted by a lack of camera focus, faulty illumination, or missing data. An algorithm is presented for fusion of multiple corrupted images of a scene using the lifting wavelet trans- form. The method employs adaptive fusion arithmetic based on matrix completion and self-adaptive regional variance estimation. Characteristics of the wavelet coefficients are used to adaptively select fusion rules. Robust principal component analysis is applied to low-frequency image components, and regional variance estimation is applied to high-frequency components. Experiments reveal that the method is effective for multifocus, visible-light, and infrared image fusion. Compared with traditional algorithms, the new algorithm not only increases the amount of preserved information and clarity but also improves robustness. © The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI. (DOI: 10.1117/1.JEI.25.1.013007)

An Adaptive Pansharpening Method by Using Weighted Least Squares Filter

Multisensor image fusion or pansharpening aims to sharpen a multispectral (MS) image by integrating the detail map derived from a panchromatic (Pan) image. The intensity–hue–saturation (IHS)-based methods are well adopted in pansharpening applications. However, the pansharpened MS images by IHS-based methods usually suffer from serious spectral distortions and local artifacts due to the mismatch between the estimated detail map and its ground truth. To overcome these defects, we propose a weighted least squares (WLS)-filter-based method in this letter. Different from existing IHS-based methods, the proposed method eliminates the influence of the low-frequency components of the Pan and MS images with the WLS filter. Moreover, the derived detail map is further refined based on the spectral signatures for different bands of the MS image. We test the proposed method on various satellites data; the experimental results demonstrate that the proposed method performs well in both spectral and spatial qualities.

A Multi Sensor Image Fusion Algorithm Based on Multi Resolution Analysis

A Multi Sensor Image Fusion Algorithm Based on Multi Resolution Analysis

Registration of dynamic multiview 2D ultrasound and late gadolinium enhanced images of the heart: Application to hypertrophic cardiomyopathy characterization.

Describing and analyzing heart multiphysics requires the acquisition and fusion of multisensor cardiac images. Multisensor image fusion enables a combined analysis of these heterogeneous modalities. We propose to register intra-patient multiview 2D+t ultrasound (US) images with multiview late gadolinium-enhanced (LGE) images acquired during cardiac magnetic resonance imaging (MRI), in order to fuse mechanical and tissue state information. The proposed procedure registers both US and LGE to cine MRI. The correction of slice misalignment and the rigid registration of multiview LGE and cine MRI are studied, to select the most appropriate similarity measure. It showed that mutual information performs the best for LGE slice misalignment correction and for LGE and cine registration. Concerning US registration, dynamic endocardial contours resulting from speckle tracking echocardiography were exploited in a geometry-based dynamic registration. We propose the use of an adapted dynamic time warping procedure to synchronize cardiac dynamics in multiview US and cine MRI. The registration of US and LGE MRI was evaluated on a dataset of patients with hypertrophic cardiomyopathy. A visual assessment of 330 left ventricular regions from US images of 28 patients resulted in 92.7% of regions successfully aligned with cardiac structures in LGE. Successfully-aligned regions were then used to evaluate the abilities of strain indicators to predict the presence of fibrosis. Longitudinal peak-strain and peak-delay of aligned left ventricular regions were computed from corresponding regional strain curves from US. The Mann-Withney test proved that the expected values of these indicators change between the populations of regions with and without fibrosis (p < 0.01). ROC curves otherwise proved that the presence of fibrosis is one factor amongst others which modifies longitudinal peak-strain and peak-delay.

Multimodal Classification of Remote Sensing Images: A Review and Future Directions

Earth observation through remote sensing images allows the accurate characterization and identification of materials on the surface from space and airborne platforms. Multiple and heterogeneous image sources can be available for the same geographical region: multispectral, hyperspectral, radar, multitemporal, and multiangular images can today be acquired over a given scene. These sources can be combined/fused to improve classification of the materials on the surface. Even if this type of systems is generally accurate, the field is about to face new challenges: the upcoming constellations of satellite sensors will acquire large amounts of images of different spatial, spectral, angular, and temporal resolutions. In this scenario, multimodal image fusion stands out as the appropriate framework to address these problems. In this paper, we provide a taxonomical view of the field and review the current methodologies for multimodal classification of remote sensing images. We also highlight the most recent advances, which exploit synergies with machine learning and signal processing: sparse methods, kernel-based fusion, Markov modeling, and manifold alignment. Then, we illustrate the different approaches in seven challenging remote sensing applications: 1) multiresolution fusion for multispectral image classification; 2) image downscaling as a form of multitemporal image fusion and multidimensional interpolation among sensors of different spatial, spectral, and temporal resolutions; 3) multiangular image classification; 4) multisensor image fusion exploiting physically-based feature extractions; 5) multitemporal image classification of land covers in incomplete, inconsistent, and vague image sources; 6) spatiospectral multisensor fusion of optical and radar images for change detection; and 7) cross-sensor adaptation of classifiers. The adoption of these techniques in operational settings will help to monitor our planet from space in the very near future.

An Image Fusion Approach based on Adaptive Fuzzy Logic Model with Local Level Processing

Image fusion has lots of application in real life to furnish a combined form of many oriented objects of different images into single image. Adaptive Fuzzy logic model with local level processing is a controlling tool to model image characteristics accurately and been successfully applied to a large number of image processing applications. In this paper an adaptive fuzzy logic model have been proposed with local level processing for fusion of multi-exposure and multi-sensor images. Experimental results demonstrate the superiority of proposed method; it offers approximately 30%-35% improvement in Universal image quality index (UIQI) as compared to Marcov Random Field (MRF) fusion method.

Multi-sensor image fusion at signal level for improved near-surface crack detection

This study aims at improving the detection of near-surface defects in magnetizable and conductive specimens by combining the measurements of eddy current, magnetic flux leakage and thermography testing. Different signal processing methods for data normalization are proposed to enable data fusion at the pixel level. These methods are applied to a test specimen which contains 10 variably-sized defects. We quantitatively evaluate the performances of a total of 29 detection methods with respect to false alarm reduction at a fixed level of true positive rate. We report that false positive rate could be reduced from 1.65% down to 0.28% by the best multi-sensor method compared to the best single-sensor performance on the smallest defect, when 50% found flaw pixels are required for successful detection.

A Region-Based Image Fusion Algorithm for Detecting Trees in Forests

Multi-sensor image fusion has been widely used in many areas. In this paper, a region-based method using NSCT and fuzzy logic is proposed to enhance trees' contour and enrich the information in images. Firstly, the required source images are segmented, which are then decomposed using the NSCT transform. In low frequency domain, the fuzzy fusion method is used. And in high frequency domain, a region-based rule is proposed to highlight the objects' detail within images. Finally, the fused images are obtained by an inverse NSCT transform. Nine pieces of objective criteria and computation time are adopted to evaluate the fusion result. Moreover 30 groups of collected images are utilized to prove the robustness of the proposed method. The results show that the algorithm could provide fused images which have a higher visual effect, combine more information and consume less time. Furthermore, this process could better capture the contour and texture of the trees.

Spatial Resolution Enhancement of Hyperspectral Images Using Unmixing and Binary Particle Swarm Optimization

Hyperspectral imaging provides high spectral resolution and thereby improved classification, detection, and recognition capabilities with respect to standard imaging systems. However, hyperspectral images generally have low spatial resolution, varying from a few to tens of meters, resulting from technical limitations such as platform data storing capacity and satellite-to-ground transmission bandwidth. Spectral unmixing provides information on pixels in terms of abundances of pure spectral signatures, without providing spatial distribution at subpixel level. Multisensor image fusion approaches can provide such information but require an additional image with higher spatial resolution that is acquired in similar conditions with the hyperspectral image. In this letter, a novel spatial resolution enhancement method using fully constrained least squares (FCLS) spectral unmixing and spatial regularization based on modified binary particle swarm optimization is proposed to achieve spatial resolution enhancement in hyperspectral images, without using an additional image with higher spatial resolution. The proposed method has a highly parallel nature with respect to its counterparts in the literature and is fit to be adapted to field-programmable gate array architecture.

An efficient fusion technique for quality enhancement of remotely sensed images

Remote-sensing satellites provide both high-resolution panchromatic and low-resolution multi-spectral images. In this paper, a pixel-level multi-sensor image fusion technique is proposed for improving the spectral quality of the fused multi-spectral and panchromatic remote-sensing images. The proposed fusion technique integrates both the intensity, hue, and saturation (IHS) and the discrete wavelet frame transform (DWFT) techniques. IHS fusion technique can provide high spatial quality and the DWFT fusion technique is both aliasing free and translation invariant, but the color distortion in both techniques is noticeable. The proposed fusion technique can improve the spectral discrepancy of the fused images, while preserving the spatial quality to an acceptable level. Landsat-5 (TM) with Spot (Pan), Landsat-7 (ETM+), and IKONOS panchromatic and multi-spectral images have been fused using the proposed technique. The statistical analysis shows that this technique improves the fusion quality compared to the other known fusion techniques such as the conventional IHS, discrete wavelet transform (DWT), DWFT, and the integrated IHS and DWT.

Focused pooling for image fusion evaluation

This paper presents the results of an investigation into optimal spatial pooling of localised quality scores for use in objective evaluation of multisensor image fusion. We propose and evaluate a two stage focused pooling method with a localised aggregation of pixel-level performance estimates into regional fusion performance scores as the first step followed by a global pooling of regional scores into a global fusion performance score. We investigate a selection of linear and non-linear global pooling methods and show that quality driven methods which take into account regional fusion performance levels exhibit optimal performance. The proposed pooling algorithm is general and applicable to any fusion performance and quality metric based on structural preservation estimates, local differences between input and fused images. Specifically, we evaluate the proposed method in conjunction with three well-known structural preservation fusion metrics against their baseline pooling methods. We show, through correlation with an extensive subjectively annotated dataset of fused images, that regional aggregation of local performance scores over 3–6° of visual angle with selection of the worst performing region as the global score can improve performance for all the structural fusion metrics tested.

A texture-based fusion scheme to integrate high-resolution satellite SAR and optical images

Image fusion of different types of satellite images is of widespread interest due to the increasing availability of various spaceborne imaging sensors. In this letter, a new multi-sensor image fusion method is presented, where a texture-based fusion rule is used to address the fusion of synthetic aperture radar (SAR) and optical images. At first, the integration of panchromatic (PAN) and SAR image is achieved in wavelet domain by taking weighted average of the corresponding input pixels. The weights are adaptively decided according to the local spatial autocorrelation characteristics of SAR texture image. This integrated image is fused into the multispectral (MS) image through generalized intensity-hue-saturation (GIHS) image fusion method to obtain the final fused image. To quantitatively evaluate and test the performance of the proposed method, comparisons with some existing fusion methods are carried out in this letter. Experimental results demonstrate that the proposed method shows better performance than other fusion methods and can get satisfactory fusion results.

A Novel Image Fusion Method based on Translation Invariant Wavelet Transforms

In this paper, a novel method based on translation-invariant wavelet decomposition has been introduced to the pixel level multisensor image fusion. The proposed fusion architecture is related to the “shift-decompose-fuse-shift” technique and consists of many steps. First, the source images to be shifted in both horizontal and vertical directions. The shifted images will be translated into the wavelet domain and by repeating “shift-translate” to get the decomposition of source images. Second, the different subband coefficients of the fused image are combined with the proposed fusion rule. Finally, the fused image will be obtained with the inverse translate and shift. Experimental results demonstrate that the proposed method fuses the useful information from source images and outperforms the discrete wavelet transform (DWT) and stationary wavelet transform (SWT).

Image fusion based on bilateral sharpness criterion in DT-CWT domain

Since last few decades, multi sensor image fusion has been an emerging field of research in remote sensing, medical imaging and variety of computer vision applications. The primary objective of image fusion lies in the formation of a perceptually enhanced image from several multi sensor images using an appropriate fusion rule. The discrete wavelet transform DWT-based image fusion techniques have been popular due to less redundancy, low computations and perfect reconstruction with short support filters. But, it suffers severely from lack of directionality, shift variance, oscillations and aliasing problems. These issues have been overcome by means of Q-shift dual-tree complex wavelet transform DT-CWT-based image fusion. In this paper, an improved DT-CWT-based image fusion technique has been proposed to compose a resultant image with better perceptual as well as quantitative image quality indices. A bilateral sharpness based weighting scheme has been implemented for the high frequency coefficients taking both gradient and its phase coherence in account. A normalised maximum gradient weighting scheme is implemented for low frequency wavelet components. The fusion results demonstrate that the proposed fusion technique is more effective and competitive in terms of entropy, total standard deviation, average gradient measure and edge intensity measure.