Image Fusion Model Research Articles

An Improved Pulse-Coupled Neural Network Model for Pansharpening.

Pulse-coupled neural network (PCNN) and its modified models are suitable for dealing with multi-focus and medical image fusion tasks. Unfortunately, PCNNs are difficult to directly apply to multispectral image fusion, especially when the spectral fidelity is considered. A key problem is that most fusion methods using PCNNs usually focus on the selection mechanism either in the space domain or in the transform domain, rather than a details injection mechanism, which is of utmost importance in multispectral image fusion. Thus, a novel pansharpening PCNN model for multispectral image fusion is proposed. The new model is designed to acquire the spectral fidelity in terms of human visual perception for the fusion tasks. The experimental results, examined by different kinds of datasets, show the suitability of the proposed model for pansharpening.

Improving phenological monitoring of winter wheat by considering sensor spectral response in spatiotemporal image fusion

Multisensor image fusion results may deviate from accurately reflecting the phenological stages of winter wheat because different responses of satellite sensors to the spectrum lead to the radiometric inconsistency between different remote sensing images. To reduce the effect of the difference in the physical electromagnetic spectrum responses between sensors on monitoring the phenological stages of winter wheat by fusion results, Sensor Spectral Response (SSR) should be considered in spatiotemporal fusion methods. This paper proposes a novel image fusion model by introducing SSR into the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM). The contribution of SSR in minimizing the effect of the system difference between sensors on image fusion products is parameterized as a calibration factor by matrixing operation, which is able to offset the systematic inconsistency between different sensor images. Linear regression equation for different land cover type and spectral band is established to calculate the weights needed in STARFM for improving the selection of neighboring spectrally similar pixels. This proposed method is evaluated using one satellite datasets including four ZY-3 (5.8 m) and Landsat 8 OLI (30 m) scenes which are acquired during the growth stages of winter wheat from seedling to harvest. Qualitative and quantitative evaluation shows that the proposed method can better monitor the phenology of winter wheat with an improved spatial and temporal consistency with the observations than STARFM.

A Deep Model for Multi-Focus Image Fusion Based on Gradients and Connected Regions

In this paper, we propose a novel unsupervised model for multi-focus image fusion based on gradients and connected regions, termed as GCF. To overcome the stumbling block of vanishing gradients in applying deep networks for multi-focus image fusion, we design the Mask-Net which can directly generate a binary mask. Thus, there is no need for hand-crafted feature extraction or fusion rules. Based on the fact that objects within the depth-of-field (DOF) have shaper appearance, i.e., larger gradients, we use the gradient relation map obtained from source images to narrow the solution domain and speed up convergence. Then, the constraint of connected region numbers is conductive to finding the more accurate binary mask. With the consistency verification strategy, the final mask can be obtained by adapting the initial binary mask to generate the fused result. Therefore, the proposed method is an unsupervised model without the need of the ground-truth data. Both qualitative and quantitative experiments are conducted on the publicly available Lytro dataset. The results show that GCF can outperform the state-of-the-art in both visual perception and objective metrics.

Image fusion employing adaptive spectral-spatial gradient sparse regularization in UAV remote sensing

Unmanned aerial vehicle (UAV) remote sensing has been widely used in vegetation phenotypes and precision agriculture. The fusion of UAV multispectral and panchromatic images has considerable research value. For example, an accurate vegetation index can be obtained. However, large geometrical distortions are observed in UAV images, contributing to the insufficiency of existing fusion algorithms. Spectrum consistency, which indicates that changes in spectral direction are always a smooth function, is investigated in this paper to solve the above problem. Spatial adaptivity is also introduced to reduce spectral distortion in the fusion process. Based on the two aspects, a multispectral and panchromatic image fusion model employing adaptive spectral-spatial gradient sparse regularization is proposed for UAV remote sensing. The separable approximation and augmented Lagrangian methods are employed to optimize this model. In the experiments, the proposed method is firstly compared with other state-of-the-art fusion algorithms, and good performance is verified by UAV datasets in terms of visual effect and objective quality analysis. Secondly, the fusion algorithm is applied in the application of a vegetation phenotype. The experiments finally demonstrate that accurate vegetation indices can be generated by adopting the proposed algorithm. This finding proves the substantial research value of the proposed algorithm in UAV remote sensing.

Comparison of Five Spatio-Temporal Satellite Image Fusion Models over Landscapes with Various Spatial Heterogeneity and Temporal Variation

In recent years, many spatial and temporal satellite image fusion (STIF) methods have been developed to solve the problems of trade-off between spatial and temporal resolution of satellite sensors. This study, for the first time, conducted both scene-level and local-level comparison of five state-of-art STIF methods from four categories over landscapes with various spatial heterogeneity and temporal variation. The five STIF methods include the spatial and temporal adaptive reflectance fusion model (STARFM) and Fit-FC model from the weight function-based category, an unmixing-based data fusion (UBDF) method from the unmixing-based category, the one-pair learning method from the learning-based category, and the Flexible Spatiotemporal DAta Fusion (FSDAF) method from hybrid category. The relationship between the performances of the STIF methods and scene-level and local-level landscape heterogeneity index (LHI) and temporal variation index (TVI) were analyzed. Our results showed that (1) the FSDAF model was most robust regardless of variations in LHI and TVI at both scene level and local level, while it was less computationally efficient than the other models except for one-pair learning; (2) Fit-FC had the highest computing efficiency. It was accurate in predicting reflectance but less accurate than FSDAF and one-pair learning in capturing image structures; (3) One-pair learning had advantages in prediction of large-area land cover change with the capability of preserving image structures. However, it was the least computational efficient model; (4) STARFM was good at predicting phenological change, while it was not suitable for applications of land cover type change; (5) UBDF is not recommended for cases with strong temporal changes or abrupt changes. These findings could provide guidelines for users to select appropriate STIF method for their own applications.

Variational Osmosis for Non-linear Image Fusion.

We propose a new variational model for non-linear image fusion. Our approach is based on the use of an osmosis energy term related to the one studied in Vogel et al. [44] and Weickert et al. [45]. The minimization of the proposed non-convex energy realizes visually plausible image data fusion, invariant to multiplicative brightness changes. On the practical side, it requires minimal supervision and parameter tuning and can encode prior information on the structure of the images to be fused. For the numerical solution of the proposed model, we develop a primal-dual algorithm and we apply the resulting minimization scheme to solve multi-modal face fusion, color transfer and cultural heritage conservation problems. Visual and quantitative comparisons to state-of-the-art approaches prove the out-performance and the flexibility of our method.

A Methodical Block based Feature Level Image Fusion Technique with Wavelet Transform using Neural Network for Satellite Images

This paper describe about the feature extraction or detection machine learning application which one is wavelet transform integrated with neural network. It has obtained an effective block based feature level with wavelet transform using neural network (BFWN) model for image fusion. In the projected BFWN model, the discrete wavelet transform (DWT) and neural network (NN) are considered for fusing IRS-1D images using LISS- III scanner about the location different areas in India. Also Quick Bird image data and Landsat 7 image data are used to carry out on the proposed BFWN method. The characteristics like contrast visibility, energy of gradient, spatial frequency, variance and edge information are under study. A Feed forward back propagation neural network is trained and tested for categorization since the learning capability of neural network makes it feasible to customize the image fusion process. The trained neural network is used to fuse the two source images. The proposed BFWN model is distinguish, with DWT alone to assess the quality of the fused image. The results obviously show that the proposed BFWN model is a capable and feasible algorithm for image fusion

The verification model of multi-focus image fusion by simulating subjective evaluation

We present a model to simulate the subjective evaluation and compare various fusion algorithms. First, we produce an all-focus image and two multi-focus images by the help of two filters. Second, we fuse two multi-focus images with two representative algorithms WT and CT. Third, we decide the optimal fusion rules of WT and CT by comparing the fused images with an all-focus image. Finally, we improve the performances of fused images by combining several algorithms. Simulation shows the verification model can compare various fusion algorithms or rules. Meanwhile, our fusion model can get better performances than other algorithms or rules.

IFCNN: A general image fusion framework based on convolutional neural network

In this paper, we propose a general image fusion framework based on the convolutional neural network, named as IFCNN. Inspired by the transform-domain image fusion algorithms, we firstly utilize two convolutional layers to extract the salient image features from multiple input images. Afterwards, the convolutional features of multiple input images are fused by an appropriate fusion rule (elementwise-max, elementwise-min or elementwise-mean), which is selected according to the type of input images. Finally, the fused features are reconstructed by two convolutional layers to produce the informative fusion image. The proposed model is fully convolutional, so it could be trained in the end-to-end manner without any post-processing procedures. In order to fully train the model, we have generated a large-scale multi-focus image dataset based on the large-scale RGB-D dataset (i.e., NYU-D2), which owns ground-truth fusion images and contains more diverse and larger images than the existing datasets for image fusion. Without finetuning on other types of image datasets, the experimental results show that the proposed model demonstrates better generalization ability than the existing image fusion models for fusing various types of images, such as multi-focus, infrared-visual, multi-modal medical and multi-exposure images. Moreover, the results also verify that our model has achieved comparable or even better results compared to the state-of-the-art image fusion algorithms on four types of image datasets.

Conditional Random Field Model for Robust Multi-Focus Image Fusion.

In this paper, a novel multi-focus image fusion algorithm based on conditional random field optimization (mf-CRF) is proposed. It is based on an unary term that includes the combined activity estimation of both high and low frequencies of the input images, while a spatially varying smoothness term is introduced, in order to align the graph-cut solution with boundaries of focused and defocused pixels. The proposed model retains the advantages of both spatial-domain methods and multi-spectral methods and by solving an energy minimization problem and finds an optimal solution for the multi-focus image fusion problem. Experimental results demonstrate the effectiveness of the proposed method that outperforms current state-of-the-art multi-focus image fusion algorithms in both qualitative and quantitative comparisons. In this paper, the successful application of the mf-CRF model in multi-modal image fusion (visible-infrared and medical) is also presented.

Nonlocal Coupled Tensor CP Decomposition for Hyperspectral and Multispectral Image Fusion

Hyperspectral (HS) super-resolution, which aims at enhancing the spatial resolution of hyperspectral images (HSIs), has recently attracted considerable attention. A common way of HS super-resolution is to fuse the HSI with a higher spatial-resolution multispectral image (MSI). Various approaches have been proposed to solve this problem by establishing the degradation model of low spatial-resolution HSIs and MSIs based on matrix factorization methods, e.g., unmixing and sparse representation. However, this category of approaches cannot well construct the relationship between the high-spatial-resolution (HR) HSI and MSI. In fact, since the HSI and the MSI capture the same scene, these two image sources must have common factors. In this paper, a nonlocal tensor decomposition model for hyperspectral and multispectral image fusion (HSI-MSI fusion) is proposed. First, the nonlocal similar patch tensors of the HSI are constructed according to the MSI for the purpose of calculating the smooth order of all the patches for clustering. Then, the relationship between the HR HSI and the MSI is explored through coupled tensor canonical polyadic (CP) decomposition. The fundamental idea of the proposed model is that the factor matrices in the CP decomposition of the HR HSI’s nonlocal tensor can be shared with the matrices factorized by the MSI’s nonlocal tensor. Alternating direction method of multipliers is used to solve the proposed model. Through this method, the spatial structure of the MSI can be successfully transferred to the HSI. Experimental results on three synthetic data sets and one real data set suggest that the proposed method substantially outperforms the existing state-of-the-art HSI-MSI fusion methods.

Image contrast expand enhancement system based on fuzzy theory

In the current age of information explosion, the extraction of relevant data from a large information pool, which includes images, has become crucial. Because of the easy availability of imaging devices, millions of images are being added to the image pool every day. Current image contrast enhancement methods have some drawbacks. First, for images captured in extreme lighting conditions, such as backlit images and extremely bright or dark images, extant image contrast enhancement methods cannot retain the brightness distribution details of the original image, resulting in image distortion. Second, the methods can only achieve global contrast enhancement but not local contrast enhancement. Third, these methods cannot satisfy the Human Visual System mapping curve, resulting in nonsmooth or distorted images. In this study, a novel image expand enhancement system based on fuzzy theory is proposed. This system has two major features: (1) the expand model is designed such that the correlations between light intensity and the intensity of an image captured in this light are considered to achieve local contrast enhancement, and (2) an extreme case of images processing step that is capable of enhancing. The proposed model has a fusion framework; therefore, the image fusion model can represent the relationship between the original image and the expanded image. Next, the fusion model describes some expected statistical properties of the desired intensity of the expand image. Thus, the details of the original image are retained. Experimental results revealed that the proposed algorithm is capable of adaptively enhancing the contrast of the original image while simultaneously extruding the details of objects in the image. The resulting enhanced image can be effectively used for image information analysis and other image-processing tasks.

Accuracy of capturing oncology facial defects with multimodal image fusion versus laser scanning

Statement of problemFabrication of conventional facial prostheses is a labor-intensive process which traditionally requires an impression of the facial defect and surrounding tissues. Inaccuracies occur during the facial moulage because of soft-tissue compression, the patient's reflex movements, or the lack of support for the impression material. A variety of 3D imaging techniques have been introduced during the production of facial prostheses. However, the accuracy of the different imaging techniques has not been evaluated sufficiently in this clinical context. PurposeThe purpose of this in vitro study was to compare the difference in accuracy of capturing oncology facial defects with multimodal image fusion and laser scanning against a cone beam computed tomography (CBCT) reference scan. Material and methodsTen gypsum casts of oncology facial defects were acquired. To produce reference models, a 3D volumetric scan was obtained using a CBCT scanner and converted into surface data using open-source medical segmentation software. This model was cropped to produce a CBCT mask using an open-source system for editing meshes. The multimodal image fusion model was created using stereophotogrammetry to capture the external facial features and a custom optical structured light scanner to record the defect. The gypsum casts were also scanned using a commercial 3D laser scanner to create the laser-scanned model. Analysis of the best fit of each experimental model to the CBCT mask was performed in MeshLab. The unsigned mean distance was used to measure the absolute deviation of each model from the CBCT mask. A paired-samples t test was conducted to compare the mean global deviation of the 2 imaging modalities from the CBCT masks (α=.05). ResultsA statistically significant difference was found in the mean global deviation between the multimodal imaging model (220 ±50 μm) and the laser-scanned model (170 ±70 μm); (t(9)=2.56, P=.031). The color error maps illustrated that the greatest error was located at sites distant to the prosthesis margins. ConclusionsThe laser-scanned models were more accurate; however, the mean difference of 50 μm is unlikely to be clinically significant. The laser scanner had limited viewing angles and a longer scan time which may limit its transferability to maxillofacial practice.

Comparison of Spatiotemporal Fusion Models for Producing High Spatiotemporal Resolution Normalized Difference Vegetation Index Time Series Data Sets

It has a great significance to combine multi-source with different spatial resolution and temporal resolution to produce high spatiotemporal resolution Normalized Difference Vegetation Index (NDVI) time series data sets. In this study, four spatiotemporal fusion models were analyzed and compared with each other. The models included the spatial and temporal adaptive reflectance model (STARFM), the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM), the flexible spatiotemporal data fusion model (FSDAF), and a spatiotemporal vegetation index image fusion model (STVIFM). The objective of is to: 1) compare four fusion models using Landsat-MODIS NDVI image from the Banan district, Chongqing Province; 2) analyze the prediction accuracy quantitatively and visually. Results indicate that STVIFM would be more suitable to produce NDVI time series data sets.

Infrared and low-light-level image fusion based on ℓ2-energy minimization and mixed-ℓ1-gradient regularization

In order to compensate for the visual defect of the low-light-level image and combine the saliency features of the infrared image, this paper proposes an infrared and low-light-level image fusion model based on ℓ2-energy minimization and mixed-ℓ1-gradient regularization. First, this novel model uses the non-subsampled shearlet transform (NSST) as a multi-scale decomposition tool to capture the low and high-frequency components of the source images. Because the NSST has good localization characteristics, excellent directional selectivity, parabolic edge characteristics, and translation invariance, it is more suitable for image decomposition and reconstruction. Secondly, for the low-frequency components that reflect the energy information, an optimization model based on ℓ2-energy minimization is adopted as its fusion rule. This new rule allows the fused image to have similar pixel intensities to the given infrared image, thus improving the visual observation of the fused image and reducing the influence of the brightness defect under weak light. Thirdly, considering that the ℓ1-norm encourages the sparseness of the gradients, this paper uses the ℓ1-gradient regularization to guide the fusion of high-frequency components. This method can greatly restore the gradient features hidden in the source images to the fused image so that the fused image will have clearer edge details. In order to verify the effectiveness of the proposed algorithm, we adopted 6 × 6 independent fusion experiments. The final experimental results show that the proposed algorithm has better visual effects in the fusion problem of low-light-level environment, and the performance of objective evaluation is also good, which is better than other existing typical methods.

MODIS ocean color product downscaling via spatio-temporal fusion and regression: The case of chlorophyll-a in coastal waters

Detailed and accurate information on the spatial variation of chlorophyll-a concentration in coastal waters is a critical component of ocean ecology and environmental research. The daily MODIS chlorophyll-a products provided by NASA, with 1 km spatial resolution, are suitable for monitoring this variation globally, but these products are too coarse to apply in practice to obtain detailed information over coastal waters. Early studies have shown that spatiotemporal data fusion techniques can be used to predict higher spatial resolution land-cover data based on time-series information in MODIS and the detailed texture from Landsat. However, this technology hasn’t been tested to determine whether it can be used to predict higher spatial-resolution data in coastal waters with rapid water movement. This study aims to answer this question by providing a method to downscale the MODIS chlorophyll-a products from 1 km spatial resolution to 30 m. The spatiotemporal data fusion model U-STFM and the regression model NASA OC2M-HI were used to combine the texture and chlorophyll-a information from Landsat and MODIS. An area with rapid water movement in Bohai Bay of the Bohai Sea, northeast China, was selected for this study. Twelve matched images from MODIS in Aqua platform and Landsat 8, taken over a period of five years (2013–2017), were used to better predict detailed remote-sensing reflectance (Rrs) on the targeted days. Landsat 8 Rrs was used as ground-truth data to assess the output. The results on Mar 10th, 2016, show: 1) The downscaled results (30 m) from the U-STFM model indicate a more stable prediction of Rrs with RMSE of 0.00177 and 0.00202 and R-squared of 0.868 and 0.881 for the blue and green bands, respectively. Results from STARFM and ESTARFM fusion models are also compared in this study. 2) High correlation between log10(U-STFM Blue/ U-STFM Green) and log10(MODIS Chl) captured by OC2M-HI regression model at 1 km scale with R-squared up to 0.85 and RMSE up to 0.742 mg/m^3. This correlation was further used to predict the final chlorophyll-a concentration prediction at 30 m scale on Mar 10th, 2016; 3) The Landsat 8 chlorophyll-a product was used as reference data to evaluate the final chlorophyll-a concentration prediction (30 m) and the original MODIS chlorophyll-a product. The result shows the final prediction (30 m) maintains the accuracy of MODIS chlorophyll-a product and highly improved the local texture details near coastal waters. Predictions on nine other targeted dates with similar conclusions were also evaluated in this paper. The results in this study suggest that low spatial-resolution (1 km) daily MODIS chlorophyll-a products can be downscaled to higher resolution (30 m) products based on the U-STFM image fusion model and NASA’s OC2M-HI regression model to better understand the dynamic patterns of chlorophyll-a concentration in coastal waters.

Visual positioning system for small-scaled spherical robot in underwater environment

Aiming at application requirements of our small-scaled spherical amphibious robots, a visual positioning system adopting the model of depth image and feature fusion was designed and implemented in this paper. Considering the narrow load space, the limited structure space and the specialized underwater application scenarios of the spherical robot (the diameter of the upper and lower hemisphere is 320 and 350 mm, respectively), a Time of Flight RGB-D camera was used, and the color image and depth image could be acquired, thus the position of objective could be calculated from image information, a calibration experiment for distance correction was designed to improve the accuracy of the positioning system. Based on the theoretical analysis and calculation, a series of underwater experiments were carried out to test the performance of the proposed positioning method. Experimental results confirmed the feasibility of the proposed visual positioning method, and the maximum error between the actual positioning result and the reference value is less than 7 mm, which could meet future demands of our amphibious spherical robot in biological monitoring and multi robot cooperation.

FWFusion: Fuzzy Whale Fusion model for MRI multimodal image fusion

Medical treatment and diagnosis require information that is taken from several modalities of images like Magnetic Resonance Imaging (MRI), Computerized Tomography and so on. The information obtained for certain ailments is often incomplete, invisible and lacking in consistent scanner performance. Hence, to overcome these issues in the image modalities, image fusion schemes are developed in the literature. This paper proposes a hybrid algorithm using fuzzy concept and a novel P-Whale algorithm, called Fuzzy Whale Fusion (FWFusion), for the fusion of MRI multimodal images. Two multimodal images from MRI (T1, T1C, T2 and FLAIR) are considered as the source images, which are fed as inputs to a wavelet transform. The transform utilized converts the images into four different bands, which are fused using two newly derived fusion factors, fuzzy fusion and whale fusion, in a weighted function. The proposed P-Whale approach combines Whale Optimization Algorithm (WOA) and Particle Swarm Optimization (PSO) for the effective selection of whale fusion factors. The performance of FWFusion model is compared to those of the existing strategies using Mutual Information (MI), Peak Signal-to-Noise Ratio (PSNR) and Root Mean Squared Error (RMSE), as the evaluation metrics. From the mean performance evaluation, it is observed that the proposed approach can achieve MI of 1.714, RMSE of 1.9 and PSNR of 27.9472.

A novel improved deep convolutional neural network model for medical image fusion

This paper proposed a novel fusion scheme for muti-modal medical images that utilizes both the features of the multi-scale transformation and deep convolutional neural network. Firstly, the source images are decomposed by the Gauss-Laplace filter and Gaussian filter into several sub-images in the first layer of network. Then, HeK-based method is used to initialize the convolution kernel of the rest layers, construct the basic unit, and use the back propagation algorithm to train the basic unit; Train multiple basic units that are sacked with the thought of SAE to get the deep stacked neural network; the proposed network is adopted to decompose the input images to obtain their own high frequency and low frequency images, and combine the our fusion rule to fuse the two high frequency and low frequency images, and put them back to the last layer of the network to get the final fusion images. The performance of our proposed fusion method is evaluated by conducting several experiments on the different medical image datasets. Experimental results demonstrate that our proposed method does not only produce better results by successfully fusing the different images, but also ensures an improvement in the various quantitative parameters as compared to other existing methods. In addition, the speed of our improved CNN method is much faster than that of comparison algorithms which have good fusion quality.

A robust adaptive spatial and temporal image fusion model for complex land surface changes

Spatial and temporal satellite image fusion (STIF) has provided a feasible alternative for generating imagery with both high spatial and temporal resolution, thus expanding the applications of existing satellite sensors. However, a critical challenge confronting the further development of STIF is to systematically and robustly address complex land surface changes, which include land cover changes without shape changes (e.g., crop rotation) and land cover changes with shape changes (e.g., urban expansion), in addition to conventional land surface changes (e.g., phenological changes of vegetation). This paper presents the Robust Adaptive Spatial and Temporal Fusion Model (RASTFM) to tackle this challenge with one prior pair of MODIS-Landsat images. In RASTFM, land surface changes are reorganized into non-shape changes (including phenological changes and land cover changes without shape changes) and shape changes (i.e., land cover changes with shape changes), which are handled differently. However, both non-shape changes and shape changes are predicted through a Non-Local Linear Regression (NL-LR) of the subject pixel's similar neighbors. A regression based high-pass modulation is also performed as a post-processing step to improve both the spatial details and spectral fidelity of the predicted Landsat image. Unlike other STIF models (e.g., the Spatial and Temporal Adaptive Reflectance Fusion Model, STARFM), RASTFM can find similar neighboring pixels more precisely through a non-local searching strategy and derives the weights of similar neighbors more rigorously via a linear regression model. As both non-shape and shape changes are treated based on the regression of similar neighboring pixels, the land surface changes are processed in a unified manner. Experiments that use one simulated and three actual MODIS-Landsat datasets featured by different types of land surface changes were conducted to demonstrate the performance of RASTFM. Comparisons with the state-of-the-art STIF models, including weighted function, unmixing and dictionary-learning methods, show that NL-LR based RASTFM can capture the land surface changes in various landscapes more accurately and robustly in a unified manner, thereby facilitating the continuous and detailed monitoring of complex and diverse land surface dynamics.