Image Fusion Process Research Articles

Infrared and visible image fusion with ResNet and zero-phase component analysis

In image fusion approaches, feature extraction and processing are key tasks, and the fusion performance is directly affected by the different features and processing methods undertaken. However, most of deep learning-based methods use deep features directly without them. This leads to the fusion performance degradation in some cases. To solve these drawbacks, in our paper, a deep features and zero-phase component analysis (ZCA) based novel fusion framework is proposed. Firstly, the residual network (ResNet) is used to extract deep features from source images. Then ZCA and l1-norm are utilized to normalize the deep features and obtain initial weight maps. The final weight maps are obtained by employing a soft-max operation in association with the initial weight maps. Finally, the fused image is reconstructed using a weighted-averaging strategy. Compared with the existing fusion methods, experimental results demonstrate that the proposed framework achieves better performance in both objective assessment and visual quality. The code of our fusion algorithm is available at https://github.com/hli1221/imagefusion_resnet50.

A Methodical Block based Feature Level Image Fusion Technique with Wavelet Transform using Neural Network for Satellite Images

This paper describe about the feature extraction or detection machine learning application which one is wavelet transform integrated with neural network. It has obtained an effective block based feature level with wavelet transform using neural network (BFWN) model for image fusion. In the projected BFWN model, the discrete wavelet transform (DWT) and neural network (NN) are considered for fusing IRS-1D images using LISS- III scanner about the location different areas in India. Also Quick Bird image data and Landsat 7 image data are used to carry out on the proposed BFWN method. The characteristics like contrast visibility, energy of gradient, spatial frequency, variance and edge information are under study. A Feed forward back propagation neural network is trained and tested for categorization since the learning capability of neural network makes it feasible to customize the image fusion process. The trained neural network is used to fuse the two source images. The proposed BFWN model is distinguish, with DWT alone to assess the quality of the fused image. The results obviously show that the proposed BFWN model is a capable and feasible algorithm for image fusion

Fusion of Medical Images using a Wavelet Methodology: A Survey

Image compression or fusion is the concept of identifying in-depth parameters of disease variables, and requires output images that preserve all the viable and prominent information that is gathered from source images without any further introduction of artifacts or unnecessary distortions. Measurement of images for prospective evaluation and image fusion depends on various performance measures, such as structure similarity index, standard deviation, edge detection, correlation coefficient and high pass correlation, average gradient, root-mean-square error, peak signal-to-noise ratio, entropy, etc. This review discusses various medical image fusion modalities focused on Principal Component Analysis, Independent Component Analysis, and wavelet transform. An introduction to the usefulness of such modalities is presented, suggesting safe hybrid modality combinations that could greatly enhance the image fusion process. Novel trends in medical image fusion techniques to achieve a perfectly desired, quality image, the future prospects of an ideal technique for medical imaging, and recognition of diseases are covered.

Joint Visualization Diagnosis of Outdoor Insulation Status With Optical and Acoustical Detections

This paper proposes a joint visualization detection method to diagnose outdoor insulation status of high-voltage (HV) equipment in substations, which applies image matching and fusion techniques to accurately combine the results of ultrasonic, ultraviolet, and infrared detection in an image. This image can then be used to quickly and clearly locate the region of corona discharge and overheating of outdoor substation HV equipment. First, ultrasonic localization visualization of corona discharge is proposed, which is based on the designed double-helix array and multisignal classification algorithm. This method provides a more intuitive way for the diagnosis of outdoor insulation status. Then, ultraviolet localization visualization method is detailed, and an adaptive affine matrix is designed for the accurate location of corona discharge. Following this, infrared-visible detection of overheating failure is introduced, and the speeded-up robust features algorithm and random sample consensus algorithm are used to implement image fusion process. Finally, the results of image fusion are provided, and location error evaluations are conducted for each module. With acceptable cost and advantages of three detection method, the proposed visualization method is used for evaluating the insulation status of substation equipment more intuitively and conveniently, and thus, has an extensive application prospect.

An effective approach to selecting the appropriate pan-sharpening method in digital change detection of natural ecosystems

In an image fusion process, the spatial resolution of a multispectral image is improved by a panchromatic band. However, due to the spatial and spectral resolution differences between these two data sets, the enhanced image may have two distortions, spatial and spectral. Therefore, to evaluate the efficiency of the pan-sharpening method, the status of these two types of distortions is examined. Unfortunately, there is still no developed acceptance index that can thoroughly investigate the quality of the pansharpened image; moreover, most of the proposed methods for reviewing the quality of output images have been developed with an emphasis on the residential area. Accordingly, to assess the quality of the pansharpened image in this study, we evaluated highly effective conventional methods, such as visual examinations, quantitative evaluation and impact analysis regarding the change detection process of mangrove forests. Finally, we suggested a simple yet efficient approach for such research in natural ecosystems. In the proposed method, based on the nature of the ecosystem, a spectral vegetation index is applied to the pansharpened images; the spectral quality of the images is further evaluated based on two parameters, 1) the areas under the curves of the histogram of the spectral vegetation index in the natural ecosystem region and 2) its centroid. The spatial quality of the pansharpened images is evaluated through implementing of two transects perpendicular to each other in the images of the spectral vegetation index, and creating a spatial deviation on them. With expert reviews and visual evaluation of the pansharpened images, the proposed method, especially in natural ecosystems, has more advantages as regards assessing the quality of the fused images. Based on the evaluations, among 11 methods of pansharpening, including Ehlers Fusion, FuzeGO, Gram-Schmidt, HPF, HCS, PCA, Modified IHS, Brovey Transform, Projective Resolution Merge, Wavelet IHS, and Wavelet PCA; the HPF method the Brovey Transform and Modified IHS methods respectively showed the best performance in the digital change detection of Mangrove forests.

Design of Secure Image Fusion Technique Using Cloud for Privacy-Preserving and Copyright Protection

Nowadays, the use of digital content or digital media is increasing day by day. Therefore, there is a need to protect the digital document from both unauthorized users and authorized users. The digital document should be protected from authorized users who try to redistribute it illegally. Digital watermarking techniques along with cryptography are insufficient to ensure an adequate level of security of digital media. The security of the transferring digital data in the modern world is also a big challenge because there is a high risk of security breaches. In this article, a secure technique of image fusion using hybrid domains (spatial and frequency) for privacy preserving and copyright protection is proposed. The proposed method provides a secure technique for the digital content in cloud environment. Two cloud services are used to develop this work, which eliminates the role of a trusted third party (TTP). First is the design of an infrastructure as a service (IaaS) to store different images with encryption processes to speed up the image fusion process and save storage space. Second, a Platform as a Service (PaaS) is used to enable the digital content to improve computation power and to increase the bandwidth. The prime objective of the proposed scheme is to transfer the digital media between a service provider and customer in a secure way using a hybrid domain along with cloud storage. Imperceptibility and robustness measures are used to calculate the performance of the proposed approach.

Fusion-based encryption scheme for cancelable fingerprint recognition

This paper presents a fingerprint image encryption scheme based on fingerprint image fusion with another visible image that is rich in details. The encryption process is performed with chaotic Baker map, which has large immunity to noise. The image fusion process is performed with the Haar wavelet transform, and it can be implemented with the average or maximum fusion rule. The fusion process is performed, because fingerprint images are not rich in details, and hence the direct application of chaotic Baker map encryption will not be efficient for encrypting this type of images. To obtain an image that is rich in details, it is possible to use another encrypted image with a strong ciphering algorithm such as the RC6. Several perspectives are considered for performance evaluation of the proposed encryption scheme including visual inspection, histogram analysis, correlation coefficient, entropy analysis, processing time, and the effect of noise after decryption. The proposed fingerprint encryption scheme is appropriate for cancelable biometric applications to preserve the privacy of users by keeping their original fingerprints away from usage in the recognition system. The simulation results demonstrate that the proposed image encryption scheme gives a proficient and secure path for unique encrypted fingerprints. Both Equal Error Rate (EER) and Area under Receiver Operating Characteristic (AROC) curve are used for performance evaluation of the proposed cancelable fingerprint recognition scheme revealing high performance.

Multifocus image fusion based on waveatom transform

Multifocus image fusion has emerged as a challenging research area due to the availability of various image-capturing devices. The optical lenses that are widely utilized in image-capturing devices have limited ‘depth-of-focus’ and, therefore, only the objects that lie within a particular depth remain ‘in-focus’, whereas all the other objects go ‘out-of-focus’. In order to obtain an image where all the objects are well focused, multifocus image fusion method based on waveatom transform is proposed. The core idea is to decompose all input images using waveatom transform and perform fusion of resultant waveatom coefficients. The waveatom coefficients with higher visibility, corresponding to sharper image intensities, are used to perform the process of image fusion. Finally, the fused image is obtained by performing inverse waveatom transform. The performance of the proposed method is demonstrated by performing fusion on different sets of multifocus images and comparing the results of the proposed method to the results of existing image fusion methods.

CNNs hard voting for multi-focus image fusion

The main idea of image fusion is gathering the necessary features and information into one image. The multi-focus image fusion process is gathering this information from the focused areas of many images and the ideal fused image have all focused part from the input images. There are many studies of multi-focus image fusion in the spatial and transform domains. Recently the multi-focus image fusion methods based on deep learning have been emerged, and they have enhanced the decision map greatly. Nevertheless, the construction of an ideal initial decision map is still difficult and inaccessible. Therefore, the previous methods have high dependency on vast post-processing algorithms. This paper proposes a new convolution neural networks (CNNs) based on ensemble learning for multi-focus image fusion. This network uses hard voting of three branches CNNs that each branch is trained on three different datasets. It is very reasonable and reliable to use various models and datasets instead of just one and it would help to the network for improving the accuracy of classification. In addition, this paper introduces new simple arranging of the patches of the multi-focus datasets that is very useful in obtaining better classification accuracy. With this new arrangement of datasets, three types of multi-focus datasets are created with the help of gradient in the directions of vertical and horizontal. This paper illustrates that the initial segmented decision map of the proposed method is very cleaner than the others, and even it is cleaner than the other final decision maps after refined with a lot of post-processing algorithms. The conducted experimental results and analysis evidently validate that the proposed network have the cleanest initial decision map and the best quality of the output fused image compared to the other state of the art methods. These comparisons are performed with various qualitative and quantitative assessments that are assessed by several fusion metrics for demonstrating the superiority of the proposed network.

Adaptive image fusion algorithm based on human visual system guided gradient transfer and total variation minimization

To effectively explore image content for image fusion and generate a full end-to-end image fusion pipeline, a parameter adaption algorithm based on gradient transfer and total variation (TV) minimization is proposed, and the fusion problem is transformed into a human visual system guided TV minimization problem, where the data fidelity term maintains the structure edge features of the image, and the regularization term preserves the gradient variation texture detail information. Then, the fusion problem is cast into the optimization of minimum energy functional model. To facilitate choice of regularization parameter, a modified fixed-point iterative method is employed to solve the optimization. Extensive experiments show that this model can obtain both rich detail and spectral information; adjusting the parameters of the proposed adaptive selection strategy can effectively improve the fusion effect while making the image fusion process work as an end-to-end pipeline. Compared to eight state-of-the-art fusion methods, the experimental results demonstrate the effectiveness of the proposed method.

Ripplet domain fusion approach for CT and MR medical image information

Multimodal medical image fusion (MIF) plays an important role as an assistant for medical professionals by providing a better visualization of diagnostic information using different imaging modalities. The process of image fusion helps the radiologists in the precise diagnosis of several critical diseases and its treatment. In this paper, the proposed framework presents a fusion approach for multimodal medical images that utilize both the features extracted by the discrete ripplet transform (DRT) and pulse coupled neural network. The DRT having different features and a competent depiction of the image coefficients provides several directional high-frequency subband coefficients. The DRT decomposition can preserve more detailed information present in the reference images and further enhance the visualization of the fused images. Firstly, the DRT is applied to decompose the reference images into several low and high-frequency subimage coefficients that are fused by computing the novel sum modified Laplacian and novel modified spatial frequency motivated pulse coupled neural model. This model is used to preserve the redundant information also. Finally, fused images are reconstructed by applying the inverse DRT. The performance of the proposed fusion approach is validated by extensive simulation on the different CT-MR image datasets. Experimental results demonstrate that the proposed method provides the better fused images in terms of visual quality along with the quantitative measures as compared to several existing fusion approaches.

Multi-Focus Image Fusion in DCT Domain using Variance and Energy of Laplacian and Correlation Coefficient for Visual Sensor Networks

The purpose of multi-focus image fusion is gathering the essential information and the focused parts from the input multi-focus images into a single image. These multi-focus images are captured with different depths of focus of cameras. A lot of multi-focus image fusion techniques have been introduced using considering the focus measurement in the spatial domain. However, the multi-focus image fusion processing is very time-saving and appropriate in discrete cosine transform (DCT) domain, especially when JPEG images are used in visual sensor networks (VSN). So the most of the researchers are interested in focus measurements calculation and fusion processes directly in DCT domain. Accordingly, many researchers developed some techniques which are substituting the spatial domain fusion process with DCT domain fusion process. Previous works in DCT domain have some shortcomings in selection of suitable divided blocks according to their criterion for focus measurement. In this paper, calculation of two powerful focus measurements, energy of Laplacian (EOL) and variance of Laplacian (VOL), are proposed directly in DCT domain. In addition, two other new focus measurements which work by measuring correlation coefficient between source blocks and artificial blurred blocks are developed completely in DCT domain. However, a new consistency verification method is introduced as a post-processing, improving the quality of fused image significantly. These proposed methods reduce the drawbacks significantly due to unsuitable block selection. The output images quality of our proposed methods is demonstrated by comparing the results of proposed algorithms with the previous algorithms.

RETRACTED: Multi-spectral and panchromatic image fusion approach using stationary wavelet transform and swarm flower pollination optimization for remote sensing applications

Abstract This paper proposes a multi-spectral (MS) and panchromatic (Pan) image fusion approach based on the flower pollination algorithm optimization (FPA). The FPA is used to get an optimal fused image. The image fusion quality depends on the choice of the weight of fusion rule. The proposed approach uses FPA to optimize the weights of a fusion rule to make a perfect image fusion process. FPA is a nature-inspired algorithm, based on the characteristics of a flower pollination process. FPA averts trapping in local optimal solution. In this paper, the remote sensing image fusion based on flower pollination algorithm is compared to several states of the art image fusion approaches including Intensity-hue-saturation (IHS) image fusion; stationary wavelets transform image fusion based on the average weight fusion rule (SWT-AW) and the image fusion based on the particle swarm optimization (PSO). The experimental results used MODIS satellite series with spatial resolutions 250 m, 500 m, and 1 km, which are low spatial resolution and multispectral images; and Pan image of SPOT satellite is high spatial resolution 10 m to produce synthetic imagery at SPOT spatial resolutions and MODIS multispectral resolution at the same time. The experimental results prove that the proposed remote sensing image fusion approach can illustrate a better performance than the other approaches. The experimental results show that the approach offers up to 20% enhancement in Peak Signal to Noise Ratio (PSNR), 1% enhancement in Structural Similarity Index (SSIM), 1% and 0.5% enhancement in entropy information (EI) than best existing particle swarm optimization (PSO) approach. The results indicate that the proposed approach outperforms over existing approaches.

Design and Development of IP for Modified Haar Wavelet Transform (MHWT) Image Fusion using FPGA

<p>The fast growth in the field of digital imaging applications in remote sensing, bio medical and other satellite applications had created an architecture studies for image fusion in capable to store large amount of data and process. An algorithm considered for the process of image fusion for implementation of FPGA is Modified Haar Wavelet Transform (MHWT) based image fusion where at the time four pixels are consider in calculation of different bands as compared to conventional Haar wavelet based image fusion. The process of modification uses far less memory and computation power. The FPGA implementation of MHWT based image fusion is done on Digilent development board with Spartan 6 series FPGA. The architecture is developed in VHDL. The timing analysis is done and report is obtained for I/O interactions, memory units etc. The architecture is made to run in cosimulation with Simulink. The design is tested with different kinds of images and run successfully. The visual analysis of the resultant fused image is achived and observed.</p>

Weighted fusion algorithm for pixel-level multiresolution images

Weighted fusion for pixel-level multiresolution images improves the image quality. The commonly used algorithm is based on wavelet transform, but the pixel quality after fusion is poor, and the fusion process is complicated. Therefore, a weighted fusion algorithm for pixel-level multiresolution images is put forward based on chaotic neural network. Firstly, pixel-level multiresolution image fusion is analyzed, and the image is preprocessed and matched. Then through the dynamic search feature of chaotic neural network to find the global optimal solution, so as to realize the weighted fusion. Experiments show that the algorithm proposed for image fusion results in image of high resolution, and the image fusion process is simple.

Medical image fusion using type-2 fuzzy and near-fuzzy set approach

Image fusion process combines the information from multiple images of one view, to acquire an informative single image which is required for computer vision applications. In case brain imaging, the MRI images illustrate the tissue anatomy and contains no functional information and PET images illustrate the brain functional information with low spatial resolution contents. The aim of image fusion is to enhance the spatial resolution of the functional images by combining them with a high resolution anatomic image. It is desirable that the fusion process conserve the original color information of PET image and the spatial characteristics of MRI image as much as possible. Unlike most previous works on multisensor image fusion, we propose an image fusion method using type-2 fuzzy and near-fuzzy set model which approximately estimate fuzzy behavior of medical images using fuzzy entropy and fuzzy correlation operators. The proposed method is tested on several categories of benchmark medical images and compared with some well-liked image fusion methods. The experimental results point out that the proposed framework can generate excellent fused image in terms of several quantitative fusion evaluation indexes, such as entropy, mutual information, average gradient and edge strength.

A fusion approach based on infrared finger vein transmitting model by using multi-light-intensity imaging

An infrared transmitting model from the observed finger vein images is proposed in this paper by using multi-light-intensity imaging. This model is estimated from many pixels’ values under different intensity light in the same scene. Due to the fusion method could be applied in biometric system, the vein images of finger captured in the system, we proposed in this paper, will be normalized and preserved the intact of the vein patterns of the biometric data from tested human’s body. From observed pixels under multi-light-intensity, the curve of the transmitting model is recovered by sliding both of the sampled curve segments and using curve-fitting. The fusion method with each pixel level weighting based on the proposed transmitting model curve is adopted by the smooth spatial and estimation of the block quality. Finally, the results shown that our approach is a convenient and practicable approach for the infrared image fusion and subsequent processing for biometric applications.

Single-Plane Dual-Modality Tomography for Multiphase Flow Imaging by Integrating Electrical Capacitance and Ultrasonic Sensors

This paper presents an experimental study of a single-plane dual-modality tomography (DMT) system integrating electrical capacitance tomography (ECT) and ultrasonic tomography (UT) for water–oil–gas three-phase flow imaging. The soft-field (ECT) and hard-field (UT) systems measure different physical parameters, making them both a fine combination in complementing each system’s limitations. The DMT system consists of 16 units of customized ECT sensors and 16 units of UT transducers, which are composited on a single-plane arrangement to perform concurrent measurement. A finite-element method is used to model and study its feasibility and performance before engaging with the actual hardware. The DMT system is carried out on a vertical test column and horizontal stratified flow to obtain sensor data from each modality by inspection of separate tomographic image using linear back-projection (LBP) image reconstruction algorithm and convolution back projection with Gaussian filtering (CBPF). A fuzzy logic pixel fusion method is used for image fusion process. To evaluate the reconstructed images, the mean structural similarity index (MSSIM) method and the mutual information method ( $M_{F})$ are employed on both the LBP and CBPF image results. The analysis and assessments proved that the CBPF method has successfully improvised the image results and thus has better MSSIM rating and $M_{F}$ over the LBP method with an enhancement up to 12.55% and 8.92%, respectively.

Design and measurement of user experience oriented auto-stereoscopic 3D display based on optical lens

In order address the issues of crosstalk, poor user experiences, and difficult assessment of image quality in auto-stereoscopic 3D displays, we propose an easy-to-implement measurement method called fault-width-assessment for auto-stereoscopic 3D applications. Based on this method, we perform a study on the influence of camera group setups and multi-view image arrangements during the image fusion process. For scenes with multiple users and a non-fixed view, we propose various schemes for camera setups and multi-view image arrangement, which contribute to improving the user experience. Finally, cross-experimental comparisons and applications of the user experience tests are implemented, which are ultimately helpful for choosing a reasonable proposal.

PET and MRI image fusion based on combination of 2-D Hilbert transform and IHS method

BackgroundThe process of medical image fusion is combining two or more medical images such as Magnetic Resonance Image (MRI) and Positron Emission Tomography (PET) and mapping them to a single image as fused image. So purpose of our study is assisting physicians to diagnose and treat the diseases in the least of the time. MethodsWe used Magnetic Resonance Image (MRI) and Positron Emission Tomography (PET) as input images, so fused them based on combination of two dimensional Hilbert transform (2-D HT) and Intensity Hue Saturation (IHS) method. Evaluation metrics that we apply are Discrepancy (Dk) as an assessing spectral features and Average Gradient (AGk) as an evaluating spatial features and also Overall Performance (O.P) to verify properly of the proposed method. ResultsIn this paper we used three common evaluation metrics like Average Gradient (AGk) and the lowest Discrepancy (Dk) and Overall Performance (O.P) to evaluate the performance of our method. Simulated and numerical results represent the desired performance of proposed method. ConclusionsSince that the main purpose of medical image fusion is preserving both spatial and spectral features of input images, so based on numerical results of evaluation metrics such as Average Gradient (AGk), Discrepancy (Dk) and Overall Performance (O.P) and also desired simulated results, it can be concluded that our proposed method can preserve both spatial and spectral features of input images.