Image Fusion Framework Research Articles

OPTICAL AND SAR IMAGE FUSION BASED ON VISUAL SALIENCY FEATURES

Abstract. With the expansion of optical and SAR image fusion application scenarios, it is necessary to integrate their information in land classification, feature recognition, and target tracking. Current methods focus excessively on integrating multimodal feature information to enhance the information richness of the fused images, while neglecting the highly corrupted visual perception of the fused results by modal differences and SAR speckle noise. To address this problem, in this paper we propose a novel optical and SAR image fusion framework named Visual Saliency Features Fusion (VSFF). We improved the decomposition algorithm of complementary feature to reduce most of the speckle noise in the initial features, and divide the image into main structure features and detail texture features. For the fusion of main structure features, we reconstruct a visual saliency features map that contains significant information from optical and SAR images, and input it together with the optical image into a total variation constraint model to compute the fusion result and achieve the optimal information transfer. Meanwhile, we construct a new feature descriptor based on Gabor wavelet, which separates meaningful detail texture features from residual noise and selectively preserves features that can improve the interpretability of fusion result. Further a fast IHS transform fusion is used to supplement the fused image with realistic color information. In a comparative analysis with five state-of-the-art fusion algorithms, VSFF achieved better results in qualitative and quantitative evaluations, and our fused images have a clear and appropriate visual perception.

Thermal-textured BIM generation for building energy audit with UAV image fusion and histogram-based enhancement

Mapping Unmanned Aerial Vehicle (UAV)-based thermal images of building façades onto a Building Information Model (BIM) can greatly support building energy audit. However, accurately registering these images into BIM is challenging due to thermal image’s unique characteristics of low texture and high distortion. To address these issues, this paper proposes a new multi-source image fusion framework for registering UAV thermal images to BIM. First, a low-cost target is designed and produced using 3D printing for calibrating UAV thermal camera. The rectified thermal images are then fused with RGB images through Homography transformation, enabling thermal images to be registered accurately with BIM. Additionally, a histogram-based approach is proposed to correct the thermally inconsistent phenomenon and enhance the contrast to produce high-contrast thermal-textured BIM. Field tests confirmed a mean image fusion error of around 5.713 pixels. A case study on a masonry building demonstrated the feasibility and effectiveness of the proposed framework.

Multi-scale infrared and visible image fusion framework based on dual partial differential equations

Image fusion aims to integrate multiple images that contain complementary information into a single image using an appropriate strategy. The fusion of infrared (IR) and visible (VIS) images can effectively integrate thermal radiation and texture information, which fuses foreground and background information into a fused image. An effective image fusion framework should transfer most of the information from the IR and VIS images to the fused image while minimizing artifacts. This paper introduces a novel multi-scale decomposition framework named Dual PDEs, which decomposes the source images into multiple layers using two different types of partial differential equations (PDEs) and fuses the feature representations at each scale layer separately with an adaptive fusion scheme. The target image is obtained from the multi-scale fused layers. Qualitative and quantitative evaluations show that the proposed framework effectively highlights the prominent areas with fewer artifacts, more explicit boundaries, and better visual effects, which outperforms existing fusion methods.

A self-supervised remote sensing image fusion framework with dual-stage self-learning and spectral super-resolution injection

Pan-sharpening is a very productive technique to enhance the spatial details of multispectral images with the aid of panchromatic images. Nowadays, deep learning-based pan-sharpening has scored tremendous achievements. However, strict requirement for training image pairs and low generalization hamper the development of supervised pan-sharpening with limited samples. Unsupervised image fusion is an effective technique to fuse images without adequate ground truth as training samples. Existing unsupervised pan-sharpening methods are commonly based on the image generator model, suffering from unsatisfactory spatial details. In this study, we proposed a self-supervised pan-sharpening method with the aid of spectral super-resolution named sSRPNet. Coarsening-scale self-learning exploits the internal information in multispectral images at a coarsening scale and trains the initial fusion model without other labels. Spectral super-resolution injection explores the missing spatial details in the initial fused images and recovers it. Degradation self-learning introduces strong spectral constraints with original multispectral images. Both reduced-resolution and full-resolution experiments on three datasets have proved the superiority of sSRPNet. Furthermore, the proposed spectral super-resolution injection can be implemented on any existing pan-sharpening algorithms, improving their performance.

EasyFuse: Easy-to-learn visible and infrared image fusion framework based on unpaired set

The major hurdle in building a visible and infrared fusion model is the necessity of large-scale pixel-aligned and time-synchronized image pairs. In this paper, we propose an easy-to-learn visible and infrared image fusion framework that does not require pairs for training. In addition to easy training using unpaired sets, our framework provides fusion images with more textures and meaningful scene information compared to previous works. In particular, to mix features from each spectrum, we newly present a feature line-up module to identify important information in each source. Additionally, in order to provide a new option for benchmark evaluation, we construct a new sequence-based visible and infrared paired dataset that is aligned and synchronized. Finally, we perform extensive experiments to verify the performance of the proposed method by both public and proposed datasets.

Robust multi-focus image fusion using focus property detection and deep image matting

Multi-focus image fusion techniques are designed to integrate the focused regions of multiple source images to produce an all-in-focus image. Unfortunately, the fusion results generated by existing methods are prone to suffer from chromatic aberrations and blurred boundary regions. In this paper, a multi-focus image fusion framework combining focus property detection and deep image matting is proposed to achieve high-quality fusion results. Specifically, we introduce spatial frequency detection blocks, which integrate the spatial frequency information of source images at different levels to achieve effective focus property detection for the initial decision. Then, the trimap acquired from the initial decision is optimized using a deep matting model so as to further detect the focused/defocused boundaries and obtain a more precise decision map. Meanwhile, a new hybrid loss function is introduced, and a large-scale dataset is created to improve the performance of the fusion framework. Finally, the fusion result is obtained by integrating the focused regions via the decision map. Compared to 18 other advanced fusion methods, the proposed method outperforms all of them in terms of the seven objective metrics evaluated on two publicly available datasets. These experimental results provide strong evidence that the proposed fusion framework excels not only in subjective vision but also in quantitative analysis. The code of the proposed method will be available at https://github.com/govenda/DIMF.

A novel endoscopy image fusion system: combine white light imaging and compound band imaging.

White light imaging (WLI) is a commonly seen examination mode in endoscopy. The particular light in compound band imaging (CBI) can highlight delicate structures, such as capillaries and tiny structures on the mucosal surface. These two modes complement each other, and doctors switch between them manually to complete the examination. This paper proposes an endoscopy image fusion system to combine WLI and CBI. We add a real-time rotatable color wheel in the light source device of the AQ-200 endoscopy system to achieve rapid imaging of two modes at the same position of living tissue. The two images corresponding to the pixel level can avoid registration and lay the foundation for image fusion. We propose a multi-scale image fusion framework, which involves Laplacian pyramid (LP) and convolutional sparse representation (CSR) and strengthens the details in the fusion rule. Volunteer experiments and ex vivo pig stomach trials are conducted to verify the feasibility of our proposed system. We also conduct comparative experiments with other image fusion methods, evaluate the quality of the fused images, and verify the effectiveness of our fusion framework. The results show that our fused image has rich details, high color contrast, apparent structures, and clear lesion boundaries. An endoscopy image fusion system is proposed, which does not change the doctor's operation and makes the fusion of WLI and CBI optical staining technology a reality. We change the light source device of the endoscope, propose an image fusion framework, and verify the feasibility and effectiveness of our scheme. Our method fully integrates the advantages of WLI and CBI, which can help doctors make more accurate judgments than before. The endoscopy image fusion system is of great significance for improving the detection rate of early lesions and has broad application prospects.

A DCT-based multiscale framework for 2D greyscale image fusion using morphological differential features

A DCT-based multiscale framework for 2D greyscale image fusion using morphological differential features

A novel framework of multimodal medical image fusion using adaptive NSST and optimized deep learning approach

ABSTRACT Multimodal medical image fusion plays a pivotal role in the medical and imaging industry. Existing works of deep learning method suffers from blurred texture characteristics and computing efficiency. Thus, a novel deep learning model is proposed for multimodal medical image fusion. Initially, an Adaptive Non-Subsampled Shearlet Transform (ANSST) approach is developed for decomposing the images, where the filter coefficient is optimized by Hybrid Water Strider-Dingo Optimization (HWS-DOX). The fusion of the high sub-bands of source image 1 and high sub-bands of source image 2 is done by an Optimized Deep Neural Network (ODNN). Then, the lower sub-bands of source image 1 and lower sub-bands of source image 2 will be fused through a weighted averaging scheme. Finally, the final fused images are attained by applying the inverse ANSST. Thus, the performance of recommended module is compared over classical heuristics and various transform methods.

CNN framework for optical image super-resolution and fusion

CNN framework for optical image super-resolution and fusion

Intelligent Diagnosis of Rolling Bearings Fault Based on Multisignal Fusion and MTF-ResNet.

Existing diagnosis methods for bearing faults often neglect the temporal correlation of signals, resulting in easy loss of crucial information. Moreover, these methods struggle to adapt to complex working conditions for bearing fault feature extraction. To address these issues, this paper proposes an intelligent diagnosis method for compound faults in metro traction motor bearings. This method combines multisignal fusion, Markov transition field (MTF), and an optimized deep residual network (ResNet) to enhance the accuracy and effectiveness of diagnosis in the presence of complex working conditions. At the outset, the acquired vibration and acoustic emission signals are encoded into two-dimensional color feature images with temporal relevance by Markov transition field. Subsequently, the image features are extracted and fused into a set of comprehensive feature images with the aid of the image fusion framework based on a convolutional neural network (IFCNN). Afterwards, samples representing different fault types are presented as inputs to the optimized ResNet model during the training phase. Through this process, the model's ability to achieve intelligent diagnosis of compound faults in variable working conditions is realized. The results of the experimental analysis verify that the proposed method can effectively extract comprehensive fault features while working in complex conditions, enhancing the efficiency of the detection process and achieving a high accuracy rate for the diagnosis of compound faults.

Real-Time Semantics-Driven Infrared and Visible Image Fusion Network.

This paper proposes a real-time semantics-driven infrared and visible image fusion framework (RSDFusion). A novel semantics-driven image fusion strategy is introduced in image fusion to maximize the retention of significant information of the source image in the fusion image. First, a semantically segmented image of the source image is obtained using a pre-trained semantic segmentation model. Second, masks of significant targets are obtained from the semantically segmented image, and these masks are used to separate the targets in the source and fusion images. Finally, the local semantic loss of the separation target is designed and combined with the overall structural similarity loss of the image to instruct the network to extract appropriate features to reconstruct the fusion image. Experimental results show that the RSDFusion proposed in this paper outperformed other comparative methods on both subjective and objective evaluation of public datasets and that the main target of the source image is better preserved in the fusion image.

Spectral- and spatial-based multi-focus image fusion method towards multi-layer nonwovens

In the microscopic imaging scenario where the object thickness exceeds the depth of field of the microscope, multi-focus image fusion (MFF) is an effective method to generate an all-in-focus image. However, for nonwoven fabric for which the captured image number is up to 100 or more, the existing methods often underperform in areas near the fiber edges, owing to image ghosting and noise accumulation caused by the platform moving. To address the above problem, this paper presents a method designed to fuse multi-layer micro-images based on the combination of spectral and spatial features of the images. Firstly, the spectral domain-based map is generated by decomposition and reconstruction of the high-frequency and low-frequency components of the images, aimed at obtaining the edge information. Simultaneously, the spatial domain-based fuse map is built through sharpness measurement, referring to visual perception. Finally, the two methods are combined via an optimized weight to obtain an all-in-focus fused image. Four groups of real-world data consisting of 100 multi-focus nonwoven images are utilized to verify the superiority of this method. The experimental results demonstrate that the proposed method can obtain satisfactory performance in terms of both human visual evaluation and objective evaluation compared with the image fusion framework based on the convolutional neural network, MFF, region-based image fusion algorithm and convolutional neural network state-of-the-art fusion methods.

CBFM: Contrast Balance Infrared and Visible Image Fusion Based on Contrast-Preserving Guided Filter

Infrared (IR) and visible image fusion is an important data fusion and image processing technique that can accurately and comprehensively integrate the thermal radiation and texture details of source images. However, existing methods neglect the high-contrast fusion problem, leading to suboptimal fusion performance when thermal radiation target information in IR images is replaced by high-contrast information in visible images. To address this limitation, we propose a contrast-balanced framework for IR and visible image fusion. Specifically, a novel contrast balance strategy is proposed to process visible images and reduce energy while allowing for detailed compensation of overexposed areas. Moreover, a contrast-preserving guided filter is proposed to decompose the image into energy-detail layers to reduce high contrast and filter information. To effectively extract the active information in the detail layer and the brightness information in the energy layer, we proposed a new weighted energy-of-Laplacian operator and a Gaussian distribution of the image entropy scheme to fuse the detail and energy layers, respectively. The fused result was obtained by adding the detail and energy layers. Extensive experimental results demonstrate that the proposed method can effectively reduce the high contrast and highlighted target information in an image while simultaneously preserving details. In addition, the proposed method exhibited superior performance compared to the state-of-the-art methods in both qualitative and quantitative assessments.

FusionJISI: A fusion algorithm based on infrared and visible images with joint involvement of source image

The available image fusion framework pays little attention to the importance of the joint involvement of source images in the whole fusion process. Due to its significance, an approach called FusionJISI combining infrared and visible image fusion algorithms with the joint involvement of source images was proposed. Fully decomposed texture features of each source image are realized to reshape the feature extraction process of source images by pre-fusing infrared and visible images, and applying the features of pre-fused images in the spatial domain. At the same time, to overcome the imaging differences caused by different wavelengths of infrared and visible light, a method to extract targeted infrared image saliency is designed to compensate for the differences between source images. Then, infrared and visible images are used as reference objects in the fusion process, and the extracted feature and base maps are constructed to be utilized in a feature similarity function that obtains the optimal solution of the function, which then makes the fusion process turn to an optimization problem and avoids the difficulty of manually designing complex fusion strategies. Apart from the available technology, the proposed method allows the source image to take a part in the whole fusion process. Experiments on the public dataset show that the fusion strategy can balance the texture gradient of infrared and visible images in the fusion results.

Multi-focus image fusion framework based on transformer and feedback mechanism

How to efficiently and accurately identify and extract the focused regions in the source image is a difficult problem in the field of multi-focus image fusion. The existing fusion methods suffer from color distortion, loss of detail information, and high time cost, which limit the subsequent processing and real-time application of fused images. Based on this, this paper proposes a multi-focus image fusion method based on Transformer and feedback mechanism. The method uses a combination of Transformer and convolutional neural network, and integrates the local information extracted by CNN and the global information obtained by transformer, which improves the accuracy of focus region identification. In addition, this paper uses a feedback mechanism to provide more contextual information so that the features can be fully utilized, which improves the performance of the network in feature fusion. This paper conducts comparison tests with seven advanced fusion methods on the Lytro and Grayscale datasets, and the results show that the algorithm in this paper is superior in both subjective and objective evaluations.

Deep polyp image enhancement using region of interest with paired supervision

Endoscopic medical imaging in complex curved intestinal structures are prone to uneven illumination, low contrast and lack of texture information. These problems may lead to diagnostic challenges. This paper described the first supervised deep learning based image fusion framework to enable the polyp region highlight through a global image enhancement and a local region of interest (ROI) with paired supervision. Firstly, we conducted a dual attention based network in global image enhancement. The Detail Attention Maps was used to preserve more image details and the Luminance Attention Maps was used to adjust the global illumination of the image. Secondly, we adopted the advanced polyp segmentation network ACSNet to obtain the accurate mask image of lesion region in local ROI acquisition. Finally, a new image fusion strategy was proposed to realize the local enhancement effect of polyp image. Experimental results show that our method can highlight the local details of the lesion area better and reach the optimal comprehensive performance with comparing with 16 traditional and state-of-the-art enhancement algorithms. And 8 doctors and 12 medical students were asked to evaluate our method for assisting clinical diagnosis and treatment effectively. Furthermore, the first paired image dataset LHI was constructed, which will be made available as an open source to research communities.

Infrared and visible image fusion based on NSST and phase consistency adaptive DUAL channel PCNN

The infrared and visible image fusion has been one of the significant research hotspots in image processing field. This paper presents a new IR and VI image fusion framework, which constructs a parameter adaptive dual channel pulse coupled neural network (PADCPCNN) with phase consistency in the non-subsampled shearlet transform (NSST) domain. Firstly, the original images are decomposed into low-pass and high-pass subbands by adopting NSST. Secondly, the weighted average fusion strategy based on phase consistency is applied to merge the low-pass subbands, and high-pass subbands are fused by the PADCPCNN with phase consistency. Additionally, in order to solve the information difference and edge detail problem of two source images, the phase consistency operator is exploited as the adaptive connection strength. Furthermore, four sets of controlled experiments are designed and fusion comparative experiments are conducted with images of typical datasets, from different backgrounds to make objective and subjective comparison. And their evaluation results show that the proposed algorithm highlights more texture and edge details than other existing main fusion methods.

AIM-MEF: Multi-exposure image fusion based on adaptive information mining in both spatial and frequency domains

Multi-Exposure Image Fusion (MEF) is a simple and effective solution to obtain high dynamic range images by integrating important information from a series of low dynamic range source images with different exposure levels. Despite promising results have been achieved, two issues remain unresolved in existing deep learning-based MEF methods. One is that the information in the frequency domain is underutilized, and the other is that the useful information of hard-to-learn pixels is not fully exploited. To address the above issues, we propose AIM-MEF: a multi-exposure image fusion framework based on adaptive information mining in both spatial and frequency domains. Concretely, we propose an image fusion loss in the frequency domain to mine the frequency domain information for the first time in the MEF field. In order to adaptively exploit the hard-to-learn but useful information in the source images, we propose a hard information mining strategy in both the spatial and the frequency domains, and perform information mining in pixel and frequency levels of the source images. In addition, we propose an adaptive fusion strategy based on information abundance to perform information mining at the image level. We compared AIM-MEF to 10 traditional and deep learning-based methods on two public datasets with eight objective metrics, and AIM-MEF outperformed these baselines in both subjective and objective evaluations. Codes will be publicly available.

BMIF: Privacy-preserving Blockchain-based Medical Image Fusion

Medical image fusion generates a fused image containing multiple features extracted from different source images, and it is of great help in clinical analysis and diagnosis. However, training a deep learning model for image fusion usually requires enormous computing power, especially for large volumes of medical data. Meanwhile, the privacy of images is also a critical issue. In this article, we propose a privacy-preserving blockchain-based medical image fusion (BMIF) framework. First, to ensure fusion performance, we design a new medical image fusion model based on convolutional neural network and Inception network and integrate the proposed model into the consensus process of blockchain. Next, to save computing power of blockchain, we design a consensus mechanism by requesting consensus nodes to train the fusion model instead of calculating useless hash values in traditional blockchain. Then, to protect data privacy, we further present an efficient homomorphic encryption to realize the training of fusion model on encrypted medical data. Finally, we conduct theoretical analysis and extensive experiments on public datasets to evaluate the feasibility and the performance of our proposed BMIF. The results exhibit that BMIF is efficient and secure, and our medical image fusion network performs better than state-of-the-art approaches.