Image Fusion Tasks Research Articles

TPP: Deep learning based threshold post-processing multi-focus image fusion method

How to accurately determine focused and defocused domains is a difficult problem in the field of multi-focused image fusion. To solve this problem, we propose a new deep learning method called threshold post processing(TPP). The TPP network model contains three architecturally different channels, and three different weighted maps can be obtained for the same input. The output of the model is fused and thresholded to obtain a final decision map, which is then used to complete the image fusion task. The decision map represents the focus condition of the source image at a pixel level. This allows TPP to retain focused regions of the source image more accurately than other existing deep learning methods. The multi-channel network model combined with the thresholding process optimizes the final decision map. It avoids the drawback of other methods that cause inaccurate discrimination of the focused area due to a single network channel and therefore cannot be corrected. Subsequent experimental results show that the images generated by TPP maintain a high level under the various image quality assessment metrics measures.

LPGAN: A LBP-Based Proportional Input Generative Adversarial Network for Image Fusion

Image fusion is the process of combining multiple input images from single or multiple imaging modalities into a fused image, which is expected to be more informative for human or machine perception as compared to any of the input images. In this paper, we propose a novel method based on deep learning for fusing infrared images and visible images, named the local binary pattern (LBP)-based proportional input generative adversarial network (LPGAN). In the image fusion task, the preservation of structural similarity and image gradient information is contradictory, and it is difficult for both to achieve good performance at the same time. To solve this problem, we innovatively introduce LBP into GANs, enabling the network to have stronger texture feature extraction and utilization capabilities, as well as anti-interference capabilities. In the feature extraction stage, we introduce a pseudo-Siamese network for the generator to extract the detailed features and the contrast features. At the same time, considering the characteristic distribution of different modal images, we propose a 1:4 scale input mode. Extensive experiments on the publicly available TNO dataset and CVC14 dataset show that the proposed method achieves the state-of-the-art performance. We also test the universality of LPGAN by fusing RGB and infrared images on the RoadScene dataset and medical images. In addition, LPGAN is applied to multi-spectral remote sensing image fusion. Both qualitative and quantitative experiments demonstrate that our LPGAN can not only achieve good structural similarity, but also retain richly detailed information.

BTMF-GAN: A multi-modal MRI fusion generative adversarial network for brain tumors

Image fusion techniques have been widely used for multi-modal medical image fusion tasks. Most existing methods aim to improve the overall quality of the fused image and do not focus on the more important textural details and contrast between the tissues of the lesion in the regions of interest (ROIs). This can lead to the distortion of important tumor ROIs information and thus limits the applicability of the fused images in clinical practice. To improve the fusion quality of ROIs relevant to medical implications, we propose a multi-modal MRI fusion generative adversarial network (BTMF-GAN) for the task of multi-modal MRI fusion of brain tumors. Unlike existing deep learning approaches which focus on improving the global quality of the fused image, the proposed BTMF-GAN aims to achieve a balance between tissue details and structural contrasts in brain tumor, which is the region of interest crucial to many medical applications. Specifically, we employ a generator with a U-shaped nested structure and residual U-blocks (RSU) to enhance multi-scale feature extraction. To enhance and recalibrate features of the encoder, the multi-perceptual field adaptive transformer feature enhancement module (MRF-ATFE) is used between the encoder and the decoder instead of a skip connection. To increase contrast between tumor tissues of the fused image, a mask-part block is introduced to fragment the source image and the fused image, based on which, we propose a novel salient loss function. Qualitative and quantitative analysis of the results on the public and clinical datasets demonstrate the superiority of the proposed approach to many other commonly used fusion methods.

Fine-grained multi-focus image fusion based on edge features

Multi-focus image fusion is a process of fusing multiple images of different focus areas into a total focus image, which has important application value. In view of the defects of the current fusion method in the detail information retention effect of the original image, a fusion architecture based on two stages is designed. In the training phase, combined with the polarized self-attention module and the DenseNet network structure, an encoder-decoder structure network is designed for image reconstruction tasks to enhance the original information retention ability of the model. In the fusion stage, combined with the encoded feature map, a fusion strategy based on edge feature map is designed for image fusion tasks to enhance the attention ability of detail information in the fusion process. Compared with 9 classical fusion algorithms, the proposed algorithm has achieved advanced fusion performance in both subjective and objective evaluations, and the fused image has better information retention effect on the original image.

DAGM-fusion: A dual-path CT-MRI image fusion model based multi-axial gated MLP

Medical imaging technology provides a good understanding of human tissue structure. MRI provides high-resolution soft tissue information, and CT provides high-quality bone density information. By creating CT-MRI fusion images of complex diagnostic situations, experts can develop diagnoses and treatment plans more quickly and precisely. We propose a dual-path CT-MRI image fusion model based on multi-axial gated MLP to create high-quality CT-MRI fusion images. The model employs the feature fusion module SFT-block to effectively integrate detailed Local-Path information guided by global Global-Path information. The fusion is completed through triple constraints, namely global constraints, local constraints, and overall constraints. We design a multi-axial gated MLP module (Ag-MLP). The multi-axial structure maintains the computational complexity linear and increases MLP's inductive bias, allowing MLP to work in shallower or pixel-level small dataset tasks. Ag-MLP and CNN are combined in the network so that the model has both globality and locality. In addition, we design a loss calculation method based on image patches that adaptively generates weights for each patch based on image pixel intensity. The details of the image are efficiently increased when patch-loss is used. Numerous studies demonstrate that the results of our model are superior to those of the latest mainstream fusion model, which are more in accordance with actual clinical diagnostic standards. The ablation studies successfully validate the performance of the model's constituent parts. It is worth mentioning that the model can also be excellently generalized to other modal image fusion tasks.

IFSepR: A General Framework for Image Fusion Based on Separate Representation Learning

This paper proposes an image fusion framework based on separate representation learning, called IFSepR. We believe that both the co-modal image and the multi-modal image have common and private features based on prior knowledge, exploiting this disentangled representation can help to image fusion, especially to fusion rule design. Inspired by the autoencoder network and contrastive learning, a multi-branch encoder with contrastive constraints is built to learn the common and private features of paired images. In the fusion stage, based on the disentangled features, a general fusion rule is designed to integrate the private features, then combining the fused private features and the common feature are fed into the decoder, reconstructing the fused image. We perform a series of evaluations on three typical image fusion tasks, including multi-focus image fusion, infrared and visible image fusion, medical image fusion. Quantitative and qualitative comparison with five state-of-art image fusion methods demonstrates the advantages of our proposed model.

PSMFF: A progressive series-parallel modality feature filtering framework for infrared and visible image fusion

Current researches in the infrared and visible image fusion task tend to focus on designing decomposition or fusion schemes for improving the feature preservation performance. However, raw images captured by multi-sensors from extreme conditions may suffer from noise, low contrast and loss of textural and structural details. These influences are difficult to be eliminated by most of the current fusion schemes. In this paper, a robust progressive series-parallel modality feature filtering framework (PSMFF) is proposed to provide a promising solution for 24/7 infrared and visible image integration. After fully considering modality-specific characteristics of infrared and visible images, the first-stage enhancement module is constructed as a dual solution system that combines the pixel-level attention-aware module and fidelity-driven intensity enhancement to overcome the noise and low contrast of source images. Moreover, to further excavate and integrate the latent inherent information of each modality, the second-stage enhancement and fusion Scheme based on the gradient-domain guide filter is proposed to realize a better extraction and aggregation of significant features from the decomposed base and detail coefficients. Notably, the proposed PSMFF method can fulfill the low-light fusion task and preserves more informative edge and texture details to obtain a satisfactory fusion result. Extensive experiments demonstrate that the proposed algorithm is superior to the state-of-the-art methods in the performance of subjective and objective evaluation.

MUFusion: A general unsupervised image fusion network based on memory unit

Existing image fusion approaches are committed to using a single deep network to solve different image fusion problems, achieving promising performance in recent years. However, devoid of the ground-truth output, in these methods, only the appearance from source images can be exploited during the training process to generate the fused images, resulting in suboptimal solutions. To this end, we advocate a self-evolutionary training formula by introducing a novel memory unit architecture (MUFusion). In this unit, specifically, we utilize the intermediate fusion results obtained during the training process to further collaboratively supervise the fused image. In this way, our fusion results can not only learn from the original input images, but also benefit from the intermediate output of the network itself. Furthermore, an adaptive unified loss function is designed based on this memory unit, which is composed of two loss items, i.e., content loss and memory loss. In particular, the content loss is calculated based on the activity level maps of source images, which can constrain the output image to contain specific information. On the other hand, the memory loss is obtained based on the previous output of our model, which is utilized to force the network to yield fusion results with higher quality. Considering the handcrafted activity level maps cannot consistently reflect the accurate salience judgement, we put two adaptive weight items between them to prevent this degradation phenomenon. In general, our MUFusion can effectively handle a series of image fusion tasks, including infrared and visible image fusion, multi-focus image fusion, multi-exposure image fusion, and medical image fusion. Particularly, the source images are concatenated in the channel dimension. After that, a densely connected feature extraction network with two scales is used to extract the deep features of the source images. Following this, the fusion result is obtained by two feature reconstruction blocks with skip connections from the feature extraction network. Qualitative and quantitative experiments on 4 image fusion subtasks demonstrate the superiority of our MUFusion, compared to the state-of-the-art methods.

Designing CNNs for Multimodal Image Restoration and Fusion via Unfolding the Method of Multipliers

Multimodal, alias, guided, image restoration is the reconstruction of a degraded image from a target modality with the aid of a high quality image from another modality. A similar task is image fusion; it refers to merging images from different modalities into a composite image. Traditional approaches for multimodal image restoration and fusion include analytical methods that are computationally expensive at inference time. Recently developed deep learning methods have shown a great performance at a reduced computational cost; however, since these methods do not incorporate prior knowledge about the problem at hand, they result in a “black box” model, that is, one can hardly say what the model has learned. In this paper, we formulate multimodal image restoration and fusion as a coupled convolutional sparse coding problem, and adopt the Method of Multipliers (MM) for its solution. Then, we use the MM-based solution to design a convolutional neural network (CNN) encoder that follows the principle of deep unfolding. To address multimodal image restoration and fusion, we design two multimodal models which employ the proposed encoder followed by an appropriately designed decoder that maps the learned representations to the desired output. Unlike most existing deep learning designs comprising multiple encoding branches followed by a concatenation or a linear combination fusion block, the proposed design provides an efficient and structured way to fuse information at different stages of the network, providing representations that can lead to accurate image reconstruction. The proposed models are applied to three image restoration tasks, as well as two image fusion tasks. Quantitative and qualitative comparisons against various state-of-the-art analytical and deep learning methods corroborate the superior performance of the proposed framework.

Unified gradient- and intensity-discriminator generative adversarial network for image fusion

This study proposes a unified gradient- and intensity-discriminator generative adversarial network for various image fusion tasks, including infrared and visible image fusion, medical image fusion, multi-focus image fusion, and multi-exposure image fusion. On the one hand, we unify all fusion tasks into discriminating a fused image’s gradient and intensity distributions based on a generative adversarial network. The generator adopts a dual-encoder–single-decoder framework to extract source image features by using different encoder paths. A dual-discriminator is employed to distinguish the gradient and intensity, ensuring that the generated image contains the desired geometric structure and conspicuous information. The dual adversarial game can tackle the generative adversarial network’s mode collapse problem. On the other hand, we define a loss function based on the gradient and intensity that can be adapted to various fusion tasks by using varying relevant parameters with the source images. Qualitative and quantitative experiments on publicly available datasets demonstrate our method’s superiority over state-of-the-art methods.

UIFGAN: An unsupervised continual-learning generative adversarial network for unified image fusion

In this paper, we propose a novel unsupervised continual-learning generative adversarial network for unified image fusion, termed as UIFGAN. In our model, for multiple image fusion tasks, a generative adversarial network for training a single model with memory in a continual-learning manner is proposed, rather than training an individual model for each fusion task or jointly training multiple tasks. We use elastic weight consolidation to avoid forgetting what has been learned from previous tasks when training multiple tasks sequentially. In each task, the generation of the fused image comes from the adversarial learning between a generator and a discriminator. Meanwhile, a max-gradient loss function is adopted for forcing the fused image to obtain richer texture details of the corresponding regions in two source images, which applies to most typical image fusion tasks. Extensive experiments on multi-exposure, multi-modal and multi-focus image fusion tasks demonstrate the advantages of our method over the state-of-the-art approaches.

DRSNFuse: Deep Residual Shrinkage Network for Infrared and Visible Image Fusion.

Infrared images are robust against illumination variation and disguises, containing the sharp edge contours of objects. Visible images are enriched with texture details. Infrared and visible image fusion seeks to obtain high-quality images, keeping the advantages of source images. This paper proposes an object-aware image fusion method based on a deep residual shrinkage network, termed as DRSNFuse. DRSNFuse exploits residual shrinkage blocks for image fusion and introduces a deeper network in infrared and visible image fusion tasks than existing methods based on fully convolutional networks. The deeper network can effectively extract semantic information, while the residual shrinkage blocks maintain the texture information throughout the whole network. The residual shrinkage blocks adapt a channel-wise attention mechanism to the fusion task, enabling feature map channels to focus on objects and backgrounds separately. A novel image fusion loss function is proposed to obtain better fusion performance and suppress artifacts. DRSNFuse trained with the proposed loss function can generate fused images with fewer artifacts and more original textures, which also satisfy the human visual system. Experiments show that our method has better fusion results than mainstream methods through quantitative comparison and obtains fused images with brighter targets, sharper edge contours, richer details, and fewer artifacts.

Image Fusion with Sparse Representation: A Novel Local Contrast-Based Preprocessing Strategy

In the field of image fusion, great progress has been achieved in sparse representation (SR)-based methods with the advent of considerable efficacious algorithms being developed. The improvement of SR-based image fusion methods mainly focuses on pursuing better signal representation as well as effective measurement to sparse coefficients. However, the preprocessing strategy, which plays an important role in SR-based fusion methods, is rarely investigated. The traditional preprocessing strategies such as the zero-mean normalized strategy result in the loss of low-frequency information and insufficient detail extraction. To address the above issues, we propose a local contrast (LC)-based preprocessing strategy, since local contrast can be harnessed to capture geometrical visual patterns as well as preserve comprehensive information. Experimental results on multifocus and multimodal image fusion tasks demonstrate that the proposed preprocessing strategy outperforms the traditional ones in terms of both visual quality and objective evaluation.

Multimodal super-resolution reconstruction of infrared and visible images via deep learning

In this paper, we propose a deep-learning-based infrared-visible images fusion method based on encoder-decoder architecture. The image fusion task is reformulated as a problem of maintaining the structure and intensity ratio of the infrared-visible image. The corresponding loss function is designed to expand the weight difference between the thermal target and the background. In addition, a single image super-resolution reconstruction based on a regression network is introduced to address the issue that traditional network mapping functions are not suitable for natural scenes. The forward generation and reverse regression models are considered to reduce the irrelevant function mapping space and approach the ideal scene data through double mapping constraints. Compared with other state-of-the-art approaches, our experimental results achieve superior performance in terms of both visual effects and objective assessments. In addition, it can stably provide high-resolution reconstruction results consistent with human visual observation while bridging the resolution gap between the infrared-visible images.

General Image Fusion for an Arbitrary Number of Inputs Using Convolutional Neural Networks.

In this paper, we propose a unified and flexible framework for general image fusion tasks, including multi-exposure image fusion, multi-focus image fusion, infrared/visible image fusion, and multi-modality medical image fusion. Unlike other deep learning-based image fusion methods applied to a fixed number of input sources (normally two inputs), the proposed framework can simultaneously handle an arbitrary number of inputs. Specifically, we use the symmetrical function (e.g., Max-pooling) to extract the most significant features from all the input images, which are then fused with the respective features from each input source. This symmetry function enables permutation-invariance of the network, which means the network can successfully extract and fuse the saliency features of each image without needing to remember the input order of the inputs. The property of permutation-invariance also brings convenience for the network during inference with unfixed inputs. To handle multiple image fusion tasks with one unified framework, we adopt continual learning based on Elastic Weight Consolidation (EWC) for different fusion tasks. Subjective and objective experiments on several public datasets demonstrate that the proposed method outperforms state-of-the-art methods on multiple image fusion tasks.

Stepwise Fusion of Hyperspectral, Multispectral and Panchromatic Images with Spectral Grouping Strategy: A Comparative Study Using GF5 and GF1 Images

Since hyperspectral satellite images (HSIs) usually hold low spatial resolution, improving the spatial resolution of hyperspectral imaging (HSI) is an effective solution to explore its potential for remote sensing applications, such as land cover mapping over urban and coastal areas. The fusion of HSIs with high spatial resolution multispectral images (MSIs) and panchromatic (PAN) images could be a solution. To address the challenging work of fusing HSIs, MSIs and PAN images, a novel easy-to-implement stepwise fusion approach was proposed in this study. The fusion of HSIs and MSIs was decomposed into a set of simple image fusion tasks through spectral grouping strategy. HSI, MSI and PAN images were fused step by step using existing image fusion algorithms. According to different fusion order, two strategies ((HSI+MSI)+PAN and HSI+(MSI+PAN)) were proposed. Using simulated and real Gaofen-5 (GF-5) HSI, MSI and PAN images from the Gaofen-1 (GF-1) PMS sensor as experimental data, we compared the proposed stepwise fusion strategies with the traditional fusion strategy (HSI+PAN), and compared the performances of six fusion algorithms under three fusion strategies. We comprehensively evaluated the fused results through three aspects: spectral fidelity, spatial fidelity and computation efficiency evaluation. The results showed that (1) the spectral fidelity of the fused images obtained by stepwise fusion strategies was better than that of the traditional strategy; (2) the proposed stepwise strategies performed better or comparable spatial fidelity than traditional strategy; (3) the stepwise strategy did not significantly increase the time complexity compared to the traditional strategy; and (4) we also provide suggestions for selecting image fusion algorithms using the proposed strategy. The study provided us with a reference for the selection of fusion strategies and algorithms in different application scenarios, and also provided an easy-to-implement solution and useful references for fusing HSI, MSI and PAN images.

Convolution neural network with edge structure loss for spatiotemporal remote sensing image fusion

ABSTRACT Spatiotemporal fusion technology provides a feasible, economical solution for generating remote sensing images with high spatiotemporal resolution. The recently proposed learning-based method achieved high accuracy; however, its network structure is relatively simple, and the deep features of the input image cannot be obtained, so that the fused image cannot restore good landform details and the quality is not very good. Moreover, most methods use a single pixel-level (MSE) loss, which makes recovering high-frequency details difficult, resulting in a reduction in the fusion accuracy. In this paper, we propose an edge structure loss, which is added to a spatiotemporal fusion network without pre training model. To fully extract the spectral information and spatial details of the image, we propose a DenseNet-BC module for image fusion tasks, which makes the features more easily transmittable in the whole network. This improvement also enables the network to perform spatiotemporal fusion with better generalizability and robustness. In addition, we propose an edge loss to further improve the accuracy of the model fusion results. Experiments with existing spatiotemporal fusion algorithms in different regions show that our proposed method is more fault tolerant and achieve a higher accuracy in terms of quality evaluation indicators and better visual effects.

Significant target analysis and detail preserving based infrared and visible image fusion

Infrared and visible image fusion aims to integrate the prominent infrared target and visible texture details as much as possible. However, infrared images are susceptible to noise pollution during the transmission process, which could reduce the quality of the fused image. To solve this problem, a novel significant target analysis and detail preserving based scheme is proposed for infrared and visible image fusion tasks. Since the infrared image has high visual saliency, the noise pixels in the infrared image are often retained into the fused result by most traditional fusion methods. To better highlight the infrared targets while reducing interference from noise pixels, an infrared target detection model is proposed based on boundary-aware salient target detection network (BASNet) and significant target analysis. To fully preserve the visual details, we employ a weight mean curvature (WMC) based multiscale transform (MST) fusion scheme which can effectively suppress noise and preserve valuable details. Difference of Gaussian (DoG) is also applied to enhance the overall details in the proposed fusion scheme. Qualitative and quantitative experimental results demonstrate that the proposed method can generate fused images with abundant texture details and prominent infrared targets, which is superior to some of the existing methods in visual effect and objective quality evaluations.

MMI-Fuse: Multimodal Brain Image Fusion With Multiattention Module

Medical imaging plays a pivotal role in the clinical diagnosis of brain disease. There are many imaging methods to detect the state of tissues in the brain. While these imaging methods have advantages, they also have shortcomings. For example, magnetic resonance imaging (MRI) contains structural information but no functional characteristics of tissue, while positron emission tomography (PET) possesses functional characteristics but no structural information. The attention mechanism has been widely used in image fusion tasks, such as fusion of infrared and visible images and medical images. However, those attention models lack a balance mechanism for multimodal image features, affecting the final fusion performance. This paper proposes an end-to-end multimodal brain image fusion framework, MMI-fuse. Specifically, we first apply an autoencoder to extract the features of source images. Then, an information preservation weighted channel spatial attention model (ICS) is proposed to fuse the image features. We set an adaptive weight according to the information preservation degree of features. Finally, we use a decoder model to restructure the fused medical image. The proposed method increased the quality of fused images and decreased the fusion time effectively by the help of the improved attention model and encoder-decoder structure. To validate the performance of the proposed method, we collected 1590 pairs of multimodal brain images from the Harvard dataset and performed extensive experiments. Seven methods and five metrics were selected for the comparison experiments. The results demonstrate that the proposed method achieved notable performance on both the visual quality and objective metric score among these seven approaches. Moreover, the proposed method takes the least time among all compared methods.

A Spatial–Spectral Dual-Optimization Model-Driven Deep Network for Hyperspectral and Multispectral Image Fusion

Deep learning, especially convolutional neural networks (CNNs), has shown very promising results for multispectral (MS) and hyperspectral (HS) image fusion (MS/HS fusion) task. Most of the existing CNN methods are based on “black-box” models that are not specifically designed for MS/HS fusion, which largely ignore the priors evidently possessed by the observed HS and MS images, and lack clear interpretability, leaving room for further improvement. In this paper, we propose an interpretable network, named as spatial-spectral dual-optimization model driven deep network (S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> DMDN), which embeds the intrinsic generation mechanism of the MS/HS fusion to the network. There are two key characteristics: (i) Explicitly encode the spatial prior and spectral prior evidently possessed by the input MS and HS images in the network architecture; (ii) Unfold an iterative spatial-spectral dual-optimization algorithm into a model driven deep network. The benefit is that the network has good interpretability and generalization capability, and the fused image is richer in semantics and more precise in spatial. Extensive experiments are conducted to prove the superiority of our proposed method over other state-of-the-art methods in terms of quantitative evaluation metrics and qualitative visual effects.