Existing Image Fusion Methods Research Articles

Infrared and visible image fusion based on semi-global weighted least squares and guided edge-aware filters

Most of the existing image fusion methods are dedicated to extracting the respective private features in the source image to reconstruct to get a fused image that contains rich information. However, its neglect of the enhancement of texture details in the source image as well as the preservation of background structures, which makes the fusion result look less impressive. To this end, an infrared and visible image fusion method based on a semi-global weighted least squares (SGWLS) method and guided edge-aware filter (GEAF) is proposed. First, a SGWLS method is applied to decompose the source image into base and detail layers containing different scale information. Then, the detail layer are texture-enhanced by using the scale coefficients. Due to the lack of edge features in the original base layer, for this reason, a guided edge-aware filter is designed, in which the enhanced detail layer was used as the guidance image as a way to perform a quadratic feature enhancement fusion for the base layer in order to retain richer gradient information. Finally, reconstruction of the fused detail and base layers by using inverse transformation to obtain the fusion result. Compared with nine state-of-the-art fusion algorithms, a large number of experimental results demonstrate the superior performance of the proposed method in highlighting texture details and background features, as well as the obvious advantages in SF and AG metrics.

COA: A novel contrast fusion algorithm for GM-APD range image and infrared image under low-photon condition

Most existing image fusion methods focus on infrared images and visible images, which underperform under complex conditions. However, they cannot effectively distinguish similar targets at near distances and it limits the target recognition, especially when the image is missing badly under low photon conditions. To solve this problem, we propose a contrast optimization algorithm (COA) for fusing the Geiger-mode avalanche photodiode (GM-APD) image and infrared image that could emphasize and distinguish similar targets in low photon conditions. The COA method includes two main parts. First, using the processed infrared image as a reference to recover the GM-APD image by using DLatLRR system methods and reference recovery method. Then, we proposed a new loss function consist of contrast fidelity term and regularization term which focus on the contrasts of the infrared image and GM-APD image, and introducing the split Bregman to solve the optimization equation. Results show that the value of contrasts between targets (TC) of the proposed method is twenty times more than the infrared image and higher than that of the other 9 fusion algorithms when the mean photon per pixel (PPP) is 1/500. Meanwhile, the proposed method can also be used for combined with the existing fusion methods to fuse the infrared image and GM-APD image, which makes accurate target identification under low photon conditions.

A general image fusion framework using multi-task semi-supervised learning

Existing image fusion methods primarily focus on solving single-task fusion problems, overlooking the potential information complementarity among multiple fusion tasks. Additionally, there has been no prior research in the field of image fusion that explores the mixed training of labeled and unlabeled data for different fusion tasks. To address these gaps, this paper introduces a novel multi-task semi-supervised learning approach to construct a general image fusion framework. This framework not only facilitates collaborative training for multiple fusion tasks, thereby achieving effective information complementarity among datasets from different fusion tasks, but also promotes the (unsupervised) learning of unlabeled data via the (supervised) learning of labeled data. Regarding the specific network module, we propose a so-called pseudo-siamese Laplacian pyramid transformer (PSLPT), which can effectively distinguish information at different frequencies in source images and discriminatively fuse features from distinct frequencies. More specifically, we take datasets of four typical image fusion tasks into the same PSLPT for weight updates, yielding the final general fusion model. Extensive experiments demonstrate that the obtained general fusion model exhibits promising outcomes for all four image fusion tasks, both visually and quantitatively. Moreover, comprehensive ablation and discussion experiments corroborate the effectiveness of the proposed method. The code is available at https://github.com/wwhappylife/A-general-image-fusion-framework-using-multi-task-semi-supervised-learning.

CROSE: Low-light enhancement by CROss-SEnsor interaction for nighttime driving scenes

An increasing number of image perception models are being utilized in the field of autonomous driving. During nighttime driving, the visual perception capabilities of a single-modality RGB sensor become compromised. To address this issue, image fusion methods that utilize multi-modality data to provide a more comprehensive visual representation for nighttime scenes. Nevertheless, the existing image fusion methods are limited by poor generalization abilities and lighting conditions, resulting in inadequate handling of the contrast and spectral corruption. In this paper, we propose a cross-sensor low-light enhancement algorithm that provides more accurate visual perceptions. Our approach employs multiple sensors as preceptors for nighttime driving scenes. Compared with typical fusion algorithms that use a simple one-stage workflow for cross-sensor data, our proposed method adopts an advanced content-enhancement strategy by recursively and interactively scaling up informative pixels. More specifically, our approach employs information measurement to describe the information and then utilizes a cross-sensor content enhancement module to dynamically enhance the mutual information between the infrared spectrum and the RGB streams. Our experiments show that CROSE effectively preserves texture details, resulting in clearer and more solid fusion results for nighttime driving scenes.

Deep Selective Fusion of Visible and Near-Infrared Images Using Unsupervised U-Net.

In low light conditions, visible (VIS) images are of a low dynamic range (low contrast) with severe noise and color, while near-infrared (NIR) images contain clear textures without noise and color. Multispectral fusion of VIS and NIR images produces color images of high quality, rich textures, and little noise by taking both advantages of VIS and NIR images. In this article, we propose the deep selective fusion of VIS and NIR images using unsupervised U-Net. Existing image fusion methods are afflicted with the low contrast in VIS images and flash-like effect in NIR images. Thus, we adopt unsupervised U-Net to achieve deep selective fusion of multiple scale features. Due to the absence of the ground truth, we use unsupervised learning by formulating an energy function as a loss function. To deal with insufficient training data, we perform data augmentation by rotating images and adjusting their intensity. We synthesize training data by degrading clean VIS images and masking clean NIR images using a circle. First, we utilize pretrained visual geometry group (VGG) to extract features from VIS images. Second, we build an encoding network to obtain edge information from NIR images. Finally, we combine all features and feed them into a decoding network for fusion. Experimental results demonstrate that the proposed fusion network produces visually pleasing results with fine details, little noise, and natural color and it is superior to state-of-the-art methods in terms of visual quality and quantitative measurements.

OMOFuse: An Optimized Dual-Attention Mechanism Model for Infrared and Visible Image Fusion

Infrared and visible image fusion aims to fuse the thermal information of infrared images and the texture information of visible images into images that are more in compliance with people’s visual perception characteristics. However, in the existing related work, the fused images have incomplete contextual information and poor fusion results. This paper presents a new image fusion algorithm—OMOFuse. At first, both the channel and spatial attention mechanisms are optimized by a DCA (dual-channel attention) mechanism and an ESA (enhanced spatial attention) mechanism. Then, an ODAM (optimized dual-attention mechanism) module is constructed to further improve the integration effect. Moreover, a MO module is used to improve the network’s feature extraction capability for contextual information. Finally, there is the loss function ℒ from the three parts of SSL (structural similarity loss), PL (perceptual loss), and GL (gap loss). Extensive experiments on three major datasets are performed to demonstrate that OMOFuse outperforms the existing image fusion methods in terms of quantitative determination, qualitative detection, and superior generalization capabilities. Further evidence of the effectiveness of our algorithm in this study are provided.

Comprehensive urban space representation with varying numbers of street-level images

Street-level imagery has emerged as a valuable tool for observing large-scale urban spaces with unprecedented detail. However, previous studies have been limited to analyzing individual street-level images. This approach falls short in representing the characteristics of a spatial unit, such as a street or grid, which may contain varying numbers of street-level images ranging from several to hundreds. As a result, a more comprehensive and representative approach is required to capture the complexity and diversity of urban environments at different spatial scales. To address this issue, this study proposes a deep learning-based module called Vision-LSTM, which can effectively obtain vector representation from varying numbers of street-level images in spatial units. The effectiveness of the module is validated through experiments to recognize urban villages, achieving reliable recognition results (overall accuracy: 91.6%) through multimodal learning that combines street-level imagery with remote sensing imagery and social sensing data. Compared to existing image fusion methods, Vision-LSTM demonstrates significant effectiveness in capturing associations between street-level images. The proposed module can provide a more comprehensive understanding of urban spaces, enhancing the research value of street-level imagery and facilitating multimodal learning-based urban research. Our models are available at https://github.com/yingjinghuang/Vision-LSTM.

MURF: Mutually Reinforcing Multi-Modal Image Registration and Fusion.

Existing image fusion methods are typically limited to aligned source images and have to "tolerate" parallaxes when images are unaligned. Simultaneously, the large variances between different modalities pose a significant challenge for multi-modal image registration. This study proposes a novel method called MURF, where for the first time, image registration and fusion are mutually reinforced rather than being treated as separate issues. MURF leverages three modules: shared information extraction module (SIEM), multi-scale coarse registration module (MCRM), and fine registration and fusion module (F2M). The registration is carried out in a coarse-to-fine manner. During coarse registration, SIEM first transforms multi-modal images into mono-modal shared information to eliminate the modal variances. Then, MCRM progressively corrects the global rigid parallaxes. Subsequently, fine registration to repair local non-rigid offsets and image fusion are uniformly implemented in F2M. The fused image provides feedback to improve registration accuracy, and the improved registration result further improves the fusion result. For image fusion, rather than solely preserving the original source information in existing methods, we attempt to incorporate texture enhancement into image fusion. We test on four types of multi-modal data (RGB-IR, RGB-NIR, PET-MRI, and CT-MRI). Extensive registration and fusion results validate the superiority and universality of MURF.

Visible and Infrared Image Fusion of Forest Fire Scenes Based on Generative Adversarial Networks with Multi-Classification and Multi-Level Constraints

Aimed at addressing deficiencies in existing image fusion methods, this paper proposed a multi-level and multi-classification generative adversarial network (GAN)-based method (MMGAN) for fusing visible and infrared images of forest fire scenes (the surroundings of firefighters), which solves the problem that GANs tend to ignore visible contrast ratio information and detailed infrared texture information. The study was based on real-time visible and infrared image data acquired by visible and infrared binocular cameras on forest firefighters’ helmets. We improved the GAN by, on the one hand, splitting the input channels of the generator into gradient and contrast ratio paths, increasing the depth of convolutional layers, and improving the extraction capability of shallow networks. On the other hand, we designed a discriminator using a multi-classification constraint structure and trained it against the generator in a continuous and adversarial manner to supervise the generator, generating better-quality fused images. Our results indicated that compared to mainstream infrared and visible image fusion methods, including anisotropic diffusion fusion (ADF), guided filtering fusion (GFF), convolutional neural networks (CNN), FusionGAN, and dual-discriminator conditional GAN (DDcGAN), the MMGAN model was overall optimal and had the best visual effect when applied to image fusions of forest fire surroundings. Five of the six objective metrics were optimal, and one ranked second-to-optimal. The image fusion speed was more than five times faster than that of the other methods. The MMGAN model significantly improved the quality of fused images of forest fire scenes, preserved the contrast ratio information of visible images and the detailed texture information of infrared images of forest fire scenes, and could accurately reflect information on forest fire scene surroundings.

FusionDiff: Multi-focus image fusion using denoising diffusion probabilistic models

Multi-focus image fusion (MFIF) is an image enhancement technology with broad application prospects that can effectively extend the depth-of-field of optical lenses. This paper presents a novel MFIF method, named FusionDiff, based on denoising diffusion probabilistic models (DDPM). FusionDiff uses DDPM to fuse two source images by iteratively performing multiple denoising operations. To predict noise accurately and train the model efficiently, a lightweight U-Net architecture is designed as the conditional noise predictor. FusionDiff does not depend on any specific activity level measurement method, fusion rule or complex feature extraction network. It overcomes many algorithm design and training difficulties in existing image fusion methods. FusionDiff is noise-resistant and can still produce outstanding fusion results from source images with noise interference. In addition, FusionDiff is a few-shot learning method, which makes it suitable for image fusion tasks where training samples are relatively scarce. Experiments show that FusionDiff outperforms representative state-of-the-art methods in both visual perception and quantitative metrics. The code is available at https://github.com/lmn-ning/ImageFusionhttps://github.com/lmn-ning/ImageFusion

An adaptive image fusion method for Sentinel-2 images and high-resolution images with long-time intervals

Sentinel-2 imagery has garnered significant attention in many earth system studies due to free access and high revisit frequency. Since its spatial resolution is insufficient for many applications, e.g., fine-grained land cover mapping, some studies employ fusion technique that combines high-resolution RGB images with Sentinel-2 multispectral images to improve the resolution of the latter. However, there are two issues in the existing image fusion methods. First, these methods usually assume that the time intervals between images are short (within several days), which is a strong assumption for large-scale high-resolution images and many real-world applications. Second, the spectral discrepancy between multispectral and RGB images could induce spectral aberrations in Sentinel-2 imagery upon fusion. To alleviate these issues, we propose an adaptive image fusion approach named S2IFNet, adaptively fusing images with long-time intervals (from months to years) and spectral inconsistency, thereby increasing the multispectral band resolution of Sentinel-2 imagery. Building on top of the feature extraction and fusion modules, we propose a spectral feature compensation module and a change-aware feature reconstruction module. The former alleviates the possible degradation of spectral attributes in Sentinel-2 imagery resulting from feature fusion. The latter integrates semantic and texture information to avoid adding fake textures caused by land cover changes over time. The experiments demonstrate that S2IFNet surpasses existing image fusion and reference-based super-resolution methods on synthetic and real datasets, yielding fusion results that are clearer and more reliable.

Shadow removal of spacecraft images with multi-illumination angles image fusion

Spacecraft components recognition is essential for space on-orbit service missions such as space docking, on-orbit maintenance, and other applications. Compared with the expensive recognition methods based on the Light Detection and Ranging (LIDAR) system, the optical camera provides an economical alternative. However, owing to the adverse illumination in space, spacecraft images often suffer from shadow and poor visibility, which seriously damages the recognition accuracy. To overcome the image quality limitation, we proposed a novel concept “Multi-Illumination Angles Image Fusion” (MIAF), which aims to fuse the information of single spacecraft in different illumination angles and reconstruct the shadow-free spacecraft image. In this study, a new dataset containing spacecraft images under multiple illumination angles is created. Then a novel image fusion model for spacecraft image shadow removal is proposed. Different from existing image fusion methods that use handcrafted fusion rules, a weight subnet block is designed to learn the optimal fusion strategy automatically. Additionally, a novel loss function combining supervision and self-supervision is adopted to train the weight subnet. To verify the advantage, the proposed method is evaluated and compared with 8 competitive image fusion methods on the released dataset. The proposed method achieves the state-of-the-art performance in both qualitative and quantitative experiment results. The dataset and MIAF implementation code are open-sourced, which can be found in (https://github.com/xiang-ao-data/Spacefuse-shadow-removal/tree/master).

Medical Image Fusion Using a New Entropy Measure Between Intuitionistic Fuzzy Sets Joint Gaussian Curvature Filter

Medical image fusion can provide a synthetical presentation of human tissue by fusing the complementary features of different source images, such as computerized tomography (CT) and magnetic resonance imaging (MRI), which contributes to clinical diagnosis and treatment. How to measure the key features of medical image is the most important issue in image fusion, intuitionistic fuzzy set (IFS) is a classical computation theory that can be employed to handle the task of feature extraction. In this work, we propose a medical image fusion method based on a new entropy measure of IFSs joint Gaussian curvature filter (GCF). First, GCF is employed to separate the medical image into a set of detailed sub-images and a base sub-image. Second, the detailed sub-images is transformed into IFSs, so that the entropy measure of IFSs is then used to present image features. Third, two specific fusion rules are respectively utilized to integrate the detailed and base sub-images. Finally, the fused image is produced by the fused sub-images based on the regulation of GCF. Several specific experiments are performed on our entropy measure to test the performance, and the existing image fusion methods are also performed to verify the performance of our image fusion method.

Multiscale aggregation and illumination‐aware attention network for infrared and visible image fusion

AbstractImage fusion plays a significant role in computer vision since numerous applications benefit from the fusion results. The existing image fusion methods are incapable of perceiving the most discriminative regions under varying illumination circumstances and thus fail to emphasize the salient targets and ignore the abundant texture details of the infrared and visible images. To address this problem, a multiscale aggregation and illumination‐aware attention network (MAIANet) is proposed for infrared and visible image fusion. Specifically, the MAIANet consists of four modules, namely multiscale feature extraction module, lightweight channel attention module, image reconstruction module, and illumination‐aware module. The multiscale feature extraction module attempts to extract multiscale features in the images. The role of the lightweight channel attention module is to assign different weights to each channel so as to focus on the essential regions in the infrared and visible images. An illumination‐aware module is employed to assess the probability distribution regarding the illumination factor. Meanwhile, an illumination perception loss is formulated by the illumination probabilities to enable the proposed MAIANet to better adjust to the changes in illumination. Experimental results on three datasets, that is, MSRS, TNO, and RoadSence, verify the effectiveness of the MAIANet in both qualitative and quantitative evaluations.

THFuse: An infrared and visible image fusion network using transformer and hybrid feature extractor

Infrared and visible image fusion aims to integrate complementary information from different types of images into one image. The existing image fusion methods are primarily based on convolutional neural network (CNN), which ignores long-range dependencies of images, resulting in the fusion network unable to generate images with good complementarity. Inspired by the importance of global information, we introduced the transformer technique into the CNN-based fusion network as a way to improve the entire image-level perception in complex fusion scenarios. In this paper, we propose an end-to-end image fusion framework based on transformer and hybrid feature extractor, which enables the network to focus on both global and local information, using the characteristics of transformer to compensate for the shortcomings of CNN itself. In our network, the dual-branch CNN module is used to extract the shallow features of images, and then the vision transformer module is used to obtain the global channel and spatial relationship in the features. Finally, the fusion results are obtained through the image reconstruction module. We calculate the loss in the features of different depths according to the different kinds of original images by using the pre-trained VGG19 network. The experimental results show the effectiveness of adding the vision transformer module. Compared with other traditional and deep learning methods, our method achieves state-of-the-art qualitative and quantitative experiments performance.

Infrared and Visible Image Fusion for Highlighting Salient Targets in the Night Scene.

The goal of infrared and visible image fusion in the night scene is to generate a fused image containing salient targets and rich textural details. However, the existing image fusion methods fail to take the unevenness of nighttime luminance into account. To address the above issue, an infrared and visible image fusion method for highlighting salient targets in the night scene is proposed. First of all, a global attention module is designed, which rescales the weights of different channels after capturing global contextual information. Second, the loss function is divided into the foreground loss and the background loss, forcing the fused image to retain rich texture details while highlighting the salient targets. Finally, a luminance estimation function is introduced to obtain the trade-off control parameters of the foreground loss function based on the nighttime luminance. It can effectively highlight salient targets by retaining the foreground information from the source images. Compared with other advanced methods, the experimental results adequately demonstrate the excellent fusion performance and generalization of the proposed method.

Fusion of overexposed and underexposed images using caputo differential operator for resolution and texture based enhancement

The visual quality of images captured under sub-optimal lighting conditions, such as over and underexposure may benefit from improvement using fusion-based techniques. This paper presents the Caputo Differential Operator-based image fusion technique for image enhancement. To effect this enhancement, the proposed algorithm first decomposes the overexposed and underexposed images into horizontal and vertical sub-bands using Discrete Wavelet Transform (DWT). The horizontal and vertical sub-bands are then enhanced using Caputo Differential Operator (CDO) and fused by taking the average of the transformed horizontal and vertical fractional derivatives. This work introduces a fractional derivative-based edge and feature enhancement to be used in conjuction with DWT and inverse DWT (IDWT) operations. The proposed algorithm combines the salient features of overexposed and underexposed images and enhances the fused image effectively. We use the fractional derivative-based method because it restores the edge and texture information more efficiently than existing method. In addition, we have introduced a resolution enhancement operator to correct and balance the overexposed and underexposed images, together with the Caputo enhanced fused image we obtain an image with significantly deepened resolution. Finally, we introduce a novel texture enhancing and smoothing operation to yield the final image. We apply subjective and objective evaluations of the proposed algorithm in direct comparison with other existing image fusion methods. Our approach results in aesthetically subjective image enhancement, and objectively measured improvement metrics.

AEFusion: A multi-scale fusion network combining Axial attention and Entropy feature Aggregation for infrared and visible images

The purpose of image fusion is to generate an image that contains more complementary information. Existing image fusion methods suffer from loss of detail information, artifacts and/or inconsistencies. To alleviate these problems, we propose a feature extraction network combining with Axial-attention, which can capture long-range semantic information while extracting multi-scale features and thus has stronger feature representation capabilities. Likewise, existing fusion strategies also suffer from loss of details. To solve this problem, a new fusion strategy is proposed, where a novel attention mechanism is constructed by applying entropy features to aggregate edge and detail features. At the same time, a new loss function is designed to constrain the network. To validate the efficiency of the proposed method, validation experiments are performed on public datasets. Compared with other fusion methods, the experimental results of the proposed method demonstrate state-of-the-art advantages in both subjective and objective evaluations. Furthermore, ablation studies illustrate the superiority of the proposed method.

MsRAN: a multi-scale residual attention network for multi-model image fusion.

Fusion is a critical step in image processing tasks. Recently, deep learning networks have been considerably applied in information fusion. But the significant limitation of existing image fusion methods is the inability to highlight typical regions of the source image and retain sufficient useful information. To address the problem, the paper proposes a multi-scale residual attention network (MsRAN) to fully exploit the image feature. Its generator network contains two information refinement networks and one information integration network. The information refinement network extracts feature at different scales using convolution kernels of different sizes. The information integration network, with a merging block and an attention block added, prevents the underutilization of information in the intermediate layers and forces the generator to focus on salient regions in multi-modal source images. Furthermore, in the phase of model training, we add an information loss function and adopt a dual adversarial structure, enabling the model to capture more details. Qualitative and quantitative experiments on publicly available datasets validate that the proposed method provides better visual results than other methods and retains more detail information.

AN INFRARED AND VISIBLE IMAGE FUSION METHOD BASED ON FUZZY PULSE COUPLED NEURAL NETWORK AND SALIENCE DETECTION

Existing image fusion methods based on pulse coupled neural network (PCNN) ignore the fact that the human brain is essential to the organic combination of neural network and fuzzy system. As a result, some pixels that carry detailed information cannot activate neurons, thereby resulting in loss of details of source images and lowering the quality of fused images. To solve this problem, the present study proposed an infrared and visible image fusion method based on fuzzy pulse coupled neural network (FPCNN) and salience detection. This proposed method constructed FPCNN by combining the fuzzy theory and pulse coupled neural network. The cartoon parts obtained from the decomposition of source images by the total variation model were fused based on FPCNN. Then, salience detection of coefficients of the convolution sparse representation of the texture parts was conducted. The texture parts were fused on the basis of salience detection results. Finally, the fused cartoon and texture parts were added to obtain the fused image. The effectiveness and superiority of the proposed method were verified by experiments. Results demonstrate that compared with the most advanced seven algorithms, the proposed algorithm is superior in terms of objective evaluation indexes such as information entropy, average gradient, structural similarity, and fusion performance based on gradient. Moreover, the proposed method has good visual effect. This study provides a reference for producing high-quality all-weather images in whole-scene description. Keywords: Image fusion, Pulse coupled neural network, Salience detection, Convolution sparse representation