Quality Of Fused Image Research Articles

Multi‐domain pseudo‐reference quality evaluation for infrared and visible image fusion

AbstractInfrared and visible image fusion involves merging the advantages of infrared and visible images to generate a composite image that encompasses thermal radiation as well as intricate texture details. Infrared and visible image fusion has garnered increasing attention, with numerous fusion methods proposed. However, how to fairly perceive the performance of fused image remains a contentious topic. This paper is dedicated to solving this problem from two perspectives (e.g., subjective and objective aspects). Firstly, an infrared and visible fusion image quality assessment dataset was constructed, including 60 pairs of infrared and visible images captured in various scenes, along with 540 fusion images with different types and degrees of distortions. Additionally, a subjective evaluation dataset of 16,200 subjective scores by 30 participants was further provided for the fused image. Secondly, to overcome the challenging assessment for infrared and visible fusion images without a real reference image, an interesting multi‐domain pseudo‐reference image quality assessment model (MPIQAM) is proposed, by comprehensively considering the thermal radiation information distortion, texture information distortion, and overall naturalness of the fused image. The proposed MPIQAM was compared with 18 mainstream objective metrics, and the experimental findings showcased a commendable level of competitiveness.

FusionOC: Research on optimal control method for infrared and visible light image fusion

Infrared and visible light image fusion can solve the limitations of single-type visual sensors and can boost the target detection performance. However, since the traditional fusion strategy lacks the controllability and feedback mechanism, the fusion model cannot precisely perceive the relationship between the requirements of the fusion task, the fused image quality, and the source image features. To this end, this paper establishes a fusion model based on the optimal controlled object and control mode called FusionOC. This method establishes two types of mathematical models of the controlled objects by verifying the factors and conflicts affecting the quality of the fused image. It combines the image fusion model with the quality evaluation function to determine the two control factors separately. At the same time, two proportional-integral-derivative (PID) control and regulation modes based on the backpropagation (BP) neural network are designed according to the control factor characteristics. The fusion system can adaptively select the regulation mode to regulate the control factor according to the user requirements or the task to make the fusion system perceive the connection between the fusion task and the result. Besides, the fusion model employs the feedback mechanism of the control system to perceive the feature difference between the fusion result and the source image, realize the guidance of the source image feature to the entire fusion process, and improve the fusion algorithm's generalization ability and intelligence level when handling different fusion tasks. Experimental results on multiple public datasets demonstrate the advantages of FusionOC over advanced methods. Meanwhile, the benefits of our fusion results in object detection tasks have been demonstrated.

Improved multi-focus image fusion using online convolutional sparse coding based on sample-dependent dictionary

Multi-focus image fusion merges multiple images captured from different focused regions of a scene to create a fully-focused image. Convolutional sparse coding (CSC) methods are commonly employed for accurate extraction of focused regions, but they often disregard computational costs. To overcome this, an online convolutional sparse coding (OCSC) technique was introduced, but its performance is still limited by the number of filters used, affecting overall performance negatively. To address these limitations, a novel approach called Sample-Dependent Dictionary-based Online Convolutional Sparse Coding (SCSC) was proposed. SCSC enables the utilization of additional filters while maintaining low time and space complexity for processing high-dimensional or large data. Leveraging the computational efficiency and effective global feature extraction of SCSC, we propose a novel method for multi-focus image fusion. Our method involves a two-layer decomposition of each source image, yielding a base layer capturing the predominant features and a detail layer containing finer details. The amalgamation of the fused base and detail layers culminates in the reconstruction of the final image. The proposed method significantly mitigates artifacts, preserves fine details at the focus boundary, and demonstrates notable enhancements in both visual quality and objective evaluation of multi-focus image fusion.

Infrared and Harsh Light Visible Image Fusion Using an Environmental Light Perception Network.

The complementary combination of emphasizing target objects in infrared images and rich texture details in visible images can effectively enhance the information entropy of fused images, thereby providing substantial assistance for downstream composite high-level vision tasks, such as nighttime vehicle intelligent driving. However, mainstream fusion algorithms lack specific research on the contradiction between the low information entropy and high pixel intensity of visible images under harsh light nighttime road environments. As a result, fusion algorithms that perform well in normal conditions can only produce low information entropy fusion images similar to the information distribution of visible images under harsh light interference. In response to these problems, we designed an image fusion network resilient to harsh light environment interference, incorporating entropy and information theory principles to enhance robustness and information retention. Specifically, an edge feature extraction module was designed to extract key edge features of salient targets to optimize fusion information entropy. Additionally, a harsh light environment aware (HLEA) module was proposed to avoid the decrease in fusion image quality caused by the contradiction between low information entropy and high pixel intensity based on the information distribution characteristics of harsh light visible images. Finally, an edge-guided hierarchical fusion (EGHF) module was designed to achieve robust feature fusion, minimizing irrelevant noise entropy and maximizing useful information entropy. Extensive experiments demonstrate that, compared to other advanced algorithms, the method proposed fusion results contain more useful information and have significant advantages in high-level vision tasks under harsh nighttime lighting conditions.

RAN: Infrared and Visible Image Fusion Network Based on Residual Attention Decomposition

Infrared image and visible image fusion (IVIF) is a research direction that is currently attracting much attention in the field of image processing. The main goal is to obtain a fused image by reasonably fusing infrared images and visible images, while retaining the advantageous features of each source image. The research in this field aims to improve image quality, enhance target recognition ability, and broaden the application areas of image processing. To advance research in this area, we propose a breakthrough image fusion method based on the Residual Attention Network (RAN). By applying this innovative network to the task of image fusion, the mechanism of the residual attention network can better capture critical background and detail information in the images, significantly improving the quality and effectiveness of image fusion. Experimental results on public domain datasets show that our method performs excellently on multiple key metrics. For example, compared to existing methods, our method improves the standard deviation (SD) by 35.26%, spatial frequency (SF) by 109.85%, average gradient (AG) by 96.93%, and structural similarity (SSIM) by 23.47%. These significant improvements validate the superiority of our proposed residual attention network in the task of image fusion and open up new possibilities for enhancing the performance and adaptability of fusion networks.

CFNet: An infrared and visible image compression fusion network

Image fusion aims to acquire a more complete image representation within a limited physical space to more effectively support practical vision applications. Although the currently popular infrared and visible image fusion algorithms take practical applications into consideration. However, they did not fully consider the redundancy and transmission efficiency of image data. To address this limitation, this paper proposes a compression fusion network for infrared and visible images based on joint CNN and Transformer, termed CFNet. First of all, the idea of variational autoencoder image compression is introduced into the image fusion framework, achieving data compression while maintaining image fusion quality and reducing redundancy. Moreover, a joint CNN and Transformer network structure is proposed, which comprehensively considers the local information extracted by CNN and the global long-distance dependencies emphasized by Transformer. Finally, multi-channel loss based on region of interest is used to guide network training. Not only can color visible and infrared images be fused directly but more bits can be allocated to the foreground region of interest, resulting in a superior compression ratio. Extensive qualitative and quantitative analyses affirm that the proposed compression fusion algorithm achieves state-of-the-art performance. In particular, rate–distortion performance experiments demonstrate the great advantages of the proposed algorithm for data storage and transmission. The source code is available at https://github.com/Xiaoxing0503/CFNet.

Dynamic and static fusion mechanisms of infrared and visible images

This paper propose a dynamic fusion mechanism of infrared and visible images, named DIFM, capable of solving the static fusion optimization problem. The DIFM correlates the image fusion quality with the image restoration quality to construct a unified optimization loss function. According to the DIFM, a dynamic image fusion network of infrared and visible images is constructed and is therefore denoted with DF-Net. Specifically, the DF-Net comprises two modules, i.e., the dynamic fusion module (DFM) and the self-learning dynamic restoration module (SLDRM). In order to solve the static fusion problem of existing methods, the DFM is proposed to learn the fusion weight dynamically. Specifically, the DFM comprises a classification module (CM) and an image fusion module (IFM), which determine whether and how to fuse source images. In addition, a unified fusion loss function is introduced to obtain more hidden features of infrared and visible images in complex environments. Therefore, the stumbling block of deep learning in image fusion, i.e., static fusion, is significantly mitigated. Extensive experiments demonstrate that the dynamic fusion optimization method neatly outperforms the state-of-the-art methods in most metrics.

Internet street view image fusion method using convolutional neural network

The use of image fusion technology in the area of information processing is continuing to advance in depth thanks to ongoing hardware advancements and related research. An enhanced convolutional neural network approach is developed to fuse visible and infrared images, and image pre-processing is carried out utilising an image alignment method with edge detection in order to gain more accurate and trustworthy image information. The performance of the fast wavelet decomposition, convolutional neural network, and modified convolutional neural network techniques is compared and examined using four objective assessment criteria. The experimental findings demonstrated that the picture alignment was successful with an offset error of fewer than 3 pixels in the horizontal direction and an angle error of less than 0.3∘ in both directions. The revised convolutional neural network method increased the information entropy, mean gradient, standard deviation, and edge detection information by an average of 46.13%, 39.40%, 19.91%, and 3.72%. The runtime of the modified approach was lowered by 19.42% when compared to the convolutional neural network method, which enhanced the algorithm’s performance and boosted the effectiveness of picture fusion. The image fusion accuracy reached 98.61%, indicating that the method has better fusion performance and is of practical value for improving image fusion quality.

Multi-focused image fusion algorithm based on multi-scale hybrid attention residual network.

In order to improve the detection performance of image fusion in focus areas and realize end-to-end decision diagram optimization, we design a multi-focus image fusion network based on deep learning. The network is trained using unsupervised learning and a multi-scale hybrid attention residual network model is introduced to enable solving for features at different levels of the image. In the training stage, multi-scale features are extracted from two source images with different focal points using hybrid multi-scale residual blocks (MSRB), and the up-down projection module (UDP) is introduced to obtain multi-scale edge information, then the extracted features are operated to obtain deeper image features. These blocks can effectively utilize multi-scale feature information without increasing the number of parameters. The deep features of the image are extracted in its test phase, input to the spatial frequency domain to calculate and measure the activity level and obtain the initial decision map, and use post-processing techniques to eliminate the edge errors. Finally, the decision map is generated and optimized, and the final fused image is obtained by combining the optimized decision map with the source image. The comparative experiments show that our proposed model achieves better fusion performance in subjective evaluation, and the quality of the obtained fused images is more robust with richer details. The objective evaluation metrics work better and the image fusion quality is higher.

G2NPAN: GAN-guided nuance perceptual attention network for multimodal medical fusion image quality assessment.

Multimodal medical fusion images (MMFI) are formed by fusing medical images of two or more modalities with the aim of displaying as much valuable information as possible in a single image. However, due to the different strategies of various fusion algorithms, the quality of the generated fused images is uneven. Thus, an effective blind image quality assessment (BIQA) method is urgently required. The challenge of MMFI quality assessment is to enable the network to perceive the nuances between fused images of different qualities, and the key point for the success of BIQA is the availability of valid reference information. To this end, this work proposes a generative adversarial network (GAN) -guided nuance perceptual attention network (G2NPAN) to implement BIQA for MMFI. Specifically, we achieve the blind evaluation style via the design of a GAN and develop a Unique Feature Warehouse module to learn the effective features of fused images from the pixel level. The redesigned loss function guides the network to perceive the image quality. In the end, the class activation mapping supervised quality assessment network is employed to obtain the MMFI quality score. Extensive experiments and validation have been conducted in a database of medical fusion images, and the proposed method is superior to the state-of-the-art BIQA method.

CABnet: A channel attention dual adversarial balancing network for multimodal image fusion

Infrared and visible image fusion aims to generate informative images by leveraging the distinctive strengths of infrared and visible modalities. These fused images play a crucial role in subsequent downstream tasks, including object detection, recognition, and segmentation. However, complementary information is often difficult to extract. Existing generative adversarial network-based methods generate fused images by modifying the distribution of source images' features to preserve instances and texture details in both infrared and visible images. Nevertheless, these approaches may result in a degradation of the fused image quality when the original image quality is low. Considering the balance of information from different modalities can improve the quality of the fused image. Hence, we introduce CABnet, a Channel Attention dual adversarial Balancing network. CABnet incorporates a channel attention mechanism to capture crucial channel features, thereby, enhancing complementary information. It also employs an adaptive factor to control the mixing distribution of infrared and visible images, which ensures the preservation of instances and texture details during the adversarial process. To enhance efficiency and reduce reliance on manual labeling, our training process adopts a semi-supervised learning strategy. Through qualitative and quantitative experiments across multiple datasets, CABnet surpasses existing state-of-the-art methods in fusion performance, notably achieving a 51.3% enhancement in signal to noise ratio and a 13.4% improvement in standard deviation.

An unsupervised multi-focus image fusion method via dual-channel convolutional network and discriminator

The challenge in multi-focus image fusion tasks lies in accurately preserving the complementary information from the source images in the fused image. However, existing datasets often lack ground truth images, making it difficult for some full-reference loss functions (such as SSIM) to effectively participate in model training, thereby further affecting the performance of retaining source image details. To address this issue, this paper proposes an unsupervised dual-channel dense convolutional method, DCD, for multi-focus image fusion. DCD designs Patch processing blocks specifically for the fusion task, which segment the source image pairs into equally sized patches and evaluate their information to obtain a reconstructed image and a set of adaptive weight coefficients. The reconstructed image is used as the reference image, enabling unsupervised methods to utilize full-reference loss functions in training and overcoming the challenge of lacking labeled data in the training set. Furthermore, considering that the human visual system (HVS) is more sensitive to brightness than color, DCD trains the dual-channel network using both RGB images and their luminance components. This allows the network to focus more on the brightness information while preserving the color and gradient details of the source images, resulting in fused images that are more compatible with the HVS. The adaptive weight coefficients obtained through the Patch processing blocks are also used to determine the degree of preservation of the brightness information in the source images. Finally, comparative experiments on different datasets also demonstrate the superior performance of DCD in terms of fused image quality compared to other methods.

Dual-band color fusion image quality assessment based on color descriptors

Dual-band color fusion image quality assessment based on color descriptors

An improved arithmetic optimization algorithm with multi-strategy for adaptive multi-spectral image fusion

Panchromatic and multi-spectral image fusion, called panchromatic sharpening, is the process of combining the spatial and spectral information of the source image into the fused image to give the image a higher spatial and spectral resolution. In order to improve the spatial resolution and spectral information quality of the image, an adaptive multi-spectral image fusion method based on an improved arithmetic optimization algorithm is proposed. This paper proposed improved arithmetic optimization algorithm, which uses dynamic stochastic search technique and oppositional learning operator, to perform local search and behavioral complementation of population individuals, and to improve the ability of population individuals to jump out of the local optimum. The method combines adaptive methods to calculate the weights of linear combinations of panchromatic and multi-spectral gradients to improve the quality of fused images. This study not only improves the quality and effect of image fusion, but also focuses on optimizing the operation efficiency of the algorithm to have real-time and high efficiency. Experimental results show that the proposed method exhibits strong performance on different datasets, improves the spatial resolution and spectral information quality of the fused images, and has good adaptability and robustness. The source code is available at: https://github.com/starboot/IAOA-For-Image-Fusion.

A new multi-focus image fusion quality assessment method with convolutional sparse representation

A new multi-focus image fusion quality assessment method with convolutional sparse representation

MFHOD: Multi-modal image fusion method based on the higher-order degradation model

The task of multimodal image fusion aims to preserve the respective advantages of each modality, such as the detailed texture information from visible light images and the salient target information from infrared images. However, images in real environments are often not perfect high-quality images but are affected by various degradation factors such as noise, blur, and glare. Existing unsupervised multimodal image fusion algorithms use undegraded source images as input to fusion networks and generate high-quality fusion images through complex fusion strategies. This single linear learning approach from high-quality to high-quality cannot be applied to the fusion task of degraded multimodal images in real environments. To address this issue, this paper proposes an end-to-end multimodal image fusion network based on a high-order local degradation model (MFHOD). Firstly, inspired by the idea of probabilistic degradation, we propose a high-order local random degradation model (HODM), which inputs the source multimodal images into the degradation model to obtain degraded images before feeding them into the network. Secondly, we design a simple and efficient dual-branch feature extraction encoder to extract deep features from images. Then, from the perspectives of image pixels, brightness, and gradients, we propose an improved composite loss composed of multiple loss functions to constrain network training. Finally, we propose a L2-norm fusion strategy for preserving brightness information in low-light nighttime images. Our MFHOD demonstrates good performance on infrared and visible light image datasets as well as medical image datasets. Experimental results show that MFHOD can effectively suppress the effects of glare, noise, and smoke in adverse environments, and also improve the quality of fusion images in low-light and nighttime environments.

Novel fusion strategy for image fusion using rescue hunt optimization-based modified guidance model

ABSTRACT A new method for image fusion introduces an inventive strategy for amalgamating data from multiple images, resulting in the creation of a single, improved output image. This approach aims to address the limitations and challenges associated with conventional fusion techniques, paving the way for improved results in various applications. In this research, a novel approach for image fusion is presented, featuring a rescue hunt optimisation-based modified guidance model (RHO-based MG model). The methodology leverages the non-subsampled contourlet transform (NSCT) and non-subsampled shear let transform (NSST) to construct the fusion transform using two input images, typically infrared and visual images. By hybridising high-frequency (HF) and low-frequency (LF) bands from both types of images, the fusion model generates the final HF and LF bands. A distinctive modified guidance strategy is employed in the development of these bands. The proposed approach utilises a rescue hunt algorithm developed through the combination of search and rescue optimisation (SARO) and grey wolf optimisation (GWO) behaviours. In image fusion, optimisation fine-tunes VGG-19 and RESNET 18 models to improve their ability to combine multiple images effectively. This process involves adjusting the models’ internal parameters using optimisation techniques. By analysing the features in different input images, these models learn to extract meaningful information and create a fused image that retains the important details from each source. This strategy is further enhanced by integrating the VGG-19 and RESNET 18 models. The fused image is composed of combined HF and LF bands, with the final result obtained through an inverse hybrid transform. Experimental results, conducted on a dataset of 25 images, demonstrate the effectiveness of the approach with metrics average gradient (AG), edge intensity (EI), PSNR, RMSE, SSIM, and variance attained the values of 24.71, 236.67, 54.96 dB, 0.22, 63.16, and 0.11, respectively. This innovative method offers a promising direction for enhancing image fusion quality and is highlighted by its unique integration of optimisation and guidance strategies.

Multi-Frequency Ultrasound Imaging Fusion Method Based on Wavelet Transform for Guided Screw Insertion.

Intraosseous ultrasound imaging can serve as a guiding tool for the placement of pedicle screws during spinal fusion surgery; thus far, there has been limited scholarly exploration of methods for intraosseous multifrequency ultrasound imaging, which can achieve simultaneous high resolution and deep penetration. The proposed method introduced a dynamic fusion strategy grounded in wavelet transformation for multifrequency image decomposition. This strategy accomplished the effective amalgamation of high-frequency ultrasound images and low-frequency ultrasound images, enabling the obtaining of fused images with enhanced details and better overall image quality. A novel near-field effect elimination method was also proposed to improve the quality of ultrasound imaging in the near-field region. Experimental evaluations were conducted on isolated bovine circle bone and sheep spine with pedicle screw tracks. The fusion images are capable of effectively detecting areas within the pedicle screw track that have either ruptured or are in close proximity to rupture, even measuring the size of breaches. Evaluation criteria, including information entropy (IE), spatial frequency (SF), average gradient (AG), mutual information (MI), structural similarity index (SSIM), and edge information-based image fusion quality metric (QAB/F), were employed to assess the fusion performance; moreover, the influence of mother wavelet function selection and decomposition levels on computational complexity and fusion image quality was thoroughly discussed. The proposed method exhibited promising potential for intraosseous imaging navigation, which can aid in accurate diagnosis, treatment planning, and monitoring in fields such as orthopedics, surgery, and interventional procedures.

HATF: Multi-Modal Feature Learning for Infrared and Visible Image Fusion via Hybrid Attention Transformer

Current CNN-based methods for infrared and visible image fusion are limited by the low discrimination of extracted structural features, the adoption of uniform loss functions, and the lack of inter-modal feature interaction, which make it difficult to obtain optimal fusion results. To alleviate the above problems, a framework for multimodal feature learning fusion using a cross-attention Transformer is proposed. To extract rich structural features at different scales, residual U-Nets with mixed receptive fields are adopted to capture salient object information at various granularities. Then, a hybrid attention fusion strategy is employed to integrate the complementing information from the input images. Finally, adaptive loss functions are designed to achieve optimal fusion results for different modal features. The fusion framework proposed in this study is thoroughly evaluated using the TNO, FLIR, and LLVIP datasets, encompassing diverse scenes and varying illumination conditions. In the comparative experiments, HATF achieved competitive results on three datasets, with EN, SD, MI, and SSIM metrics reaching the best performance on the TNO dataset, surpassing the second-best method by 2.3%, 18.8%, 4.2%, and 2.2%, respectively. These results validate the effectiveness of the proposed method in terms of both robustness and image fusion quality compared to several popular methods.

Shortwave infrared and visible light image fusion method based on dual discriminator GAN

In a tactical warfare setting, the efficacy of target detection becomes profoundly compromised due to prevalent environmental factors such as smoke, dust, and atmospheric interference. Such impediments invariably undermine the precision and reliability of identifying pivotal targets, thereby precipitating potentially dire ramifications. Remarkably, short-wave infrared technology has exhibited unparalleled proficiency in elucidating target attributes even amidst challenging conditions characterized by smoke, fog, or haze. Against this backdrop, the present study delineates a pioneering algorithmic framework that seamlessly amalgamates the imperatives of image registration and fusion. This is achieved through the deployment of an advanced dual-discriminator Generative Adversarial Network (GAN), specifically tailored for amalgamating short-wave infrared and visible light imagery within smoke-obscured contexts. Our methodology commences with the introduction of an augmented Speeded-Up Robust Features (SURF) algorithm, meticulously designed to rectify inherent misalignments within the input imagery. Subsequent enhancements encompass the refinement of the generator’s loss function and the integration of a multi-scale convolutional kernel, thereby facilitating the extraction and amalgamation of a more expansive array of salient features. This concerted effort culminates in the elevation of image fusion quality. To corroborate the efficacy and robustness of our proposed framework, rigorous validation procedures were conducted utilizing a meticulously curated dataset comprising short-wave infrared and visible light images. Empirical evaluations, encompassing both subjective and objective comparative analyses, unequivocally affirm the superior performance metrics of our fusion network. Specifically, our methodology surpasses alternative fusion techniques across multiple dimensions, including visual fidelity, perceptual quality, and structural congruence of synthesized images.