Image Fusion Algorithm Research Articles

TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network.

The end-to-end image fusion framework has achieved promising performance, with dedicated convolutional networks aggregating the multi-modal local appearance. However, long-range dependencies are directly neglected in existing CNN fusion approaches, impeding balancing the entire image-level perception for complex scenario fusion. In this paper, therefore, we propose an infrared and visible image fusion algorithm based on the transformer module and adversarial learning. Inspired by the global interaction power, we use the transformer technique to learn the effective global fusion relations. In particular, shallow features extracted by CNN are interacted in the proposed transformer fusion module to refine the fusion relationship within the spatial scope and across channels simultaneously. Besides, adversarial learning is designed in the training process to improve the output discrimination via imposing competitive consistency from the inputs, reflecting the specific characteristics in infrared and visible images. The experimental performance demonstrates the effectiveness of the proposed modules, with superior improvement against the state-of-the-art, generalising a novel paradigm via transformer and adversarial learning in the fusion task.

Infrared and Visible Image Fusion Algorithm Based on Improved Residual Swin Transformer and Sobel Operators

Infrared and Visible Image Fusion Algorithm Based on Improved Residual Swin Transformer and Sobel Operators

Single image defogging via multi-exposure image fusion and detail enhancement

Outdoor cameras play an important role in monitoring security and social governance. As a common weather phenomenon, haze can easily affect the quality of camera shooting, resulting in loss and distortion of image details. This paper proposes an improved multi-exposure image fusion defogging technique based on the artificial multi-exposure image fusion (AMEF) algorithm. First, the foggy image is adaptively exposed, and the fused image is subsequently obtained via multiple exposures. The fusion weight is determined by the saturation, contrast, and brightness. Finally, the image fused by a multi-scale Laplacian algorithm is enhanced with simple adaptive details to obtain a clearer defogging image. It is subjectively and objectively verified that this algorithm can obtain more image details and distinct picture colors without a priori information, effectively improving the defogging ability.

Retracted: An Image Fusion Algorithm Based on Improved RGF and Visual Saliency Map.

[This retracts the article DOI: 10.1155/2022/1693531.].

Design of multi-objective guided filter infrared thermal image fusion for hypervelocity impact damages detection

Spacecraft hypervelocity impact damage assessment is becoming increasingly important as the space debris environment deteriorates. Common single-target guided filtering methods are hardly capable of coping with the complex damage infrared thermal image fusion requirements of hypervelocity impacts. In this paper, a two-layer infrared reconstructed image fusion algorithm based on Multi-objective guided filtering (MOGF) is proposed. The aim is to keep the filtered input and output images similar while reducing thermal image noise and retaining as much of the tiny defects as possible in the fusion process. MOGF combines a multi-objective evolutionary optimization algorithm to simultaneously optimize multiple guided filter cost functions in the form of vectors in order to obtain the optimal set of guided filter weight parameters. Given the problem of irregular frontier surfaces in practical multi-objective optimization problems, a weight vector adaptive adjustment strategy is introduced to obtain a more uniform population. Finally, the optimal weight set is passed to the full pixel fusion layer to complete the complex damage infrared reconstructed image fusion. The experiments demonstrate that the proposed fusion method considering multiple targets has higher performance in hypervelocity impact complex damage fusion compared to the general single target fusion method.

The Value and Realization of Choral Conducting Talent Cultivation for Music Teachers in the Context of Deep Learning

Abstract Focusing on virtual chorus, this paper analyzes the training process of VR technology and music choral conductor talents. After the construction of the online data platform, the image splicing, image alignment and image fusion algorithms of VR technology are integrated with the Harris corner detection improvement algorithm to realize the virtual chorus in the context of deep learning. The influence of family support and school aesthetic education on various dimensions of psychological needs for talent cultivation was modeled using structural equation modeling. Based on the data of music teachers using VR technology, mediated effect analysis and total effect analysis were conducted to analyze the factors influencing the use of imagery with VR technology. ANOVA was utilized to compare the data on the four dimensions of rhythmic memory, pitch, melodic memory, and listening ability to verify the effectiveness of VR technology on music teachers’ talent cultivation. The total effect analysis showed that the order of the influence of each influencing factor on the intention to use was 0.361>social influence 0.263>perceived usefulness 0.263>perceived ease of use 0.251>computer self-efficacy 0.169>convenience conditions 0.162>job relevance 0.134>simulation 0.075.

Research on the application of CNN algorithm based on chaotic recursive diagonal model in medical image processing

Abstract In this paper, the image processing capability of the CNN algorithm under the chaotic recursive diagonal model is explored from two aspects of medical image fusion and compression. By analyzing the structure of the chaotic recursive diagonal model, it is possible to combine it with a neural network. A convolutional neural network is used to automatically extract the focusing features of an image and output the probability of a pixel focusing. Combining the convolutional layer to extract image features with the activation function to nonlinearly map the feature map to achieve the effect of image fusion. Focusing on the exploration of the CNN algorithm for image fusion in image compression application processes. The results show that in the image fusion experiments, the CNN algorithm for image fusion data MI mean value is 6.1051, variance is 0.4418. QY mean value is 0.9859. The variance value is 0.0014. Compared to other algorithms, CNN in the image fusion effect has the effect of better distinguishing the edge details and making the appropriate decision. The CNN algorithm of the compression time is shorter. The time used in the compression of the X-chest image is 2.75s, which is 0.42 less than other algorithms. This study provides a new research perspective for medical image processing and is beneficial to improving the efficiency of medical image processing.

OMOFuse: An Optimized Dual-Attention Mechanism Model for Infrared and Visible Image Fusion

Infrared and visible image fusion aims to fuse the thermal information of infrared images and the texture information of visible images into images that are more in compliance with people’s visual perception characteristics. However, in the existing related work, the fused images have incomplete contextual information and poor fusion results. This paper presents a new image fusion algorithm—OMOFuse. At first, both the channel and spatial attention mechanisms are optimized by a DCA (dual-channel attention) mechanism and an ESA (enhanced spatial attention) mechanism. Then, an ODAM (optimized dual-attention mechanism) module is constructed to further improve the integration effect. Moreover, a MO module is used to improve the network’s feature extraction capability for contextual information. Finally, there is the loss function ℒ from the three parts of SSL (structural similarity loss), PL (perceptual loss), and GL (gap loss). Extensive experiments on three major datasets are performed to demonstrate that OMOFuse outperforms the existing image fusion methods in terms of quantitative determination, qualitative detection, and superior generalization capabilities. Further evidence of the effectiveness of our algorithm in this study are provided.

Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture

In this study, we introduce the U-Swin fusion model, an effective and efficient transformer-based architecture designed for the fusion of multi-focus microscope images. We utilized the Swin-Transformer with shifted window and path merging as the encoder for extracted hierarchical context features. Additionally, a Swin-Transformer-based decoder with patch expansion was designed to perform the un-sampling operation, generating the fully focused image. To enhance the performance of the feature decoder, the skip connections were applied to concatenate the hierarchical features from the encoder with the decoder up-sample features, like U-net. To facilitate comprehensive model training, we created a substantial dataset of multi-focus images, primarily derived from texture datasets. Our modulators demonstrated superior capability for multi-focus image fusion to achieve comparable or even better fusion images than the existing state-of-the-art image fusion algorithms and demonstrated adequate generalization ability for multi-focus microscope image fusion. Remarkably, for multi-focus microscope image fusion, the pure transformer-based U-Swin fusion model incorporating channel mix fusion rules delivers optimal performance compared with most existing end-to-end fusion models.

Comparative Analysis of Pixel-Level Fusion Algorithms and a New High-Resolution Dataset for SAR and Optical Image Fusion

Synthetic aperture radar (SAR) and optical images often present different geometric structures and texture features for the same ground object. Through the fusion of SAR and optical images, it can effectively integrate their complementary information, thus better meeting the requirements of remote sensing applications, such as target recognition, classification, and change detection, so as to realize the collaborative utilization of multi-modal images. In order to select appropriate methods to achieve high-quality fusion of SAR and optical images, this paper conducts a systematic review of current pixel-level fusion algorithms for SAR and optical image fusion. Subsequently, eleven representative fusion methods, including component substitution methods (CS), multiscale decomposition methods (MSD), and model-based methods, are chosen for a comparative analysis. In the experiment, we produce a high-resolution SAR and optical image fusion dataset (named YYX-OPT-SAR) covering three different types of scenes, including urban, suburban, and mountain. This dataset and a publicly available medium-resolution dataset are used to evaluate these fusion methods based on three different kinds of evaluation criteria: visual evaluation, objective image quality metrics, and classification accuracy. In terms of the evaluation using image quality metrics, the experimental results show that MSD methods can effectively avoid the negative effects of SAR image shadows on the corresponding area of the fusion result compared with CS methods, while model-based methods exhibit relatively poor performance. Among all of the fusion methods involved in the comparison, the non-subsampled contourlet transform method (NSCT) presents the best fusion results. In the evaluation using image classification, most experimental results show that the overall classification accuracy after fusion is better than that before fusion. This indicates that optical-SAR fusion can improve land classification, with the gradient transfer fusion method (GTF) yielding the best classification results among all of these fusion methods.

An image fusion-based method for recovering the 3D shape of roll surface defects

Most of the existing studies on roll surface defects focus on qualitative detection and lack quantitative analysis, while the commonly used methods for detecting the three-dimensional shape of small objects such as defects are the stylus method, laser scanning method, and structured light scanning method, but these methods are difficult to accurately measure the complex defect variations on the roll surface. In this paper, we propose a method for recovering the 3D shape of roll surface defects based on image fusion. The traditional 3D reconstruction problem is transformed into a 2D image fusion problem using a focusing method. The non-subsampled shear wave transform is used as the base algorithm for image fusion, combined with an enhanced fusion strategy called modified multi-state pulse-coupled neural network to obtain a fully focused image. The method achieves 3D shape recovery of defects by modeling the relationship between the defect depth, the fully focused image, and the original image. To evaluate the performance of the method, experiments were carried out using data involving craters and scratches on the roll surface. This method significantly improves the quality of defect detection images, with a 98% better gradient and a 28% increase in overall image quality. Additionally, it keeps 3D reconstruction errors under 4%, ensuring high accuracy and noise resistance.

Study on InSAR image fusion for improved visualization of active landslides in alpine valley areas: A case in the Batang Region, China

InSAR observation with a wide range and high sensitivity is one of the main techniques for active landslide identification, but due to the influence of the SAR satellite side-view imaging method and observation angle, single-track images in alpine valley areas facing geometric distortion, such as shadowing, layover, and foreshortening, which leads to ineffective observation in some areas. Combining the ascending and descending observation results is expected to compensate for this deficiency to a large extent. This work constructs a pixel-level image fusion method to achieve the fusion of InSAR ascending and descending orbit deformation results, consisting of three algorithms: the DMPI (decision matrix-guided pixel-level InSAR image fusion algorithm), the DMFS (deformable median filter based on decision feedback), and the QSDS (superpixel-based dual-space InSAR image fusion quality evaluation algorithm). To verify the effect, the Batang section of the Jinsha River with high mountain canyon terrain and frequent active landslides is selected. The results show that using the DMPI fused InSAR ascending and descending orbit images can improve spatial perception by enhancing the texture gradient value of remote sensing images by a maximum of two orders of magnitude, compared with the single-track InSAR observation images. Meanwhile, the number of landslides identified increased by 31.5%, and the total area increased by 50.9% on average, improving the accuracy of target detection and identification. In noise reduction, DMFS improves fusion image readability and feature detail by removing SPN (salt and pepper noise) while preserving target edge detail. Furthermore, the QSDS which can evaluate the quality of InSAR fusion images from the perspective of human eye vision, provides an adequate basis for determining the interpretability of InSAR images. This fusion method can partially improve the question of incomplete ground observation due to radar line of sight occlusion in the high mountain canyon area, thus better presenting the InSAR observation results and helping to comprehensively and conveniently interpret the InSAR deformers.

Multi-modal medical image fusion via multi-dictionary and truncated Huber filtering

Multi-modal medical image fusion provides comprehensive and objective descriptions of lesions for clinical medical assistance. However, retaining useful information while achieving noise robustness remains challenging for existing techniques. In this paper, we propose a novel medical image fusion algorithm based on multi-dictionary convolutional sparse representation. Especially, truncated Huber filtering is first introduced to achieve detail-base layer decomposition of source images. Subsequently, multiple-dictionary decisions and nuclear energy-based rules are proposed to fuse the details and base layers, respectively. The fused image is reconstructed by synthesizing the fused detail and base components. The proposed model effectively fuses the source global structure and texture information and exhibits strong robustness against noise. Experiments involving extensive noise-free and noisy anatomical and functional medical image fusion on a public dataset covering five fusion categories demonstrate that the proposed method outperforms other state-of-the-art methods in subjective and objective evaluations. The source code of this study is publicly available at https://github.com/JEI981214/MDHU-fusion.

Performance comparison of different medical image fusion algorithms for clinical glioma grade classification with advanced magnetic resonance imaging (MRI)

Non-invasive glioma grade classification is an exciting area in neuroimaging. The primary purpose of this study is to investigate the performance of different medical image fusion algorithms for glioma grading purposes by fusing advanced Magnetic Resonance Imaging (MRI) images. Ninety-six subjects underwent an Apparent diffusion coefficient (ADC) map and Susceptibility-weighted imaging (SWI) MRI scan. After preprocessing, the different medical image fusion methods used to fuse ADC maps and SWI were Principal Component Analysis (PCA), Structure-Aware, Discrete Cosine Harmonic Wavelet Transform (DCHWT), Deep-Convolutional Neural network (DNN), Dual-Discriminator conditional generative adversarial network (DDcGAN), and Laplacian Re-Decomposition (LRD). The Entropy, standard deviation (STD), peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and Relative Signal Contrast (RSC) were calculated for qualitative and quantitative analysis. We found high fused image quality with LRD and DDcGAN methods. Further quantitative analysis showed that RSCs in fused images in Low-Grade glioma (LGG) were significantly higher than RSCs in High-Grade glioma (HGG) with PCA, DCHWT, LRD, and DDcGAN. The Receiver Operating Characteristic (ROC) curve test highlighted that LRD and DDcGAN have the highest performance for glioma grade classification. Our work suggests using the DDcGAN and LRD networks for glioma grade classification by fusing ADC maps and SWI images.

Co-saliency detection with two-stage co-attention mining and individual calibration

Learning-based methods have become popular in co-salient object detection (CoSOD). However, existing methods suffer from two challenging issues, including mining the inter-image co-attention and calibrating the intra-image salient objects. Moreover, the training data is insufficient. To address these challenges, we propose an end-to-end network using Two-stage Co-attention mining and Individual Calibration (TCIC) to predict the co-salient objects. Firstly, a two-stage co-attention mining architecture (TCM), including a classified co-attention module (CCM) and a focal co-attention module (FCM), is designed to model inter-image relationships. In the first stage, a CCM is applied to capture the classification interactions of multiple images, tentatively extracting the co-attention. In the second stage, we propose an FCM to adaptively suppress and aggregate multiple salient features, aiming to recalibrate the co-attention in the first stage. Secondly, considering the shape features and location information offered by the boundary features, an edge guidance module (EGM) is embedded into the individual calibration architecture (ICA) to calibrate individuals. Besides, we also adopt a co-attention transfer strategy (CTS) to keep the consistency of the co-attention during feature transfer in the decoder. Finally, TCM and ICA are integrated into a unified end-to-end framework to predict fine-grained boundary-preserving results. Besides, an image fusion algorithm (IFA) is tailored without extra pixel-level annotations for automatic generation of the composite images, aiming to supplement the training dataset. Experimental results on three prevailing testing datasets show the superiority of the proposed method in terms of various evaluation metrics.

A three-layer decomposition method based on structural texture perception for fusion of CT and MRI images

The fusion of computed tomography (CT) images and magnetic resonance imaging (MRI) images has made essential contribution to clinical diagnosis because of its ability of complementary visual information enhancement and redundancy elimination. However, some methods that involve upsampling and downsampling operations may lose information during image processing, which affects the fusion results. In this paper, a medical image fusion algorithm based on structural texture perception is proposed. The proposed method demonstrates enhanced performance in preserving both detailed texture information and energy information of source images. First, the source image is decomposed into a signal strength layer, a base layer, and a texture layer based on the proposed three-layer decomposition framework. Then, the signal strength layers are fused using a saliency detection method based on iterative least squares. The texture layer is fused using the principle of spatial frequency maximization. For the fusion of the base layer, a non-linear function is designed to calculate the fusion weights. Finally, the final fusion result is obtained through image reconstruction. The proposed algorithm is compared with nine state-of-the-art fusion algorithms to verify its superiority. Experimental results show that the proposed method can effectively preserve the intensity information of CT images and the detailed texture of MRI images.

SW-UNet: a U-Net fusing sliding window transformer block with CNN for segmentation of lung nodules.

Medical images are information carriers that visually reflect and record the anatomical structure of the human body, and play an important role in clinical diagnosis, teaching and research, etc. Modern medicine has become increasingly inseparable from the intelligent processing of medical images. In recent years, there have been more and more attempts to apply deep learning theory to medical image segmentation tasks, and it is imperative to explore a simple and efficient deep learning algorithm for medical image segmentation. In this paper, we investigate the segmentation of lung nodule images. We address the above-mentioned problems of medical image segmentation algorithms and conduct research on medical image fusion algorithms based on a hybrid channel-space attention mechanism and medical image segmentation algorithms with a hybrid architecture of Convolutional Neural Networks (CNN) and Visual Transformer. To the problem that medical image segmentation algorithms are difficult to capture long-range feature dependencies, this paper proposes a medical image segmentation model SW-UNet based on a hybrid CNN and Vision Transformer (ViT) framework. Self-attention mechanism and sliding window design of Visual Transformer are used to capture global feature associations and break the perceptual field limitation of convolutional operations due to inductive bias. At the same time, a widened self-attentive vector is used to streamline the number of modules and compress the model size so as to fit the characteristics of a small amount of medical data, which makes the model easy to be overfitted. Experiments on the LUNA16 lung nodule image dataset validate the algorithm and show that the proposed network can achieve efficient medical image segmentation on a lightweight scale. In addition, to validate the migratability of the model, we performed additional validation on other tumor datasets with desirable results. Our research addresses the crucial need for improved medical image segmentation algorithms. By introducing the SW-UNet model, which combines CNN and ViT, we successfully capture long-range feature dependencies and break the perceptual field limitations of traditional convolutional operations. This approach not only enhances the efficiency of medical image segmentation but also maintains model scalability and adaptability to small medical datasets. The positive outcomes on various tumor datasets emphasize the potential migratability and broad applicability of our proposed model in the field of medical image analysis.

Multispectral and Hyperspectral Image Fusion Based on Joint-Structured Sparse Block-Term Tensor Decomposition

Multispectral and hyperspectral image fusion (MHF) aims to reconstruct high-resolution hyperspectral images by fusing spatial and spectral information. Unlike the traditional canonical polyadic decomposition and Tucker decomposition models, the block-term tensor decomposition model is able to improve the quality of fused images using known endmember and abundance information. This paper presents an improved hyperspectral image fusion algorithm. Firstly, the two abundance matrices are combined into a single bulk matrix to promote structural sparsity by introducing the L2,1-norm to eliminate the scaling effects present in the model. Secondly, the counter-scaling effect is eliminated by adding the L2-norm to the endmember matrix. Finally, the chunk matrix and the endmember matrix are coupled together, and the matrix is reorganized by adding the L2,1-norm to the matrix to facilitate chunk elimination and solved using an extended iterative reweighted least squares (IRLS) method, focusing on the problem of the inability to accurately estimate the tensor rank in the chunk-term tensor decomposition model and the noise/artifact problem arising from overestimation of rank. Experiments are conducted on standard and local datasets, and the fusion results are compared and analyzed in four ways: visual result analysis, metric evaluation, time of the algorithm, and classification results, and the experimental results show that the performance of the proposed method is better than the existing methods. An extensive performance evaluation of the algorithms is performed by conducting experiments on different datasets. The experimental results show that the proposed algorithm achieves significant improvements in terms of reconstruction error, signal-to-noise ratio, and image quality compared with the existing methods. Especially in the case of a low signal-to-noise ratio, the proposed algorithm shows stronger robustness and accuracy. These results show that the proposed algorithm has significant advantages in dealing with multispectral high-resolution hyperspectral data.

Image Fusion Algorithm using Grey Wolf optimization with Shuffled Frog Leaping Algorithm

Data fusion is a “formal framework in which are expressed the means and tools for the alliance of data originating from different sources.” It aims at obtaining information of greater quality; the exact definition of 'greater quality will depend upon the application. It is a famous technique in digital image processing and is very important in medical image representation for clinical diagnosis. Previously many researchers used many meta-heuristic optimization techniques in image fusion, but the problem of local optimization restricted their searching flow to find optimum search results. In this paper, the Grey Wolf Optimization (GWO) algorithm with the help of the Shuffled Frog Leaping Algorithm (SFLA) has been proposed. That helps to find the object and allows doctors to take some action. The optimization algorithm is examined with a demonstrated example in order to simplify its steps. The result of the proposed algorithm is compared with other optimization algorithms. The proposed method's performance was always the best among them.

P07.04.A ELASTIC IMAGE FUSION OF INTRAOPERATIVE ULTRASOUND AND PREOPERATIVE MRI FOR IMPROVING NAVIGATION ACCURACY IN BRAIN TUMOR SURGERY

Abstract BACKGROUND In brain surgery, neuronavigation based on preoperative MRI is limited by various physical and operational factors, such as surgery-induced brain shift. Using navigated intraoperative ultrasound (iUS) imaging and 3D iUS image reconstruction may help improve neuronavigation. However, the registration of MRI and US scans can lead to spatial inconsistencies between the two modalities. This study aims to evaluate the clinical integration of navigated iUS imagining and its potential to improve the accuracy of resection during brain tumor surgery by comparing the performance of rigid and elastic MRI-iUS image fusion algorithms during retrospective evaluation. MATERIAL AND METHODS N = 50 patients with brain tumors were prospectively enrolled in this trial. The neuronavigation system (BrainLab AG) used in the study allowed for the reconstruction and integration of 3D iUS imaging, which was acquired multiple times throughout the surgery using a tracked ultrasound probe (BK Medical). Prototypes of automatic rigid and elastic MRI-iUS image fusion algorithms (Brainlab Elements Image Fusion) were retrospectively applied to improve co-registration accuracy between the MRI and iUS data. To evaluate the image co-registration accuracy, anatomical landmarks were defined in the MRI and iUS scans and used to measure Euclidean distances between each landmark pair. RESULTS Improvements in the image quality over the course of the study indicated that skills in US handling are essential. Contrast behavior and tumor structure influence the visibility of the tumor in the US. Artifacts caused by hemostatic agents challenge the US image quality and the fluid-filled resection cavity requires experience in probe handling. In terms of image fusion, significant mismatches of the iUS and MRI data were reported. For scans acquired before resection, rigid fusion using a novel co-registration algorithm may enhance the registration accuracy with mean target registration errors for two representative patients of 6.5/8.9 mm and 3.7/3.4 mm before and after registration update, respectively. For iUS scans acquired after tumor resection, rigid fusion is inherently limited due to non-linear brain shifts. Therefore, elastic image fusion has been applied to provide deformable MR-iUS image co-registration. CONCLUSION Navigated iUS imaging can be integrated into clinical routine and the use of MR-iUS image co-registration for updating navigation is a promising tool to compensate brainshift during surgical resection. However, further studies are needed to quantify the accuracy and assess whether iUS could potentially replace the use of iMRI in resection control of selected cases.