Image Fusion Research Articles

SDFuse: Semantic-injected dual-flow learning for infrared and visible image fusion

Infrared and visible image fusion (IVIF) strives to render fused results that preserve the strengths of the source images (e.g., texture details and thermal highlights) while boosting downstream tasks. However, existing semantic-driven methods typically combine segmentation and fusion tasks via cascade architecture, which fails to sufficiently exploit semantic knowledge. To address this limitation, an innovative semantic-injected network with dual flows is proposed for IVIF, called SDFuse. Concretely, we first apply two pseudo-Siamese feature extraction modules (FEM) to obtain shallow structure features, which involve skip connections to prevent the disappearance of information. Subsequently, dual flows are constructed to support the joint learning paradigm of two tasks. The segmentation flow excavates abundant semantic knowledge from shallow features through fulfilling high-level segmentation task. The fusion flow is introduced to achieve the aggregation of highlights and textures. Note that SGM in fusion flow effectuates the injection of deep semantic knowledge into shallow structural features. Eventually, the region consistency module (RCM) with group normalization leads the network to focus on foreground objects and generate context-consistent fusion images. Our fusion results not only effectively fuse the information of the source image but also have advantages in downstream tasks. Extensive experiments over multiple representative benchmarks prove that SDFuse outperforms current state-of-the-art approaches.

SAR Image Fusion Classification Based on Improved D-S Evidence Theory

Applications for classifying Synthetic Aperture Radar (SAR) images are critical to environmental monitoring, urban planning, and land resource surveying. Fusion approaches work well for increasing SAR image categorization accuracy. However, effective processing of uncertainty information is frequently overlooked by conventional fusion classification systems. This paper presented a fusion classification approach for SAR images based on improved Dempster-Shafer (D-S) evidence theory, taking into account the uncertainty and possible conflicts in the classifiers’ output. First, after adaptively identifying the optimal hyperparameters of models, the SAR data is identified using Gradient Boosting Machine (GBM), Multilayer Perceptron (MLP), and Random Forest (RF). Different weights are then allocated in accordance with the variations in classifier performance. Ultimately, enhanced D-S evidence theory is used to integrate the output of these classifiers. In addition, uncertainty in classification results is also modeled and visualized. Experiments with Flevoland and Oberpfaffenhofen datasets show that the accuracy of this method is 85.21% and 91.88% respectively, outperforming both the soft voting ensemble classifier (SVE) and D-S evidence theory fusion method. This approach enhances the comprehension and capacity to manage uncertainty in the model output in addition to increasing the accuracy and reliability of the classification of SAR data.

An unsupervised multi-focus image fusion method via dual-channel convolutional network and discriminator

The challenge in multi-focus image fusion tasks lies in accurately preserving the complementary information from the source images in the fused image. However, existing datasets often lack ground truth images, making it difficult for some full-reference loss functions (such as SSIM) to effectively participate in model training, thereby further affecting the performance of retaining source image details. To address this issue, this paper proposes an unsupervised dual-channel dense convolutional method, DCD, for multi-focus image fusion. DCD designs Patch processing blocks specifically for the fusion task, which segment the source image pairs into equally sized patches and evaluate their information to obtain a reconstructed image and a set of adaptive weight coefficients. The reconstructed image is used as the reference image, enabling unsupervised methods to utilize full-reference loss functions in training and overcoming the challenge of lacking labeled data in the training set. Furthermore, considering that the human visual system (HVS) is more sensitive to brightness than color, DCD trains the dual-channel network using both RGB images and their luminance components. This allows the network to focus more on the brightness information while preserving the color and gradient details of the source images, resulting in fused images that are more compatible with the HVS. The adaptive weight coefficients obtained through the Patch processing blocks are also used to determine the degree of preservation of the brightness information in the source images. Finally, comparative experiments on different datasets also demonstrate the superior performance of DCD in terms of fused image quality compared to other methods.

Deep M2CDL: Deep Multi-Scale Multi-Modal Convolutional Dictionary Learning Network.

For multi-modal image processing, network interpretability is essential due to the complicated dependency across modalities. Recently, a promising research direction for interpretable network is to incorporate dictionary learning into deep learning through unfolding strategy. However, the existing multi-modal dictionary learning models are both single-layer and single-scale, which restricts the representation ability. In this paper, we first introduce a multi-scale multi-modal convolutional dictionary learning ( M2CDL) model, which is performed in a multi-layer strategy, to associate different image modalities in a coarse-to-fine manner. Then, we propose a unified framework namely Deep M2CDL derived from the M2CDL model for both multi-modal image restoration (MIR) and multi-modal image fusion (MIF) tasks. The network architecture of Deep M2CDL fully matches the optimization steps of the M2CDL model, which makes each network module with good interpretability. Different from handcrafted priors, both the dictionary and sparse feature priors are learned through the network. The performance of the proposed Deep M2CDL is evaluated on a wide variety of MIR and MIF tasks, which shows the superiority of it over many state-of-the-art methods both quantitatively and qualitatively. In addition, we also visualize the multi-modal sparse features and dictionary filters learned from the network, which demonstrates the good interpretability of the Deep M2CDL network.

Privacy-preserving federated transfer learning for defect identification from highly imbalanced image data in additive manufacturing

Defect identification is a crucial task for process monitoring and quality evaluation in additive manufacturing (AM). Deep learning (DL) has shown great potential for diverse fields, but some challenges have hindered the application in AM process monitoring. Firstly, DL-based methods are driven by big data and require a large number of training data. However, in reality, defective data is often rare, so different AM manufacturers can collaborate to train a global model for detecting defects. Nevertheless, due to privacy concerns and conflicts of interest, data-sharing between different manufacturers might be dangerous. Additionally, the heterogeneities among manufacturers’ data leads to the domain shift, making it difficult to obtain a well-generalized model. Moreover, the imbalance issue of powder-spreading defects seriously damages the performance of the defect recognition. In this paper, we proposed FTLAM, a federated transfer learning method, to address the above issues. Concretely, to solve the data privacy concerns, federated learning (FL) is utilized to collaboratively train a global model between different AM manufacturers without exchanging original data. Furthermore, a client-oriented transfer method is proposed to mitigate the heterogeneities across multiple clients and improve the model's generalization. Meanwhile, a multi-loss joint optimization approach is designed to alleviate the data imbalance. Extensive experiments on laser powder bed fusion (LPBF) image datasets of powder-spreading have demonstrated that our FTLAM can obtain satisfactory performance of powder spreading defect identification.

Multimodal image fusion on ECG signals for congestive heart failure classification

Multimodal image fusion on ECG signals for congestive heart failure classification

Measurement method for live chicken shank length based on improved ResNet and fused multi-source information

Phenotypic parameters are crucial reference indicators in poultry breeding. However, the chickens shank length is still manually measured, which is time-consuming and labor-intensive. Additionally, the measurement results are difficult to unify due to the subjective factors of different individuals. To address this issue, this paper proposed a method for live chicken shank length measurement (SLM). It enriches chicken shank feature by fusing visible images and infrared images. The fusion images are then input into a deep regression model based on the improved ResNet. The measurement model used ResNet as its backbone and introduces Squeeze-and-Excitation (SE) blocks and a Spatial Pyramid Pooling (SPP) block, resulting in more precise and stable shank length measurements. The average coefficient of variation, average floating error, average standard deviation and Pearson correlation coefficient for shank length measurements using the fusion images are 0.21 %, 0.49 %, 0.181 mm and 0.996, respectively, compared with using single visible or infrared image, the accuracy and stability are obviously improved. That indicated combining deep learning model and fusion information, the SLM proposed in this paper can achieve a more precise, reliable and standardized measurement of live chicken shank length.

LRFE-CL: A self-supervised fusion network for infrared and visible image via low redundancy feature extraction and contrastive learning

Aiming to amalgamate the distinct yet complementary attributes inherent to infrared and visible spectra, Infrared and Visible Image Fusion (IVIF) endeavors to preserve an optimal intensity distribution while curtailing the superfluous data within the resultant image. Nonetheless, it is an incontrovertible reality that the majority of extant methods are prone to engendering an undesirable redundancy of features, in addition to a marked degradation throughout the fusion process. To address the issue, we proposed a self-supervised fusion network for infrared and visible image via low redundancy feature extraction and contrastive learning, termed as LRFE-CL. Specifically, the Low Redundancy Feature Extraction Network (SSFN-LRAE) has been developed to effectively extract low-redundancy features from infrared and visible images using the self-supervised strategy and the Channel Shift Attention Module (CSAM). Subsequently, the Contrastive Fusion Decoder (CFDN) is designed to fuse multi-layer features extracted from the SSFN-LRAE. To prevent feature degradation, a novel regularized contrastive loss mechanism targeting multi-layer features has been formulated to optimize our CFDN, ensuring the fusion of comprehensive texture details and brightness information. Our comprehensive experimental results unequivocally demonstrate the exceptional performance of our method, both in terms of visual effects and fusion performance.

Novel Entropy for Enhanced Thermal Imaging and Uncertainty Quantification

This paper addresses the critical need for precise thermal modeling in electronics, where temperature significantly impacts system reliability. We emphasize the necessity of accurate temperature measurement and uncertainty quantification in thermal imaging, a vital tool across multiple industries. Current mathematical models and uncertainty measures, such as Rényi and Shannon entropies, are inadequate for the detailed informational content required in thermal images. Our work introduces a novel entropy that effectively captures the informational content of thermal images by combining local and global data, surpassing existing metrics. Validated by rigorous experimentation, this method enhances thermal images’ reliability and information preservation. We also present two enhancement frameworks that integrate an optimized genetic algorithm and image fusion techniques, improving image quality by reducing artifacts and enhancing contrast. These advancements offer significant contributions to thermal imaging and uncertainty quantification, with broad applications in various sectors.

HyperFace: A Deep Fusion Model for Hyperspectral Face Recognition.

Face recognition has been well studied under visible light and infrared (IR) in both intra-spectral and cross-spectral cases. However, how to fuse different light bands for face recognition, i.e., hyperspectral face recognition, is still an open research problem, which has the advantages of richer information retention and all-weather functionality over single-band face recognition. Thus, in this research, we revisit the hyperspectral recognition problem and provide a deep learning-based approach. A new fusion model (named HyperFace) is proposed to address this problem. The proposed model features a pre-fusion scheme, a Siamese encoder with bi-scope residual dense learning, a feedback-style decoder, and a recognition-oriented composite loss function. Experiments demonstrate that our method yields a much higher recognition rate than face recognition using only visible light or IR data. Moreover, our fusion model is shown to be superior to other general-purpose image fusion methods that are either traditional or deep learning-based, including state-of-the-art methods, in terms of both image quality and recognition performance.

Improving Spatial Resolution Of Satellite Image Using Data Fusion Method

While many remote sensing and GIS applications require both the spatialresolution and spectral resolution be high, image fusion, or in other words, imagesharpening, is a useful technique. To date, numerous image fusion techniques havebeen developed to combine the clear geometric features of the panchromatic imageand the color information of the multispectral image. This paper compares theresults of four different pixel based fusion techniques, HLS, Broveytransformation,Gram–Schmidt and HSV techniques used to merge the ETM+multispectral image with (28.5 m)spatial resolution and panchromatic imagecaptured by SPOT satellite with (10m) spatial resolution, correlation coefficient ,root mean square error and peak signal to noise ratio criteria have been used toachieve the comparison among the using techniques

High-intensity focused ultrasound strategies for treating prostate cancer

Abstract Prostate cancer (PCa) is a significant health concern globally, necessitating effective treatment options. Typical treatment methods for early stage, particularly localized PCa, encompass radical procedures, such as radical prostatectomy (RP) and radiotherapy (RT), and nonradical focal therapy (FT). FT is a focused approach mainly used for treating small lesions limited to a specific zone of the prostate. Its objective is to achieve cancer control when minimizing damage to benign tissue. High-intensity focused ultrasound (HIFU) is one of the most used modalities in FT for the management of PCa. The progress in HIFU technology showcases continuous advancements, offering clinicians a variety of strategies to cater to diverse patient requirements. The advancements include the development of transrectal and transurethral HIFU machines that offer enhanced treatment distances, magnetic resonance imaging (MRI) fusion capabilities, real-time monitoring, and precise ablation. These improvements contribute to increased treatment effectiveness and better outcomes for patients. This narrative review aims to summarize the use of HIFU technology and its evolution, offering diverse options to clinicians, and explores the safety, effectiveness, and quality of different HIFU strategies, such as whole-gland ablation, hemigland ablation, and focal ablation. We conclude that nonwhole-gland HIFU offers similar cancer control with better short-term functional outcomes and fewer complications compared to whole-gland ablation. Combining HIFU with transurethral resection of the prostate (TURP) improves urinary function and reduces catheterization time. Focal ablation and hemigland ablation show promise in achieving cancer control when preserving continence and potency.

Bayesian fusion of multispectral and panchromatic images using a multi-mode and multiorder gradient tensor-based l 1 / 2 sparse model

ABSTRACT In this letter, based on the tensor representation modelling, we propose a multi-mode and multi-order gradient tensor-based non-convex model (M2GTNM) for Bayesian fusion of multispectral (MS) and panchromatic (Pan) images, which aims at producing the high-resolution MS (HRMS) images. Specifically, by modelling the MS image as the order-3 tensor, we mainly develop the multi-mode and multi-order gradient tensor sparse priors of MS image for fusion. For the spectral preservation of low-resolution MS (LRMS) image, the spectral fidelity constraint between HRMS and LRMS images is imposed. For the spatial-mode prior modelling, the multi-order spatial gradient tensor-based non-convex l 1 / 2 sparse prior between HRMS and Pan images is particularly imposed. Moreover, for the spectral-mode prior modelling, the spectral gradient tensor-based non-convex l 1 / 2 sparse prior between HRMS and upsampled LRMS images is further imposed. Then, we apply the alternating direction method of multipliers to optimize the proposed model. Finally, the reduced-scale and full-scale fusion experiments both validate the effectiveness of M2GTNM method.

Concrete Highway Crack Detection Based on Visible Light and Infrared Silicate Spectrum Image Fusion.

Cracks provide the earliest and most immediate visual response to structural deterioration of asphalt pavements. Most of the current methods for crack detection are based on visible light sensors and convolutional neural networks. However, such an approach obviously limits the detection to daytime and good lighting conditions. Therefore, this paper proposes a crack detection technique cross-modal feature alignment of YOLOV5 based on visible and infrared images. The infrared spectrum characteristics of silicate concrete can be an important supplement. The adaptive illumination-aware weight generation module is introduced to compute illumination probability to guide the training of the fusion network. In order to alleviate the problem of weak alignment of the multi-scale feature map, the FA-BIFPN feature pyramid module is proposed. The parallel structure of a dual backbone network takes 40% less time to train than a single backbone network. As determined through validation on FLIR, LLVIP, and VEDAI bimodal datasets, the fused images have more stable performance compared to the visible images. In addition, the detector proposed in this paper surpasses the current advanced YOLOV5 unimodal detector and CFT cross-modal fusion module. In the publicly available bimodal road crack dataset, our method is able to detect cracks of 5 pixels with 98.3% accuracy under weak illumination.

Dual-band color fusion image quality assessment based on color descriptors

Dual-band color fusion image quality assessment based on color descriptors

Discrete Wavelet Transformation based Multimodal Medical Image Fusion for Disease Identification

In medical science, image processing techniques play a significant function. Computational automation of the treatment is the most authentic and prominent method. The disease of the brain is identified using Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET). Many more scan variations of MRI and PET have been executed for the medical diagnosis. The medical expert needs a solid strain of the computational scan and it’s related for diagnosis. The current era of computer research is turning towards clinical diagnosis and etiological analysis based on multimodal image processing. Based on individual medical modalities the accuracy of diseases diagnosis is decreases. Medical community requires a high accuracy in the disease’s identification based on multimodal scan image data. For the diagnosis and treatment of disorders requires precise information that is attained through various modalities of medical images such as Computed Tomography (CT), Positron Emission Tomography (PET), and Magnetic Resonance Imaging (MRI). In image processing the image fusion is the method of merging two images into a single picture. The obtained single fused image using various multi-modality medical images is enhanced anatomical, highly desirable spectral information compared to the raw single scanned image. This multi – modal fused image is useful for clinical diagnosis of medical experts. In this research work, the system is prepared for the preprocessing of the MRI and PET scan images. The pre-processing techniques enhance the quality of the input images which are degraded and non-readable. For the pre-processing approach we have applied the Gaussian filters of spatial filtering techniques. The enhanced images passed to the fusion of different region of brain images using Discrete Wavelet Transform (DWT). The system achieved around 90- 95% more accurate outcomes by diluting the color change. The outcome fused image is achieved without losing the spectral and anatomical data. The experiment has been tested on Alzheimer’s, normal axis, and normal coronal brain disease images dataset. The quantitative and graphical analysis indicates that’s the Discrete Wavelet Transform (DWT) significantly improves the quality of fused images.

Semantic-aware normalizing flow with feature fusion for image anomaly detection

In the field of computer vision, anomaly detection is a binary classification task used to identify exceptional instances within image datasets. Typically, it can be divided into two aspects: texture defect detection and semantic anomaly detection. Existing methods often use pre-trained feature extractors to singly capture semantic or spatial features of images, and then employ different classifiers to handle these two types of anomaly detection tasks. However, these methods fail to fully utilize the synergistic relationship between these two types of features, resulting in algorithms that excel in one type of anomaly detection task but perform poorly in the other type. Therefore, we propose a novel approach that successfully combines these two types of features into a normalizing flow learning module to address both types of anomaly detection tasks. Specifically, we first adopt a pre-trained Vision Transformer (ViT) model to capture both texture and semantic features of input images. Subsequently, using the semantic features as input, we design a novel normalizing flow model to fit the semantic distribution of normal data. In addition, we introduce a feature fusion module based on attention mechanisms to integrate the most relevant texture and semantic information between these two types of features, significantly enhancing the model’s ability to simultaneously represent the spatial texture and semantic features of the input image. Finally, We conduct comprehensive experiments on well-known semantic and texture anomaly detection datasets, namely Cifar10 and MVTec, to evaluate the performance of our proposed method. The results demonstrate that our model achieves outstanding performance in both semantic and texture anomaly detection tasks, particularly achieving state-of-the-art results in semantic anomaly detection.

Multi-factor weighted image fusion method for high spatiotemporal tracking of reservoir drawdown area and its vegetation dynamics

Reservoir drawdown areas (RDAs) with distinct dry-wet cycles and vegetation dynamics have emerged as significant hotspots for carbon-related activities. However, high-resolution spatiotemporal tracking of the variations and vegetation dynamics of RDAs remains challenging because they often change dramatically and are controlled by both human activities and natural factors. Herein, a modified image fusion method was proposed to capture rapid variations in RDAs by integrating impact factor information into the analysis. The capability of the proposed method was tested in the Danjiangkou (DJK) Reservoir as it is the largest artificial freshwater lake in Asia with a highly variable RDA, since it is surrounded by gently sloping plains or hills. The results showed that the modified workflow produced reliable predictions (r=0.83,RMSE=0.097) compared with the original Enhanced Spatial and Temporal Adaptive Reflectance Fusion Model (ESTARFM) workflow (r=0.60,RMSE=0.195), demonstrating improved capability for mapping water surface changes and vegetation dynamics. Using the proposed method, the 15-d variations in the RDA were derived from 2013 to 2022 with a 30-m resolution. The interannual maximum RDA was estimated to be 278 km2 after the dam was elevated in 2013. The Normalized Difference Vegetation Index (NDVI) decreased as inundation frequency (IF) increased. Mean NDVI in the growing season (May–October) decreased by 0.109 (17.6 %) and 0.156 (26.0 %) under 30 %–40 % IF and 60 %–70 % IF, respectively, compared with the vegetation under 0 %–10 % IF, which was referred as “natural” vegetation considering its rare inundation. Moreover, the mean growing season length decreased to only 63 and 19 d for 30 %–40 % IF and 60 %–70 % IF, respectively. Furthermore, 77.3 % of the RDA exhibited a decrease in NDVI, whereas 22.7 % showed an unusual increase, possibly due to the selection of dominant species well-adapted to inundation during vegetation succession. Overall, this study not only proposed a new method for the high spatiotemporal monitoring of RDAs, but also highlighted the importance of variations and vegetation dynamics within RDAs for accurate estimation of their carbon budgets.

Unpaired high-quality image-guided infrared and visible image fusion via generative adversarial network

Current infrared and visible image fusion (IVIF) methods lack ground truth and require prior knowledge to guide the feature fusion process. However, in the fusion process, these features have not been placed in an equal and well-defined position, which causes the degradation of image quality. To address this challenge, this study develops a new end-to-end model, termed unpaired high-quality image-guided generative adversarial network (UHG-GAN). Specifically, we introduce the high-quality image as the reference standard of the fused image and employ a global discriminator and a local discriminator to identify the distribution difference between the high-quality image and the fused image. Through adversarial learning, the generator can generate images that are more consistent with high-quality expression. In addition, we also designed the laplacian pyramid augmentation (LPA) module in the generator, which integrates multi-scale features of source images across domains so that the generator can more fully extract the structure and texture information. Extensive experiments demonstrate that our method can effectively preserve the target information in the infrared image and the scene information in the visible image and significantly improve the image quality.

Breast cancer lesion segmentation based on co-learning feature fusion and Transformer

The PET/CT imaging technology combining positron emission tomography (PET) and computed tomography (CT) is the most advanced imaging examination method currently, and is mainly used for tumor screening, differential diagnosis of benign and malignant tumors, staging and grading. This paper proposes a method for breast cancer lesion segmentation based on PET/CT bimodal images, and designs a dual-path U-Net framework, which mainly includes three modules: encoder module, feature fusion module and decoder module. Among them, the encoder module uses traditional convolution for feature extraction of single mode image; The feature fusion module adopts collaborative learning feature fusion technology and uses Transformer to extract the global features of the fusion image; The decoder module mainly uses multi-layer perceptron to achieve lesion segmentation. This experiment uses actual clinical PET/CT data to evaluate the effectiveness of the algorithm. The experimental results show that the accuracy, recall and accuracy of breast cancer lesion segmentation are 95.67%, 97.58% and 96.16%, respectively, which are better than the baseline algorithm. Therefore, it proves the rationality of the single and bimodal feature extraction method combining convolution and Transformer in the experimental design of this article, and provides reference for feature extraction methods for tasks such as multimodal medical image segmentation or classification.