Fusion Task Research Articles

Fusing Satellite Precipitation Products Based on Top–Down and Bottom–Up Approaches and an Improved Double Instrumental Variable Method for the Chuanyu Region, China, from 2007 to 2019

Precipitation is one of the crucial variables in the hydrological and ecological cycles. High-quality precipitation data are of great importance for climate change, water resource management, and agricultural research over complex terrains. Recently, satellite precipitation products have been produced based on different retrieval algorithms, mainly the top–down and bottom–up approaches. Fusing satellite precipitation products based on these two different approaches may combine their advantages and achieve a better data quality for describing precipitation. In this paper, an improved double instrumental variable (IMDIV) method is proposed for data quality enhancement by merging IMERG (integrated multi-satellite retrievals for global precipitation measurement), which is based on the top–down approach, and SM2RAIN (soil moisture to rain), which is based on the bottom–up approach. In detail, IMERG-Early (IMERG early run) and IMERG-Final (IMERG final run) are merging with SM2RAIN at a daily scale, respectively. Rain gauge station records from GHCNd (Global Historical Climatology Network Daily) are used to evaluate the original and fused precipitation products for the Chuanyu region, China, from 2007 to 2019. The results show that the proposed IMDIV method outperforms the original DIV method on precipitation fusion tasks. Moreover, the proposed IMDIV-EAS (fusing IMERG-Early and SM2RAIN) and IMDIV-FIS (fusing IMERG-Final and SM2RAIN) products both outperform the original precipitation products IMERG and SM2RAIN, with higher correlation coefficients (R) of 0.603 and 0.634; better RMSEs of 5.136 and 5.088 mm/day; and better biases of 0.514 and 0.509 mm/day. The results demonstrate the effectiveness of the proposed method and the high quality of the fused products, which could be useful for hydrology and climate studies.

FusionCPP: Cooperative fusion of infrared and visible light images based on PCNN and PID control systems

The imaging results of infrared and visible light sensors are highly complementary. Thus, combining the two images can compensate for the shortcomings of a single sensor. However, previous fusion methods have not fully utilized the collaborative guidance of infrared and visible images in the fusion process, leading to a poor correlation between extracted features and cannot guarantee feature fusion accuracy. In order to address these challenges, this paper proposes a collaborative fusion method for infrared and visible images based on pulse-coupled neural networks (PCNN) and proportional-integral-derivative (PID) control systems, called FusionCPP. First, a coupled neural network with a dual pulse structure is designed for the feature extraction section. Unlike traditional PCNN, this network includes two pulse generators, one of which is utilized to extract the salient features of the image, while the other is employed to extract the common pulse layer of the source image and multiply the pulse layer by the coupling iteration value to obtain the base layer. Second, a detail layer is obtained by subtracting the base layer from the source image. This method accurately extracts salient and detailed features based on the coupling between source images. Finally, a closed-loop PID control system is utilized as the fusion strategy in the image fusion task, which judges the difference between the fusion result and the source image in real-time through the feedback effect of the system. The controller adaptively adjusts the fusion weight based on the difference to fuse image features into the new image accurately. Experimental results indicate that the proposed FusionCPP can maintain significant contrast and rich textures. Besides, the proposed FusionCPP has extended to multi-focus image fusion and target detection tasks to verify its effectiveness.

MLFFusion: Multi-level feature fusion network with region illumination retention for infrared and visible image fusion

Infrared image and visible light image fusion is widely used in night vision, surveillance, military and other fields. The focus of the fusion task is to integrate complementary information in visible and infrared light images and eliminate redundant information. In addition, most of the fusion tasks are performed in the harsh environment of low light, and it is worth studying how to maintain the lighting information of the fusion results. In order to solve the problems existing, firstly, we designs a multi-level feature module to fusion multi-source information. Different from the parallel layer fusion strategy of the traditional network, we proposed a fusion strategy combine parallel layers and depth layers. Secondly, we add attention computing to the feature extraction network to improve the performance of the feature extraction network. Thirdly, in order to make the fusion image have good illumination information, we design the area illumination retention module, improved the performance of the fusion algorithm in low-light environments. A large number of experiments show that the proposed method has excellent performance and will perform better in low light environment. In addition, the proposed algorithm also shows great potential in multi modal object detection.

FusionDiff: Multi-focus image fusion using denoising diffusion probabilistic models

Multi-focus image fusion (MFIF) is an image enhancement technology with broad application prospects that can effectively extend the depth-of-field of optical lenses. This paper presents a novel MFIF method, named FusionDiff, based on denoising diffusion probabilistic models (DDPM). FusionDiff uses DDPM to fuse two source images by iteratively performing multiple denoising operations. To predict noise accurately and train the model efficiently, a lightweight U-Net architecture is designed as the conditional noise predictor. FusionDiff does not depend on any specific activity level measurement method, fusion rule or complex feature extraction network. It overcomes many algorithm design and training difficulties in existing image fusion methods. FusionDiff is noise-resistant and can still produce outstanding fusion results from source images with noise interference. In addition, FusionDiff is a few-shot learning method, which makes it suitable for image fusion tasks where training samples are relatively scarce. Experiments show that FusionDiff outperforms representative state-of-the-art methods in both visual perception and quantitative metrics. The code is available at https://github.com/lmn-ning/ImageFusionhttps://github.com/lmn-ning/ImageFusion

SFPFusion: An Improved Vision Transformer Combining Super Feature Attention and Wavelet-Guided Pooling for Infrared and Visible Images Fusion.

The infrared and visible image fusion task aims to generate a single image that preserves complementary features and reduces redundant information from different modalities. Although convolutional neural networks (CNNs) can effectively extract local features and obtain better fusion performance, the size of the receptive field limits its feature extraction ability. Thus, the Transformer architecture has gradually become mainstream to extract global features. However, current Transformer-based fusion methods ignore the enhancement of details, which is important to image fusion tasks and other downstream vision tasks. To this end, a new super feature attention mechanism and the wavelet-guided pooling operation are applied to the fusion network to form a novel fusion network, termed SFPFusion. Specifically, super feature attention is able to establish long-range dependencies of images and to fully extract global features. The extracted global features are processed by wavelet-guided pooling to fully extract multi-scale base information and to enhance the detail features. With the powerful representation ability, only simple fusion strategies are utilized to achieve better fusion performance. The superiority of our method compared with other state-of-the-art methods is demonstrated in qualitative and quantitative experiments on multiple image fusion benchmarks.

Multi-focus image fusion using structure-guided flow

Existing deep learning based methods have shown their advantages in multi-focus image fusion task. However, most methods still suffer from inaccurate focus region detection. In this paper, we employ the property of part-whole relationships embedded by the Capsule Network (CapsNet) to solve the problem. Specifically, we introduce CapsNet in multi-focus image fusion task, and design a structure-guided flow module, which fully utilizes structure information to help locate focus regions. CapsNet is introduced to extract structure features by supervising gradient information of the image. Compared with traditional convolutional neural networks (CNNs), CapsNet takes into account the correlation of features from different positions, such that it encodes more compact features. Once structure features are obtained, a flow alignment module is introduced to learn flow field between structure and image features, and propagate effectively structure features to image features to make confident focus region detection. Experimental results show the proposed method achieves robust fusion performance on three publicly available multi-focus datasets, and outperforms or is comparable to the state-of-the-art methods.

LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images.

Deep learning based fusion methods have been achieving promising performance in image fusion tasks. This is attributed to the network architecture that plays a very important role in the fusion process. However, in general, it is hard to specify a good fusion architecture, and consequently, the design of fusion networks is still a black art, rather than science. To address this problem, we formulate the fusion task mathematically, and establish a connection between its optimal solution and the network architecture that can implement it. This approach leads to a novel method proposed in the paper of constructing a lightweight fusion network. It avoids the time-consuming empirical network design by a trial-and-test strategy. In particular we adopt a learnable representation approach to the fusion task, in which the construction of the fusion network architecture is guided by the optimisation algorithm producing the learnable model. The low-rank representation (LRR) objective is the foundation of our learnable model. The matrix multiplications, which are at the heart of the solution are transformed into convolutional operations, and the iterative process of optimisation is replaced by a special feed-forward network. Based on this novel network architecture, an end-to-end lightweight fusion network is constructed to fuse infrared and visible light images. Its successful training is facilitated by a detail-to-semantic information loss function proposed to preserve the image details and to enhance the salient features of the source images. Our experiments show that the proposed fusion network exhibits better fusion performance than the state-of-the-art fusion methods on public datasets. Interestingly, our network requires a fewer training parameters than other existing methods.

Trans2Fuse: Empowering image fusion through self-supervised learning and multi-modal transformations via transformer networks

Image fusion enhances a single image by integrating information from multiple sources with complementary data. Present end-to-end fusion methods often face overfitting or intricate parameter tuning due to inadequate task-specific training data. To address this, two-stage approaches utilize encoder–decoder networks trained on extensive natural image datasets, yet suffer from limited performance due to domain disparities. In this work, we devise a novel encoder–decoder fusion framework and introduce a self-supervised scheme based on destruction–reconstruction. This approach facilitates task-specific feature learning by proposing three auxiliary tasks: pixel intensity non-linear transformation for multi-modal fusion, brightness transformation for multi-exposure fusion, and noise transformation for multi-focus fusion. By randomly selecting one task during model training, we mutually reinforce different fusion tasks, enhancing network generalizability. We innovate an encoder combining Convolutional Neural Network (CNN) and Transformer to extract both local and global features. Rigorous evaluations against 11 traditional and deep learning-based methods span four benchmark datasets: infrared-visible fusion, medical fusion, multi-exposure fusion, and multi-focus fusion. Comprehensive assessments, encompassing nine metrics from diverse viewpoints, consistently demonstrate the superior performance of our approach in all scenarios. We will make our code, datasets, and fused images publicly available.

Graph neural network based unsupervised influential sample selection for brain multigraph population fusion.

Graph neural networks (GNNs) have witnessed remarkable proliferation due to the increasing number of applications where data is represented as graphs. GNN-based multigraph population fusion methods for estimating population representative connectional brain templates (CBT) have recently led to improvements, especially in network neuroscience. However, prior studies do not consider how an individual training brain multigraph influences the quality of GNN training for brain multigraph population fusion. To address this issue, we propose two major sample selection methods to quantify the influence of a training brain multigraph on the brain multigraph population fusion task using GNNs, in a fully unsupervised manner: (1) GraphGradIn, in which we use gradients w.r.t GNN weights to trace changes in the centeredness loss of connectional brain template during the training phase; (2) GraphTestIn, in which we exclude a training brain multigraph of interest during the refinement process in the test phase to infer its influence on the CBT centeredness loss. Next, we select the most influential multigraphs to build the training set for brain multigraph population fusion into a CBT. We conducted extensive experiments on brain multigraph datasets to show that using a dataset of influential training samples improves the learned connectional brain template in terms of centeredness, discriminativeness, and topological soundness. Finally, we demonstrate the use of our methods by discovering the connectional fingerprints of healthy and neurologically disordered brain multigraph populations including Alzheimer's disease and Autism spectrum disorder patients. Our source code is available at https://github.com/basiralab/GraphGradIn.

Adversarial attacks on multi-focus image fusion models

Multi-focus image fusion aims to create an all-in-focus clear image by fusing a set of partially focused images. In recent years, various multi-focus image fusion methods based on deep learning have been proposed, but there is no thorough study evaluating their robustness for adversarial attacks. In this paper, we investigate the robustness of deep learning based multi-focus image fusion models to adversarial attacks. First, we generated adversarial examples which can significantly reduce the fusion quality of image fusion models. Then, a metric defocus attack intensity (DAI) was proposed to quantitatively evaluate the robustness of different models for adversarial attacks. At last, we analyzed the factors affecting model robustness, including model size and post-processing steps. Besides, we successfully attacked recent image fusion models in the black-box scene by utilizing the transferability of adversarial examples. Experimental results show that state-of-the-art image fusion models are also vulnerable to adversarial attacks, and some observations in image classifier robustness studies are not transferable to image fusion task.

MBRARN: multibranch residual attention reconstruction network for medical image fusion.

Medical image fusion aims to integrate complementary information from multimodal medical images and has been widely applied in the field of medicine, such as clinical diagnosis, pathology analysis, and healing examinations. For the fusion task, feature extraction is a crucial step. To obtain significant information embedded in medical images, many deep learning-based algorithms have been proposed recently and achieved good fusion results. However, most of them can hardly capture the independent and underlying features, which leads to unsatisfactory fusion results. To address these issues, a multibranch residual attention reconstruction network (MBRARN) is proposed for the medical image fusion task. The proposed network mainly consists of three parts: feature extraction, feature fusion, and feature reconstruction. Firstly, the input medical images are converted into three scales by image pyramid operation and then are input into three branches of the proposed network respectively. The purpose of this procedure is to capture the local detailed information and the global structural information. Then, convolutions with residual attention modules are designed, which can not only enhance the captured outstanding features, but also make the network converge fast and stably. Finally, feature fusion is performed with the designed fusion strategy. In this step, a new more effective fusion strategy is correspondently designed for MRI-SPECT based on the Euclidean norm, called feature distance ratio (FDR). The experimental results conducted on Harvard whole brain atlas dataset demonstrate that the proposed network can achieve better results in terms of both subjective and objective evaluation, compared with some state-of-the-art medical image fusion algorithms.

Combining transformers with CNN for multi-focus image fusion

Recently, deep convolutional neural network (CNN) based methods for multi-focus image fusion have achieved adequate performance. However, most of them cannot obtain spatially continuous results, especially in smooth regions and edges between focused and defocused regions. In this paper, we propose a novel end-to-end method, which merits both Transformers and CNNs, as a strong alternative for multi-focus image fusion task. Transformer has advantages over a CNN in that it can extract global features. It is able to make the fusion results to be spatially consistent. The proposed architecture consists of CNN and transformer branches, where transformer branches take feature map patches as inputs and leverages the transformer to propagate global contexts among patches. Moreover, in order to improve feature representation, we introduce online knowledge distillation learning strategy (KDL). The strategy achieves better interactions between global features and local features. Specifically, we design hard target and soft target by simply yet effectively ensembling outputs of two branches, which are used to supervise CNN and transformer branches. The experiments demonstrate the superiority of our proposed architecture and achieve competitive results with state-of-the-art methods.

HiPrompt: Few-Shot Biomedical Knowledge Fusion via Hierarchy-Oriented Prompting.

Medical decision-making processes can be enhanced by comprehensive biomedical knowledge bases, which require fusing knowledge graphs constructed from different sources via a uniform index system. The index system often organizes biomedical terms in a hierarchy to provide the aligned entities with fine-grained granularity. To address the challenge of scarce supervision in the biomedical knowledge fusion (BKF) task, researchers have proposed various unsupervised methods. However, these methods heavily rely on ad-hoc lexical and structural matching algorithms, which fail to capture the rich semantics conveyed by biomedical entities and terms. Recently, neural embedding models have proved effective in semantic-rich tasks, but they rely on sufficient labeled data to be adequately trained. To bridge the gap between the scarce-labeled BKF and neural embedding models, we propose HiPrompt, a supervision-efficient knowledge fusion framework that elicits the few-shot reasoning ability of large language models through hierarchy-oriented prompts. Empirical results on the collected KG-Hi-BKF benchmark datasets demonstrate the effectiveness of HiPrompt.

Joint coupled dictionaries-based visible-infrared image fusion method via texture preservation structure in sparse domain

Joint sparse coding (JSC) and coupled dictionary learning (CDL) have been successful in visible and infrared image fusion tasks. However, JSC fusion methods using a single dictionary cannot perfectly characterize different modal signals. The CDL-based fusion methods ignore the role of high-frequency components in texture preservation and sharpness. In this study, we design fusion rules while considering the uniqueness and connection of different feature spaces to alleviate this issue. In the proposed fusion method, coupled dictionaries are first used to establish the relationship between infrared and visible images. Then, we use joint sparse coding and coupled dictionaries to calculate sparse coefficients via the joint sparse model. Finally, the improved coefficient fusion strategy realizes the conversion and synthesis between different spatial images. Experimental results on the TNO and RoadScene datasets show that the proposed method achieves promising results in terms of objective and subjective performance compared with state-of-the-art algorithms.

Performance in a Simulated Virtual Reality Anterior Cervical Discectomy and Fusion Task: Disc Residual, Rate of Removal, and Efficiency Analyses.

Anterior cervical discectomy and fusion (ACDF) is among the most common spine procedures. The Sim-Ortho virtual reality simulator platform contains a validated ACDF simulated task for performance assessment. This study aims to develop a methodology to extract three-dimensional data and reconstruct and quantitate specific simulated disc tissues to generate novel metrics to analyze performance metrics of skilled and less skilled participants. We used open-source platforms to develop a methodology to extract three-dimensional information from ACDF simulation data. Metrics generated included, efficiency index, disc volumes removed from defined regions, and rate of tissue removal from superficial, central, and deep disc regions. A pilot study was performed to assess the utility of this methodology to assess expertise during the ACDF simulated procedure. The system outlined, extracts data allowing the development of a methodology which accurately reconstructs and quantitates 3-dimensional disc volumes. In the pilot study, data sets from 27 participants, divided into postresident, resident, and medical student groups, allowed assessment of multiple novel metrics, including efficiency index (surgical time spent in actively removing disc), where the postresident group spent 61.8% of their time compared with 53% and 30.2% for the resident and medical student groups, respectively ( P = .01). During the annulotomy component, the postresident group removed 47.4% more disc than the resident groups and 102% more than the medical student groups ( P = .03). The methodology developed in this study generates novel surgical procedural metrics from 3-dimensional data generated by virtual reality simulators and can be used to assess surgical performance.

DATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer

The fusion of infrared and visible images aims to generate a composite image that can simultaneously contain the thermal radiation information of an infrared image and the plentiful texture details of a visible image to detect targets under various weather conditions with a high spatial resolution of scenes. Previous deep fusion models were generally based on convolutional operations, resulting in a limited ability to represent long-range context information. In this paper, we propose a novel end-to-end model for infrared and visible image fusion via a dual attention Transformer termed DATFuse. To accurately examine the significant areas of the source images, a dual attention residual module (DARM) is designed for important feature extraction. To further model long-range dependencies, a Transformer module (TRM) is devised for global complementary information preservation. Moreover, a loss function that consists of three terms, namely, pixel loss, gradient loss, and structural loss, is designed to train the proposed model in an unsupervised manner. This can avoid manually designing complicated activity-level measurement and fusion strategies in traditional image fusion methods. Extensive experiments on public datasets reveal that our DATFuse outperforms other representative state-of-the-art approaches in both qualitative and quantitative assessments. The proposed model is also extended to address other infrared and visible image fusion tasks without fine-tuning, and the promising results demonstrate that it has good generalization ability. The source code is available at https://github.com/tthinking/DATFuse.

GIPC-GAN: an end-to-end gradient and intensity joint proportional constraint generative adversarial network for multi-focus image fusion

As for the problems of boundary blurring and information loss in the multi-focus image fusion method based on the generative decision maps, this paper proposes a new gradient-intensity joint proportional constraint generative adversarial network for multi-focus image fusion, with the name of GIPC-GAN. First, a set of labeled multi-focus image datasets using the deep region competition algorithm on a public dataset is constructed. It can train the network and generate fused images in an end-to-end manner, while avoiding boundary errors caused by artificially constructed decision maps. Second, the most meaningful information in the multi-focus image fusion task is defined as the target intensity and detail gradient, and a jointly constrained loss function based on intensity and gradient proportional maintenance is proposed. Constrained by a specific loss function to force the generated image to retain the information of target intensity, global texture and local texture of the source image as much as possible and maintain the structural consistency between the fused image and the source image. Third, we introduce GAN into the network, and establish an adversarial game between the generator and the discriminator, so that the intensity structure and texture gradient retained by the fused image are kept in a balance, and the detailed information of the fused image is further enhanced. Last but not least, experiments are conducted on two multi-focus public datasets and a multi-source multi-focus image sequence dataset and compared with other 7 state-of-the-art algorithms. The experimental results show that the images fused by the GIPC-GAN model are superior to other comparison algorithms in both subjective performance and objective measurement, and basically meet the requirements of real-time image fusion in terms of running efficiency and mode parameters quantity.

Combining Slow and Fast: Complementary Filtering for Dynamics Learning

Modeling an unknown dynamical system is crucial in order to predict the future behavior of the system. A standard approach is training recurrent models on measurement data. While these models typically provide exact short-term predictions, accumulating errors yield deteriorated long-term behavior. In contrast, models with reliable long-term predictions can often be obtained, either by training a robust but less detailed model, or by leveraging physics-based simulations. In both cases, inaccuracies in the models yield a lack of short-time details. Thus, different models with contrastive properties on different time horizons are available. This observation immediately raises the question: Can we obtain predictions that combine the best of both worlds? Inspired by sensor fusion tasks, we interpret the problem in the frequency domain and leverage classical methods from signal processing, in particular complementary filters. This filtering technique combines two signals by applying a high-pass filter to one signal, and low-pass filtering the other. Essentially, the high-pass filter extracts high-frequencies, whereas the low-pass filter extracts low frequencies. Applying this concept to dynamics model learning enables the construction of models that yield accurate long- and short-term predictions. Here, we propose two methods, one being purely learning-based and the other one being a hybrid model that requires an additional physics-based simulator.

Cross-Modal Sentiment Analysis of Text and Video Based on Bi-GRU Cyclic Network and Correlation Enhancement

Cross-modal sentiment analysis is an emerging research area in natural language processing. The core task of cross-modal fusion lies in cross-modal relationship extraction and joint feature learning. The existing research methods of cross-modal sentiment analysis focus on static text, video, audio, and other modality data but ignore the fact that different modality data are often unaligned in practical applications. There is a long-term time dependence among unaligned data sequences, and it is difficult to explore the interaction between different modalities. The paper proposes a sentiment analysis model (UA-BFET) based on feature enhancement technology in unaligned data scenarios, which can perform sentiment analysis on unaligned text and video modality data in social media. Firstly, the model adds a cyclic memory enhancement network across time steps. Then, the obtained cross-modal fusion features with interaction are applied to the unimodal feature extraction process of the next time step in the Bi-directional Gated Recurrent Unit (Bi-GRU) so that the progressively enhanced unimodal features and cross-modal fusion features continuously complement each other. Secondly, the extracted unimodal text and video features taken jointly from the enhanced cross-modal fusion features are subjected to canonical correlation analysis (CCA) and input into the fully connected layer and Softmax function for sentiment analysis. Through experiments executed on unaligned public datasets MOSI and MOSEI, the UA-BFET model has achieved or even exceeded the sentiment analysis effect of text, video, and audio modality fusion and has outstanding advantages in solving cross-modal sentiment analysis in unaligned data scenarios.

TPP: Deep learning based threshold post-processing multi-focus image fusion method

How to accurately determine focused and defocused domains is a difficult problem in the field of multi-focused image fusion. To solve this problem, we propose a new deep learning method called threshold post processing(TPP). The TPP network model contains three architecturally different channels, and three different weighted maps can be obtained for the same input. The output of the model is fused and thresholded to obtain a final decision map, which is then used to complete the image fusion task. The decision map represents the focus condition of the source image at a pixel level. This allows TPP to retain focused regions of the source image more accurately than other existing deep learning methods. The multi-channel network model combined with the thresholding process optimizes the final decision map. It avoids the drawback of other methods that cause inaccurate discrimination of the focused area due to a single network channel and therefore cannot be corrected. Subsequent experimental results show that the images generated by TPP maintain a high level under the various image quality assessment metrics measures.