Fusion Task Research Articles

MD³Net: Integrating Model-Driven and Data-Driven Approaches for Pansharpening

Pansharpening is a special image fusion task of reconstructing a high-resolution multispectral (HRMS) image by integrating a panchromatic (PAN) image of high spatial resolution and a low-resolution multispectral (LRMS) image. To handle such an ill-posed multi-modal fusion task, in this paper, we propose a novel pansharpening method, referred to as model-driven and data-driven network (MD³Net), which combines model-driven and data-driven approaches. The architecture design of MD³Net is inspired from the traditional model constructed based on domain knowledge and thus making its network topology explainable and its input/output predictable. In order to further explore the powerful learning ability of deep learning based approaches, we introduce the deep prior into the MD³Net as its implicit regularization, thus improving its data adaptability and representation capability. Comprehensive experiments conducted on both reduced and full resolution of several acknowledged datasets have qualitatively and quantitatively verified the superiority of our network compared to a benchmark consisting of several state-of-the-art approaches. The code can be downloaded from https://github.com/YinsongYan/M3DNet..

A Two-Level Dynamic Adaptive Network for Medical Image Fusion

The research of deep learning-based methods for image fusion has become a current hotspot. Medical image fusion with the problem of few samples also lacks a unified end-to-end model for the input of different modal pairs. In this article, we propose a two-level dynamic adaptive network for medical image fusion, which addresses the above two problems and provides a unified fusion framework to take the advantage of different modal pairs. Specifically, we develop a dynamic meta-learning method on task level, which achieves a dynamical meta-knowledge transfer from the heterogeneous task of multifocus image fusion to medical image fusion by dynamic convolution decomposition (DCD). Then, we provide an efficient adaptive fusion method on multimodal feature level, which uses dynamic attention mechanism and dynamic channel fusion mechanism to fuse features of different aspects. For model evaluation, we have done the qualitative and quantitative tests on the transferred multifocus deep network and verified its superior fusion performance. On this basis, the experiments are carried out on the public datasets of the two most commonly used modal pairs (computerized tomography (CT)-magnetic resonance imaging (MRI) and positron emission tomography (PET)-MRI) and show that our hierarchical model is superior to the state-of-the-art methods in terms of visual effects and quantitative measurement. Our code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zhanglabNKU/TDAN</uri> .

A new method of multi-robot cooperative exploration

This paper aims at the task of the map fusion and environmental exploration method. The multi-robot collaborative map creation is based on the local map by each individual robot, and the nodes in the local map are matched to obtain the association between multiple maps. Multi-robot task allocation is also a process of maximising the benefits of resource utilisation. How to allocate resources reasonably so that each robot can quickly complete the corresponding tasks in coordination, and how to solve the problem of the heavy tasks for individual robots or few tasks for individual robots, multitask allocation based on market law can be a solution. The new topological scene map in this paper is used to represent the environment and carry out robot exploration. It can be seen that multi-robot cooperative exploration is more efficient efficiency.

MMI-Fuse: Multimodal Brain Image Fusion With Multiattention Module

Medical imaging plays a pivotal role in the clinical diagnosis of brain disease. There are many imaging methods to detect the state of tissues in the brain. While these imaging methods have advantages, they also have shortcomings. For example, magnetic resonance imaging (MRI) contains structural information but no functional characteristics of tissue, while positron emission tomography (PET) possesses functional characteristics but no structural information. The attention mechanism has been widely used in image fusion tasks, such as fusion of infrared and visible images and medical images. However, those attention models lack a balance mechanism for multimodal image features, affecting the final fusion performance. This paper proposes an end-to-end multimodal brain image fusion framework, MMI-fuse. Specifically, we first apply an autoencoder to extract the features of source images. Then, an information preservation weighted channel spatial attention model (ICS) is proposed to fuse the image features. We set an adaptive weight according to the information preservation degree of features. Finally, we use a decoder model to restructure the fused medical image. The proposed method increased the quality of fused images and decreased the fusion time effectively by the help of the improved attention model and encoder-decoder structure. To validate the performance of the proposed method, we collected 1590 pairs of multimodal brain images from the Harvard dataset and performed extensive experiments. Seven methods and five metrics were selected for the comparison experiments. The results demonstrate that the proposed method achieved notable performance on both the visual quality and objective metric score among these seven approaches. Moreover, the proposed method takes the least time among all compared methods.

Energy-Efficiency Computation Offloading Strategy in UAV Aided V2X Network With Integrated Sensing and Communication

Integrated Sensing and Communications (ISAC) technology can jointly design radio sensing and communication functionalities, which enable 6G Ere to have the ability to &#x201C;see&#x201D; the physical world rather than communication-only. Benefitting from ISAC, vehicular-to-everything (V2X) networks may efficiently complete high-precision traffic environment perception. Furthermore, with the assistance of flexibly deployed Unmanned Aerial Vehicles (UAVs), the V2X networks overcome the limited sensing range of sensors equipped on a vehicle and guarantee safe driving. This paper proposes an energy-efficient computation offloading strategy for multiple sensor data fusion in UAV Aided V2X Network supported by Integrated Sensing and Communication. Firstly, a vehicle-UAV cooperative perception architecture is proposed to perceive a wide range of traffic environments. Secondly, we introduce a computation offloading strategy jointly considering offloading decisions and dynamic computing resource allocation. Finally, a successive convex approximation (SCA) algorithm transforms a non-convex formulation problem into a tractable convex approximation problem. The simulation results show that the strategy proposed in this paper reduces the UAV energy consumption and data fusion task processing delay.

Res2Fusion: Infrared and Visible Image Fusion Based on Dense Res2net and Double Nonlocal Attention Models

Infrared and visible image fusion intends to generate a synthetic image with superior scene representation and better visual perception. The existing deep learning-based fusion methods merely make use of the convolution operation to extract features with a local receptive field, without fully considering their multiscale and long-range dependency characteristics, which may fail to preserve some essential global context from source images. To this end, we develop a novel and efficient fusion network based on dense Res2net and double nonlocal attention models, termed Res2Fusion. We introduce Res2net and dense connections into the encoder network with multiple available receptive fields, which is used to extract the multiscale features, and can retain as much meaningful information as possible for fusion tasks. In addition, we develop double nonlocal attention models as a fusion layer to model long-range dependencies on the local features. Specifically, these attention models can refine feature maps obtained by the encoder network to more focus on prominent infrared targets and distinct visible details. Finally, the comprehensive attention maps are used to generate a fused result through the simple decoder network. Extensive experiments demonstrate that the proposed method can simultaneously retain highlighted infrared targets and rich visible details and transcends other state-of-the-art fusion methods in terms of subjective and objective evaluation. The corresponding code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Zhishe-Wang/Res2Fusion</uri> .

Creating Positive, Negative, and Neutral God Concept Primes and Testing Their Impact on Scrupulosity Relevant Tasks and Symptoms

ABSTRACT Three studies were conducted to develop primes that activate differing beliefs about God and to determine if these activated beliefs impact scrupulosity symptoms. In Study 1, potential positive, negative, and neutral primes were created from Bible verses and rated on arousal, valence, familiarity, and complexity. In Study 2, the selected primes were tested to determine if they differentially activated positive and negative beliefs about God. In Study 3, participants were randomly assigned to read one of three primes (positive, negative, or neutral). Participants then completed a thought-induction task and measures of scrupulosity and thought-action fusion. There were no significant differences in reactions to the thought-induction task or self-report measures of scrupulosity or thought-action fusion between conditions. An exploratory analysis revealed that participants in the negative condition had higher state anxiety following the prime compared to participants in the positive condition. Results demonstrate the potential of the primes to activate negative beliefs about God and highlight the possible detrimental impact negative beliefs about God may have on state anxiety. However, they do not suggest that the primes impact scrupulosity and related constructs.

Unified framework based on multiscale transform and feature learning for infrared and visible image fusion

Infrared image and visible image can provide the thermal radiation and optical information of objects respectively. Hence, fusing their complementary information for a scene is essential for both human vision and machine perception. The adaptability of image fusion by multiscale transform (MST) is weak because of the fixed fusion rule. Meanwhile, the image fusion algorithm by supervised learning is hard to realize for lack of large labeled data. Therefore, we propose a unified framework for infrared and visible-image fusion via MST and feature learning. The reasonable fusion rule is designed by learning the attention response via training a novel attention architecture built-in the attention-guided generative adversarial network, which does not need the fused result as the ground truth. The experiment results demonstrate the validity of feature learning and the superiority of the proposed algorithm against other state-of-the-art methods in image fusion task. And the proposed method is one of the effective ways to promote the development of deep learning in the field of insufficient labeled data.

Two‐stage progressive residual learning network for multi‐focus image fusion

Boundary artifacts and color detail distortion are easily caused by the common multi-focus image fusion methods. In order to solve this problem, we propose a two-stage progressive residual learning network for multi-focus image fusion. The proposed network can progressively learn color information and detail features through end-to-end mapping. The whole network is composed of two sub-networks: the initial fusion block network and the enhanced fusion block network. First, the color information in the source image is fused by the initial fusion block network to generate the initial fusion image. Then on the basis of the initial fusion image, the detailed features of the source image are further fused by enhanced the fusion network to form the final fusion image. In order to solve the problem of lack of groundtruth when multi-focus image fusion is carried out with supervised method, the multi-focus image fusion problem is compared to the easy-to-solve image restoration problem. A synthetic dataset for network training is generated by ”degenerating” the VOC2012 dataset according to the set rules. After training, the method works well for fusion tasks without further processing. Experimental results show that the proposed method is superior to the existing methods in subjective visual perception and objective quantitative evaluation.

Improvements in Multi-Document Abstractive Summarization using Multi Sentence Compression with Word Graph and Node Alignment

The present work proposes a scheme for multi-document abstractive text summarization using node-aligned Word Graph based representation of clustered sentences. In the first step, the proposed scheme uses SBERT embedding for representing the sentences as fixed-size vectors. The sentences belonging to the same cluster are then represented using Word Graph, in which words of different sentences are aligned based on their semantic and syntactic similarities. The advantage of the above representation is that it utilizes alignment information of words between pairs of similar sentences to merge nodes in the Word Graph, and thereby facilitating the generation of sentences with multiple chunks of information. A sentence scoring function assisted by an intensification function is used to measure the grammaticality and informativeness of the generated sentences. Integer Linear Programming has been used to make the final selection of the scored sentences for generating the abstract. Experiments conducted for the task of sentence fusion and multi-document summarization demonstrate superior performance in comparison with the state-of-the-art techniques available in the literature.

FIRe-GAN: a novel deep learning-based infrared-visible fusion method for wildfire imagery

Wildfire detection is of paramount importance to avoid as much damage as possible to the environment, properties, and lives. In this regard, the fusion of thermal and visible information into a single image can potentially increase the robustness and accuracy of wildfire detection models. In the field of visible-infrared image fusion, there is a growing interest in Deep Learning (DL)-based image fusion techniques due to their reduced complexity; however, the most DL-based image fusion methods have not been evaluated in the domain of fire imagery. Additionally, to the best of our knowledge, no publicly available dataset contains visible-infrared fused fire images. In the present work, we select three state-of-the-art (SOTA) DL-based image fusion techniques and evaluate them for the specific task of fire image fusion, and compare the performance of these methods on selected metrics. Finally, we also present an extension to one of the said methods, that we called FIRe-GAN, that improves the generation of artificial infrared and fused images.

Employing Multimodal Machine Learning for Stress Detection.

In the current information age, the human lifestyle has become more knowledge-oriented, leading to sedentary employment. This has given rise to a number of health and mental disorders. Mental wellness is one of the most neglected, however crucial, aspects of today's fast-paced world. Mental health issues can, both directly and indirectly, affect other sections of human physiology and impede an individual's day-to-day activities and performance. However, identifying the stress and finding the stress trend for an individual that may lead to serious mental ailments is challenging and involves multiple factors. Such identification can be achieved accurately by fusing these multiple modalities (due to various factors) arising from a person's behavioral patterns. Specific techniques are identified in the literature for this purpose; however, very few machine learning-based methods are proposed for such multimodal fusion tasks. In this work, a multimodal AI-based framework is proposed to monitor a person's working behavior and stress levels. We propose a methodology for efficiently detecting stress due to workload by concatenating heterogeneous raw sensor data streams (e.g., face expressions, posture, heart rate, and computer interaction). This data can be securely stored and analyzed to understand and discover personalized unique behavioral patterns leading to mental strain and fatigue. The contribution of this work is twofold: firstly, proposing a multimodal AI-based strategy for fusion to detect stress and its level and, secondly, identifying a stress pattern over a period of time. We were able to achieve 96.09% accuracy on the test set in stress detection and classification. Further, we were able to reduce the stress scale prediction model loss to 0.036 using these modalities. This work can prove important for the community at large, specifically those working sedentary jobs, to monitor and identify stress levels, especially in current times of COVID-19.

A Deep and Supervised Atrous Convolutional Model for Multi-Focus Image Fusion

Due to the limitations of equipment and technology, few images captured by a single sensor can describe all the information in a scene. Multi-focus image fusion techniques provide an effective means of integrating images acquired from different sensors to produce a single fused image that is richer in scene information. To obtain better fusion results, we propose a novel deep learning structure for multi-focus image fusion to achieve end-to-end network structure without post-processing procedures. Specifically, an innovative Siamese multi-scale feature extraction module was introduced to extract multi-scale image features. Besides, aiming to enhance the quality of the fused image, we further designed a new adaptive fusion strategy by using the attention mechanism instead of following the previous manual design. Extensive experimental results confirm that the proposed algorithm is more effective, which surpasses the existing state-of-the-art multi-focus image fusion methods both quantitatively and qualitatively. Furthermore, additional expanded experiment conducted on the infrared and visible image dataset demonstrates that the proposed fusion framework is also capable of performing other fusion tasks with excellent fusion performance. The code of the proposed fusion method is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/govenda/</uri> MSPA-Fuse.

English

In order to achieve the multi-focus image fusion task, a sparse representation method based on quaternion for multi-focus image fusion is proposed in this paper. Firstly, the RGB color information of each pixel in the color image is represented by quaternion based on the relevant knowledge of computational mathematics, and the color image pixel is processed as a whole vector to maintain the relevant information between the three color channels. Secondly, the dictionary represented by quaternion and the sparse coefficient represented by quaternion are obtained by using the our proposed sparse representation model. Thirdly, the coefficient fusion is carried out by using the “max-L1” rule. Finally, the fused sparse coefficient and dictionary are used for image reconstruction to obtain the quaternion fused image, which is then converted into RGB color multi-focus fused image. Our method belongs to computational mathematics, and uses the relevant knowledge in the field of computational mathematics to help us carry out the experiment. The experimental results show that the method has achieved good results in visual quality and objective evaluation.

A light-weight, efficient, and general cross-modal image fusion network

Existing cross-modal image fusion methods pay limited research attention to image fusion efficiency and network architecture. However, the efficiency and accuracy of image fusion have an important impact on practical applications. To solve this problem, we propose a light-weight, efficient, and general cross-modal image fusion network, termed as AE-Netv2. Firstly, we analyze the influence of different network architectures (e.g., group convolution, depth-wise convolution, inceptionNet, squeezeNet, shuffleNet, and multi-scale module) on image fusion quality and efficiency, which provides a reference for the design of image fusion architecture. Secondly, we explore the commonness and characteristics of different image fusion tasks, which provides a research basis for further research on the continuous learning characteristics of the human brain. Finally, positive sample loss is added to the similarity loss to reduce the difference of data distribution of different cross-modal image fusion tasks. Comprehensive experiments demonstrate the superiority of our method compared to state-of-the-art methods in different fusion tasks at a real-time speed of 100+ FPS on GTX 2070. Compared with the fastest image fusion method based on deep learning, the efficiency of AE-Netv2 is improved by 2.14 times. Compared with the image fusion model with the smallest model size, the size of our model is reduced by 11.59 times.

Hyperspectral and Multispectral Image Fusion by Deep Neural Network in a Self-Supervised Manner

Compared with multispectral sensors, hyperspectral sensors obtain images with high- spectral resolution at the cost of spatial resolution, which constrains the further and precise application of hyperspectral images. An intelligent idea to obtain high-resolution hyperspectral images is hyperspectral and multispectral image fusion. In recent years, many studies have found that deep learning-based fusion methods outperform the traditional fusion methods due to the strong non-linear fitting ability of convolution neural network. However, the function of deep learning-based methods heavily depends on the size and quality of training dataset, constraining the application of deep learning under the situation where training dataset is not available or of low quality. In this paper, we introduce a novel fusion method, which operates in a self-supervised manner, to the task of hyperspectral and multispectral image fusion without training datasets. Our method proposes two constraints constructed by low-resolution hyperspectral images and fake high-resolution hyperspectral images obtained from a simple diffusion method. Several simulation and real-data experiments are conducted with several popular remote sensing hyperspectral data under the condition where training datasets are unavailable. Quantitative and qualitative results indicate that the proposed method outperforms those traditional methods by a large extent.

Hyperspectral image fusion based on L-NMF

The fusion of a hyperspectral image (HSI) and a panchromatic image (PI) creates data that is beneficial to subsequent processing. Based on the traditional nonnegative matrix factorization (NMF)-based fusion method, the logarithmic hyperbolic cosine function is introduced as a spectral constraint. Under the constraint, the original HSI is decomposed into endmember and abundance matrices. The abundance matrix is then enhanced using PI, and the fused HSI is finally obtained with NMF reconstruction. In this way, the fused data have comparable spatial detail and spectral information. According to the experiments on the simulated and real data, the proposed fusion method obtains balanced fused results, and it is more suitable for the HSI fusion task.

Image fusion meets deep learning: A survey and perspective

Image fusion, which refers to extracting and then combining the most meaningful information from different source images, aims to generate a single image that is more informative and beneficial for subsequent applications. The development of deep learning has promoted tremendous progress in image fusion, and the powerful feature extraction and reconstruction capabilities of neural networks make the fused results promising. Recently, several latest deep learning technologies have made image fusion explode, e.g., generative adversarial networks, autoencoder, etc. However, a comprehensive review and analysis of latest deep-learning methods in different fusion scenarios is lacking. To this end and in this survey, we first introduce the concept of image fusion, and classify the methods from the perspectives of the deep architectures adopted and fusion scenarios. Then, we review the state-of-the-art on the use of deep learning in various types of image fusion scenarios, including the digital photography image fusion, the multi-modal image fusion and the sharpening fusion. Subsequently, the evaluation for some representative methods in specific fusion tasks are performed qualitatively and quantitatively. Moreover, we briefly introduce several typical applications of image fusion, including photography visualization, RGBT object tracking, medical diagnosis, and remote sensing monitoring. Finally, we provide the conclusion, highlight the challenges in image fusion, and look forward to potential future research directions.

An ensemble deep learning method as data fusion system for remote sensing multisensor classification

Because of the great achievements in designing remote sensing sensors, the extraction of useful information from multisource remote sensing data remains a challenging problem. Most of the recent research projects have applied single deep learning systems for data fusion and classification. The idea of using ensemble deep learning algorithms through a multisensor fusion system can improve the performance of data fusion tasks. In this research, however, a multi-sensor classification strategy, which is based on deep learning ensemble procedure and decision fusion framework, is investigated for the fusion of Light Detection and Ranging (LiDAR), Hyperspectral Images (HS), and very high-resolution Visible (Vis) images. This research proposes a basic classifier based on deep Convolutional Neural Network (CNN) in which the softmax layer is replaced by a Support Vector Machine (CNN-SVM). Then, a random feature selection is applied to generate two separate CNN-SVM ensemble systems, one for LiDAR and Vis and the other one for HS data. To overcome the similarity and overfitting between the deep features and the classifiers provided by two ensemble systems and to select the best subsets of the classifiers, two diversity measures select the most diverse combinations of the classifiers. Finally, a decision fusion method combines the obtained diverse classifiers from CNN ensembles. Results demonstrate that the proposed method achieves higher accuracy, and its performance outperforms some of the existing methods. The proposed ensemble CNN method improved single deep CNN, random forest, and Adaboost between 2% to 10% in terms of classification accuracy.

Two Stages Pan-Sharpening Details Injection Approach Based on Very Deep Residual Networks

Pan-sharpening is a fusion task, which aims to combine a low spatial resolution multispectral (MS) image with a high spatial resolution single band panchromatic (PAN) image to produce a high spatial and spectral Pan-sharpened image. The success of a Pan-sharpening technique depends on its ability to boost the spatial quality of the MS image while preserving its spectral feature. To this end, we propose in this article a new two-stage detail injection approach allowing to reconstruct fine structures based on convolutional neural networks (CNNs). First, generalized Laplacian pyramid gain injections CNN is performed to estimate the optimal values of the injection gains for each MS band to inject spatial details extracted from the PAN image. Next, the result is enhanced by injecting the details missing using the power of deep residual learning. The quantitative and qualitative results on data sets from different satellites show that the proposed approach can achieve higher performances in both spatial and spectral qualities compared to the state of the art as well as the new CNN-based methods.