Existing Fusion Approaches Research Articles

CAF-ODNN: Complementary attention fusion with optimized deep neural network for multimodal fake news detection

Fake news is a real problem; unfortunately, it seems to worsen. Even though some false news detection methods have made significant progress, current multimodal approaches integrate cross-modal features directly without considering uncorrelated semantic representations may introduce noise into the multimodal features. This phenomenon reduces model accuracy by obscuring subtle differences between text and images crucial for identifying fake news. Uncorrelated semantics also reduce the detection accuracy since the identification often relies on these subtle differences. To address these challenges, we propose a unified Complementary Attention Fusion with an Optimized Deep Neural Network (CAF-ODNN) that captures subtle cross-modal relationships for multimodal fake news detection. CAF introduces image captioning to represent images semantically, allowing bidirectional complementary attention between modalities based on a scaled dot product to learn fine-grained correlations. A dedicated alignment and normalization component is incorporated to calibrate fused representations based on channel statistics, ensuring the semantics are preserved across modalities during the interaction, thus improving upon the simple concatenation used in existing fusion approaches. To improve feature extraction, an Optimized Deep Neural Network (ODNN) is implemented that exploits compositional learning. ODNN is designed with three fully connected layers to learn higher-level representations from CAF-fused features. Model parameters are then systematically tuned beyond standard random search techniques to identify configurations, maximizing feature quality and detection accuracy. Our proposed method outperforms comparable approaches on standard metrics on four real-world datasets, highlighting the importance of complementary attention fusion with optimization in identifying fake news.

Bayesian transfer filter for industrial processes with additional delayed slow-rate measurements

In industrial processes, fast-rate online measurements (FOMs) are commonly associated with delayed slow-rate measurements (DSMs), provided by off-line laboratory analysis. In this paper, we propose a Bayesian transfer filter to transfer the information/knowledge contained in the DSMs to the FOMs. Different from the existing fusion approaches that weight the FOMs and DSMs equally, the proposed method signs the FOMs as the target domain and the DSMs as the source domain. The predicted distribution of states in the target domain is modified by minimizing the Kullback-Leibler divergence by fixing the distribution of measurements. In this way, the resulting algorithm can make a good trade-off between the information transferred from the source domain and the information contained in the target domain. The efficiency of the proposed method is demonstrated by a two-dimensional moving target example and an industrial distillation column case. Furthermore, by comparing it to other existing methods, such as the Kalman Filter and modified delayed track-to-track fusion approach, this method makes a particular accuracy improvement in estimating states.

Joint low-rank and sparse decomposition for infrared and visible image sequence fusion

Image fusion for infrared and visible images is intended for integrating the daylight information and the hidden target information into a single synthetic image that is more suitable for high-level vision tasks, such as image segmentation, target recognition. Existing techniques that employ the same representation to extract the same characteristics of various source images may not be suitable for infrared and visible image sequence fusion due to ignoring the correlation information between the infrared and visible image sequences. In order to overcome this problem, this paper presents a new fusion method of infrared and visible image sequences via joint low-rank and sparse decomposition (JLRSD) model. First, infrared and visible image sequences are respectively decomposed via joint low-rank and sparse representation to obtain the common low-rank components, specific low-rank components and sparse low-rank components. Second, a novel weighted average technique with phase congruency and guided filtering is put forward to fuse the specific low-rank components. Then, the average fusion rule is adopted to fuse the sparse low-rank components, which aims to preserve more target information. Finally, the fusion results are obtained by summing the common low-rank component, the fused specific low-rank component, and the fused sparse low-rank component. As compared with the several existing fusion approaches, experimental results demonstrate that our method can provide superior achievement in terms of subjective evaluation and objective assessment.

Deep remote fusion: development of improved deep CNN with atrous convolution-based remote sensing image fusion

ABSTRACT Various remote sensing images seek more attention with their high temporal and spatial resolution in the applications of earth science. Conversely, it is a difficult task for a single satellite to enhance high-quality images owing to its cost and technical constraints. It aims to suggest a novel remote sensing image fusion model for overcoming the limitations of the existing fusion approaches, where effective fusion rules were generated. Initially, the low frequency of MS image is used to enhance the PAN image using the optimized wavelet transform. Then, the enhanced PAN and MS images are used by the Improved Deep Convolutional Neural Network with Atrous Convolutions (IDCNN-AC) for getting the high-quality fused images. Further, the parameters in the IDCNN-AC are optimized using Hybrid Harris Hawks Dingo Optimization (HHHDO) for enhancing the fusion performance. Finally, the simulation outcome shows the efficient performance of the proposed image fusion model using different quantitative measures.

Geryon

Vision-based drone-view object detection suffers from severe performance degradation under adverse conditions (e.g., foggy weather, poor illumination). To remedy this, leveraging complementary mmWave radar has become a trend. However, existing fusion approaches seldom apply to drones due to i) the aggravated sparsity and noise of point clouds from low-cost commodity radars, and ii) explosive sensing data and intensive computations leading to high latency. To address these issues, we design Geryon, an edge assisted object detection system on drones, which utilizes a suit of approaches to fully exploit the complementary advantages of camera and mmWave radar on three levels: (i) a novel multi-frame compositing approach utilizes camera to assist radar to address the aggravated sparsity and noise of radar point clouds; (ii) a saliency area extraction and encoding approach utilizes radar to assist camera to reduce the bandwidth consumption and offloading latency; (iii) a parallel transmission and inference approach with a lightweight box enhancement scheme further reduces the offloading latency while ensuring the edge-side accuracy-latency trade-off by the parallelism and better camera-radar fusion. We implement and evaluate Geryon with four datasets we collect under foggy/rainy/snowy weather and poor illumination conditions, demonstrating its great advantages over other state-of-the-art approaches in terms of both accuracy and latency.

Reliable Scene Recognition Approach for Mobile Robots with Limited Resources Based on Deep Learning and Neuro-Fuzzy Inference

Indoor scene recognition is complex due to the commonality shared between different spaces. Still, when it comes to robotics applications, the uncertainty increases due to illumination change, motion blur, interruption due to external light sources, and cluttered environments. Most existing fusion approaches do not consider the uncertainty, and others have a high computational cost that may not suit robots with limited resources. To mitigate these issues, this paper proposes a reliable indoor scene recognition approach for mobile robots with limited resources based on robust deep convolutional neural networks (CNNs) feature extractors and neuro-fuzzy inference to consider the uncertainty of the data. All CNN feature extractors are pre-trained on the Imagenet dataset and used in the manner of transfer learning. The performance of our fusion method has been assessed on a customized MIT-67 dataset and for real-time processing on a Locobot robot. We also compare the proposed method with two standard fusion methods---Early Feature Fusion (EFF) and Weighted Average Late Fusion (WALF). The experimental results demonstrate that our method achieves competitive results with a precision of 94%, and it performs well on the Locobot robot with a speed of 3.1 frames per second.

MDFN: Mask deep fusion network for visible and infrared image fusion without reference ground-truth

A single infrared image or visible image cannot clearly present texture details and infrared information of the scene in poor illumination, bad weather, or other complex conditions. Thus, it is necessary to fuse the infrared and visible images into one image. In this paper, we propose a novel deep fusion architecture for fusing visible and infrared images without any reference ground-truth. Different from existing deep image fusion methods which directly output the fused images, a weight score corresponding to each pixel is estimated by our network to determine the contributions of two source images. This strategy transfers the valuable information in source images to the fused image. Considering the salient thermal radiation information in the infrared image, a mask of the infrared image is generated and used to preserve valuable contents in the infrared and visible images for the fused image. Furthermore, a hybrid loss is designed to make the fused image consistent with two source images. On account of the weight estimation, the mask strategy, and the hybrid loss, the images fused by our proposed method jointly maintain the thermal radiation and texture details, achieving state-of-the-art performance compared with existing fusion approaches. Our code is publicly available at https://github.com/NlCxg/MDFN.

MEFuse: end-to-end infrared and visible image fusion method based on multibranch encoder

In recent years, deep learning-based image fusion methods have achieved promising fusion performance. However, existing fusion approaches always focus on increasing the network depth to enhance the feature extraction capacity. In addition, fusion is performed in a fixed feature space, which further degrades the performance. To this end, we proposed an end-to-end infrared and visible image fusion method based on a multibranch encoder, coined MEFuse. In principle, MEFuse increased the number of branches of the encoder, thus mapping the source images into different feature spaces, increasing the diversity of the extracted feature maps, and enhancing the learning capacity of the network. In addition, a multistage fusion strategy was designed to achieve multigranularity image fusion. First, the cascaded multimodal images were merged in the input stage; thus the primary visual fusion was realized in the coding process. Second, the feature maps from multiple coding branches were fused to fuse multimodal information comprehensively. Finally, the long-range lateral connection sent the input appearance of each encoding branch to the decoder to guide the image reconstruction process. Experiments on the benchmark data sets demonstrate that, compared with the existing methods, the proposed MEFuse achieved superior performance in both human perception and objective evaluation.

A Novel GA-Based Optimized Approach for Regional Multimodal Medical Image Fusion With Superpixel Segmentation

For multimodal medical image fusion problems, most of the existing fusion approaches are based on pixel-level. However, the pixel-based fusion method tends to lose local and spatial information as the relationships between pixels are not considered appropriately, which has much influence on the quality of the fusion results. To address this issue, a region-based multimodal medical image fusion framework is proposed based on superpixel segmentation and a post-processing optimization method in this paper. In this framework, the average image of the source medical images is firstly obtained by a weighted averaging method. To effectively obtain homogeneous regions and preserve the complete information of image details, the fast linear spectral clustering(LSC) superpixel algorithm is carried out to segment the average image and get superpixel labels. For each region of the medical images, log-gabor filter(LGF) and sum modified laplacian(SML) are adopted to extract texture feature and contrast feature for the measurement of region importance. The most important regions are selected and the decision map is generated by comparison. Moreover, to get a more accurate decision map, a new post-processing optimized method based on genetic algorithm(GA) is given. A weighted strategy is applied to the extracted features and the weighting factor can be adaptively adjusted by GA. The effectiveness of the proposed fusion method is validated by conducting experiments on eight pairs of medical images from diverse modalities. In addition, seven other mainstream medical image fusion methods are adopted for comparing the performance of fusion. Experimental results in terms of qualitative and quantitative evaluation demonstrate that the proposed method can achieve state-of-the-art performance for multimodal medical image fusion problems.

Nonsubsampled contourlet domain visible and infrared image fusion framework for fire detection using pulse coupled neural network and spatial fuzzy clustering

The rapidly spreading forest fire is always uncontrollable and unpredictable and also has a disastrous effect on the environment as well as human individuals. This is also able to wipe out large acres of forest as well as the agricultural lands. Therefore, in this paper, a fire detection approach is presented in the nonsubsampled contourlet (NSCT) domain by extracting the fused fire regions of visible and infrared (IR) images using spatial fuzzy C-means clustering (SpFCM). Firstly, the NSCT is applied to decompose the source visible and IR images into one low and several high-frequency components. Low-frequency NSCT component is fused using a pulse coupled neural network (PCNN) motivated by the sum-modified Laplacian to retain the maximum information available in both the source images. Local log Gabor energy based fusion rule is employed to fuse the high-frequency NSCT components that are able to preserve the maximum detail information. Later, the fused image is reconstructed by applying the inverse NSCT. Finally, all the fire pixels are identified in the fused images and segmented using the fuzzy clustering approach having spatial information also. Furthermore, several experiments are conducted to evaluate the fire detection ability of the proposed framework in terms of visual appearance as well as several performance evaluation parameters. Experimental results show the superiority of the proposed approach over the other existing fusion approaches by improving all the performance parameters.

Ripplet domain fusion approach for CT and MR medical image information

Multimodal medical image fusion (MIF) plays an important role as an assistant for medical professionals by providing a better visualization of diagnostic information using different imaging modalities. The process of image fusion helps the radiologists in the precise diagnosis of several critical diseases and its treatment. In this paper, the proposed framework presents a fusion approach for multimodal medical images that utilize both the features extracted by the discrete ripplet transform (DRT) and pulse coupled neural network. The DRT having different features and a competent depiction of the image coefficients provides several directional high-frequency subband coefficients. The DRT decomposition can preserve more detailed information present in the reference images and further enhance the visualization of the fused images. Firstly, the DRT is applied to decompose the reference images into several low and high-frequency subimage coefficients that are fused by computing the novel sum modified Laplacian and novel modified spatial frequency motivated pulse coupled neural model. This model is used to preserve the redundant information also. Finally, fused images are reconstructed by applying the inverse DRT. The performance of the proposed fusion approach is validated by extensive simulation on the different CT-MR image datasets. Experimental results demonstrate that the proposed method provides the better fused images in terms of visual quality along with the quantitative measures as compared to several existing fusion approaches.

CT and MR image information fusion scheme using a cascaded framework in ripplet and NSST domain

The fusion of multimodal medical information is considered as an assisted approach for the medical professionals. Computed tomography and magnetic resonance (CT–MR) medical image fusion are able to help the radiologist in precise diagnosis of disease and deciding the required treatment in accord with the patient's condition. Therefore, a cascaded framework is proposed in this study that presents a fusion approach for multimodal medical information in ripplet transform (RT) and non-subsampled shearlet (NSST) domain. The RT and NSST having different features are utilised in a cascade manner that provides several directional decomposition coefficients and increases shift invariance information in the fused images. At the first stage decomposition, a biologically inspired neural model, motivated by novel sum-modified Laplacian and spatial frequency is utilised to fuse the low- and high-frequency coefficients, respectively, and the max fusion rule based on regional energy is utilised at stage 2. This model is used to preserve the redundant information also. The fusion performance is also validated by extensive simulations performed on different CT–MR image datasets of different diseases. Experimental results demonstrate that the proposed method provides better fused images in terms of visual quality along with the quantitative indices compared with several existing fusion approaches.

Nonsubsampled shearlet domain fusion techniques for CT–MR neurological images using improved biological inspired neural model

The fusion of multimodality medical images performs a very crucial role in the clinical diagnosis, analysis and the treatment of especially in critical diseases. It is considered as an assisted approach for the radiologist by providing the composite images having significant diagnostic information acquired from the source images. The main purpose of this work is to develop an efficient framework for fusing the multimodal medical images. Three different fusion techniques are proposed in this paper that presents the CT and MR medical image fusion in nonsubsampled shearlet transform (NSST) domain using the adaptive spiking neural model. The NSST having different features and a competent depiction of the image coefficients provides several directional decomposition coefficients. Maximum selection approach and regional energy are utilized for low frequency coefficients fusion. Spatial frequency, novel modified spatial frequency and novel sum modified Laplacian motivated spiking model are used for every high frequency subimage component. Finally, fused images are reconstructed by applying inverse NSST. The performance of proposed fusion techniques is validated by extensive simulations performed on different CT-MR image datasets using proposed and other thirty seven existing fusion approaches in terms of both the subjective and objective manner. The results revealed that the proposed techniques provide better visualization of resultant images and higher quantitative measures compared to several existing fusion approaches.

Fusion of Visible and Thermal Images Using a Directed Search Method for Face Recognition

A new image fusion algorithm based on the visible and thermal images for face recognition is presented in this paper. The new fusion algorithm derives the benefit from both the modalities images. The proposed fusion process is the weighted sum of thermal and visible face information with two weighting factors [Formula: see text] and [Formula: see text], respectively. The weighting factors are calculated using a directed search algorithm automatically. The proposed fusion framework is evaluated through extensive experiments using UGC-JU face database. Experiments are of three fold. Firstly, individual modalities images are used separately for human face recognition. Secondly, fused face images using the proposed method are used for recognition purpose. The highest level of accuracy achieved by using the proposed method is about 98.42%. Lastly, the three existing fusion methods are applied on the same face database for comparison with the results of the proposed method. All the results demonstrate significant performance improvements in recognition over individual modalities and some of the existing fusion approaches, suggesting that fusion is a viable approach that deserves further study and consideration.

Fusion of CT and MR images using an improved wavelet based method

The fusion of multimodal medical images plays an important role in many clinical applications as it can support more accurate information than any individual source image. This paper presents a novel approach for fusion of computed tomography (CT) and magnetic resonance (MR) images based on wavelet transform. The medical images to be fused are firstly decomposed into multiscale representations by the wavelet transform. Then, by considering the physical meaning of wavelet coefficients and the characteristics of the CT and MR images, the coefficients of the low frequency band and high frequency bands are treated with different schemes: the former is performed with a maximum-selection (MS) rule, and the latter is convolved with a Laplacian operator followed by a MS rule. Finally, the fused image is reconstructed by using the inverse wavelet transform with the combined wavelet coefficients. The performance of our method is qualitatively and quantitatively compared with some existing fusion approaches. The experimental results can demonstrate that the proposed method is a promising and effective technique for fusion of CT and MR images.

Unification of Evidence-Theoretic Fusion Algorithms: A Case Study in Level-2 and Level-3 Fingerprint Features

This paper formulates an evidence-theoretic multimodal unification approach using belief functions that take into account the variability in biometric image characteristics. While processing nonideal images, the variation in the quality of features at different levels of abstraction may cause individual classifiers to generate conflicting genuine-impostor decisions. Existing fusion approaches are nonadaptive and do not always guarantee optimum performance improvements. We propose a contextual unification framework to dynamically select the most appropriate evidence-theoretic fusion algorithm for a given scenario. In the first approach, the unification framework uses deterministic rules to select the most appropriate fusion algorithm; while in the second approach, the framework intelligently learns from the input evidences using a 2nu-granular support vector machine. The effectiveness of the unification approach is experimentally validated by fusing match scores from level-2 and level-3 fingerprint features. Compared to existing fusion algorithms, the proposed unification approach is computationally efficient, and the verification accuracy is not compromised even when conflicting decisions are encountered.

Quality-augmented fusion of level-2 and level-3 fingerprint information using DSm theory

Existing algorithms that fuse level-2 and level-3 fingerprint match scores perform well when the number of features are adequate and the quality of images are acceptable. In practice, fingerprints collected under unconstrained environment neither guarantee the requisite image quality nor the minimum number of features required. This paper presents a novel fusion algorithm that combines fingerprint match scores to provide high accuracy under non-ideal conditions. The match scores obtained from level-2 and level-3 classifiers are first augmented with a quality score that is quantitatively determined by applying redundant discrete wavelet transform to a fingerprint image. We next apply the generalized belief functions of Dezert–Smarandache theory to effectively fuse the quality-augmented match scores obtained from level-2 and level-3 classifiers. Unlike statistical and learning based fusion techniques, the proposed plausible and paradoxical reasoning approach effectively mitigates conflicting decisions obtained from classifiers especially when the evidences are imprecise due to poor image quality or limited fingerprint features. The proposed quality-augmented fusion algorithm is validated using a comprehensive database which comprises of rolled and partial fingerprint images of varying quality with arbitrary number of features. The performance is compared with existing fusion approaches for different challenging realistic scenarios.