Proposed Fusion Scheme Research Articles

Deep evidential fusion with uncertainty quantification and reliability learning for multimodal medical image segmentation

Single-modality medical images generally do not contain enough information to reach an accurate and reliable diagnosis. For this reason, physicians commonly rely on multimodal medical images for comprehensive diagnostic assessments. This study introduces a deep evidential fusion framework designed for segmenting multimodal medical images, leveraging the Dempster–Shafer theory of evidence in conjunction with deep neural networks. In this framework, features are first extracted from each imaging modality using a deep neural network, and features are mapped to Dempster–Shafer mass functions that describe the evidence of each modality at each voxel. The mass functions are then corrected by the contextual discounting operation, using learned coefficients quantifying the reliability of each source of information relative to each class. The discounted evidence from each modality is then combined using Dempster’s rule of combination. Experiments were carried out on a PET-CT dataset for lymphoma segmentation and a multi-MRI dataset for brain tumor segmentation. The results demonstrate the ability of the proposed fusion scheme to quantify segmentation uncertainty and improve segmentation accuracy. Moreover, the learned reliability coefficients provide some insight into the contribution of each modality to the segmentation process.

Multimodal and Multiuser Semantic Communications for Channel-Level Information Fusion

Recently, semantic communication emerged as one of the enabling technologies for intelligent communications and multimodal information fusion plays an important role therein, where various sensors collect environment information to generate complementary multimodal data. In this paper, we propose a new multimodal information fusion scheme for multi-user semantic communications, namely channel-level information fusion, taking into account the nature of signal transmission in wireless channels. In the proposed scheme, wireless channel acts as a medium to fuse multimodal data and the received signal is viewed as the fused information, such that a receiver can retrieve semantic information without a need to perform multiuser signal detection. In addition, semantic precoding is used to mitigate the randomness effect of wireless channels in fusion. A case study is conducted to validate the effectiveness of the proposed fusion scheme, followed by a discussion on the open issues and challenges to be tackled.

Advancing multimodal medical image fusion: an adaptive image decomposition approach based on multilevel Guided filtering.

With the rapid development of medical imaging methods, multimodal medical image fusion techniques have caught the interest of researchers. The aim is to preserve information from diverse sensors using various models to generate a single informative image. The main challenge is to derive a trade-off between the spatial and spectral qualities of the resulting fused image and the computing efficiency. This article proposes a fast and reliable method for medical image fusion depending on multilevel Guided edge-preserving filtering (MLGEPF) decomposition rule. First, each multimodal medical image was divided into three sublayer categories using an MLGEPF decomposition scheme: small-scale component, large-scale component and background component. Secondly, two fusion strategies-pulse-coupled neural network based on the structure tensor and maximum based-are applied to combine the three types of layers, based on the layers' various properties. The three different types of fused sublayers are combined to create the fused image at the end. A total of 40 pairs of brain images from four separate categories of medical conditions were tested in experiments. The pair of images includes various case studies including magnetic resonance imaging (MRI) , TITc, single-photon emission computed tomography (SPECT) and positron emission tomography (PET). We included qualitative analysis to demonstrate that the visual contrast between the structure and the surrounding tissue is increased in our proposed method. To further enhance the visual comparison, we asked a group of observers to compare our method's outputs with other methods and score them. Overall, our proposed fusion scheme increased the visual contrast and received positive subjective review. Moreover, objective assessment indicators for each category of medical conditions are also included. Our method achieves a high evaluation outcome on feature mutual information (FMI), the sum of correlation of differences (SCD), Qabf and Qy indexes. This implies that our fusion algorithm has better performance in information preservation and efficient structural and visual transferring.

Adaptive convolutional sparsity with sub-band correlation in the NSCT domain for MRI image fusion

Objective. Multimodal medical image fusion (MMIF) technologies merges diverse medical images with rich information, boosting diagnostic efficiency and accuracy. Due to global optimization and single-valued nature, convolutional sparse representation (CSR) outshines the standard sparse representation (SR) in significance. By addressing the challenges of sensitivity to highly redundant dictionaries and robustness to misregistration, an adaptive convolutional sparsity scheme with measurement of the sub-band correlation in the non-subsampled contourlet transform (NSCT) domain is proposed for MMIF. Approach. The fusion scheme incorporates four main components: image decomposition into two scales, fusion of detail layers, fusion of base layers, and reconstruction of the two scales. We solved a Tikhonov regularization optimization problem with source images to obtain the base and detail layers. Then, after CSR processing, detail layers were sparsely decomposed using pre-trained dictionary filters for initial coefficient maps. NSCT domain’s sub-band correlation was used to refine fusion coefficient maps, and sparse reconstruction produced the fused detail layer. Meanwhile, base layers were fused using averaging. The final fused image was obtained via two-scale reconstruction. Main results. Experimental validation of clinical image sets revealed that the proposed fusion scheme can not only effectively eliminate the interference of partial misregistration, but also outperform the representative state-of-the-art fusion schemes in the preservation of structural and textural details according to subjective visual evaluations and objective quality evaluations. Significance. The proposed fusion scheme is competitive due to its low-redundancy dictionary, robustness to misregistration, and better fusion performance. This is achieved by training the dictionary with minimal samples through CSR to adaptively preserve overcompleteness for detail layers, and constructing fusion activity level with sub-band correlation in the NSCT domain to maintain CSR attributes. Additionally, ordering the NSCT for reverse sparse representation further enhances sub-band correlation to promote the preservation of structural and textural details.

Elasticity unleashed: Fine-grained cloud scaling through distributed three-way decision fusion with multi-head attention

With the proliferation of cloud computing, elastic resource scaling has emerged as a critical challenge. Cloud platforms must efficiently allocate resources to match dynamic workloads. This study addresses the key trade-off between responsiveness and prudence in scaling a novel distributed three-way decision fusion approach. Distributed three-way decisions of immediate, delayed, or no scaling are made at coarse and fine-time granularity levels. Subsequently, a multi-head attention mechanism intelligently fuses these decisions, weighing the relevance of the different granularities to synthesize an integrated scaling strategy. The reactiveness of sudden workload changes are balanced during fusion by considering long-term trends. Comprehensive experiments on real-world data demonstrated that the proposed fusion strategy substantially improves resource efficiency and adherence to service-level agreements compared to existing methods. Multi-head attention imparts autonomy in adapting to diverse operating conditions. The proposed principled methodology and attention-based decision aggregation hold significance for efficient, adaptive, and interpretable cloud scaling mechanisms.

Improving self-supervised learning model for audio spoofing detection with layer-conditioned embedding fusion

The application of voice recognition systems has increased by a great deal with technology. This has allowed adversaries to falsely claim access to these systems by spoofing the identity of a target speaker. The existing supervised learning (SL)-based countermeasures are yet to provide a complete solution against the newly evolving spoofing attacks. To tackle this problem, we explore self-supervised learning (SSL)-based frameworks. At first, we implement widely used SSL frameworks, where our target is identifying spoofed speech. We report a considerable performance improvement over the SL state-of-the-art baseline as a whole. Then, we perform an attack-wise comparative analysis between SL and SSL frameworks. While the SSL performs better in most cases, there are certain attacks where the SL outperforms it. Hence, we hypothesize that there is scope to jointly utilize information effectively included by both these models for better performance. To do that, we first perform conventional weighted score fusion between the SL and best-performing SSL models, which reduces the EER, outperforming both the state-of-the-art SL and best-performing SSL framework. Then, we propose an embedding fusion scheme that minimizes the embedding distribution between the selected SL and SSL representations. To select the appropriate layers, we perform a comprehensive statistical analysis. The proposed fusion scheme outperforms the score fusion method and shows that the SSL performance can be improved by effectively including learned knowledge from the SL framework. The final EER achieved on the ASVspoof 2019 logical access (LA) database is 0.177%, a significant improvement over our baseline. Using the ASVspoof 2021 LA as a blind evaluation dataset, our proposed embedding fusion scheme reduces the EER to 2.666%.

A Low-Cost and Robust Multi-Sensor Data Fusion Scheme for Heterogeneous Multi-Robot Cooperative Positioning in Indoor Environments

The latest development of multi-robot collaborative systems has put forward higher requirements for multi-sensor fusion localization. Current position methods mainly focus on the fusion of the carrier’s own sensor information, and how to fully utilize the information of multiple robots to achieve high-precision positioning is a major challenge. However, due to the comprehensive impact of factors such as poor performance, variety, complex calculations, and accumulation of environmental errors used by commercial robots, the difficulty of high-precision collaborative positioning is further exacerbated. To address this challenge, we propose a low-cost and robust multi-sensor data fusion scheme for heterogeneous multi-robot collaborative navigation in indoor environments, which integrates data from inertial measurement units (IMUs), laser rangefinders, cameras, and so on, into heterogeneous multi-robot navigation. Based on Discrete Kalman Filter (DKF) and Extended Kalman Filter (EKF) principles, a three-step joint filtering model is used to improve the state estimation and the visual data are processed using the YOLO deep learning target detection algorithm before updating the integrated filter. The proposed integration is tested at multiple levels in an open indoor environment following various formation paths. The results show that the three-dimensional root mean square error (RMSE) of indoor cooperative localization is 11.3 mm, the maximum error is less than 21.4 mm, and the motion error in occluded environments is suppressed. The proposed fusion scheme is able to satisfy the localization accuracy requirements for efficient and coordinated motion of autonomous mobile robots.

OFDM sensing based on deep learning

Detection of OFDM signals has received paramount interest especially for spectrum sensing in cognitive radios. We present a deep learning-based OFDM signal detection method, namely OFDM-DetNet, which combines feature learning and classifier in a single densely connected convolutional network (DenseNet) structure. OFDM-DetNet completes the detection in an end-to-end manner and utilizes the raw in-phase (I) and quadrature (Q) components of the received signal as the input and the detection result as the output. In addition, a confidence-based fusion method, namely OFDM-DetNet-F, is proposed to further improve the detection performance under low signal-to-noise ratio (SNR) scenarios without retraining the network. We construct three datasets with simulations and three datasets with real-world signals to evaluate the performance of the proposed methods. Both simulation results and experimental results indicate the superior performance of the proposed OFDM-DetNet compared with two deep learning-based OFDM detection methods, i.e., stacked autoencoder (SAE)-based method and covariance matrix (CM)-based detection method, in terms of probability of detection. Furthermore, the performance can be further improved with our proposed fusion scheme in low SNR region.

A fusion scheme for eliminating input interference induced by spelling errors

Spelling errors are a prevalent form of interference that may result from human handwriting or natural language processing systems, such as optical character recognition. This type of interference presents a challenge for natural language processing systems that rely on context-based embeddings to comprehend the exact semantics of a sentence. The proposed fusion scheme is the main focus of this paper, which aims to correct spelling errors in sentences. The scheme employs both the original input and the masked input. The masked input is acquired by utilizing a detection module to mask any spelling errors present in the original input. By utilizing the masked input, the scheme is capable of learning the real sentence semantic information that is not affected by spelling errors. Utilizing the original input can prevent the loss of input information resulting from masking operations. Finally, the utilization of the Kullback-Leibler divergence is employed to establish concordance between the two output distributions within the framework, consequently facilitating the acquisition of comprehensive input information, all while mitigating the influence of spelling errors. Experiments were conducted on two widely used benchmarks. Compared with previous methods, the approach presented herein exhibited superior performance as it achieved improvements on two benchmarks.

PADR: Towards Personalized Adverse Drug Reaction Prediction by Modeling Multi-sourced Data.

Predicting adverse drug reactions (ADRs) of drugs is one of the most critical steps in drug development. By pre-estimating the adverse reactions, researchers and drug development companies can greatly prevent the potential ADR risks and tragedies. However, the current ADR prediction methods suffer from several limitations. First, the prediction results are based on pure drug-related information, which makes them impossible to be directly applied for the personalized ADR prediction task. The lack of personalization of models also makes rare adverse events hard to be predicted. Therefore, it is of great interest to develop a new personalized ADR prediction method by introducing additional sources, e.g., patient health records. However, few methods have tried to use additional sources. In the meantime, the variety of different source formats and structures makes this task more challenging. To address the above challenges, we propose a novel personalized multi-sourced-based drug adverse reaction prediction model named pADR. pADR first works on every single source to transform them into proper representations. Next, a hierarchical multi-sourced Transformer is designed to automatically model the interactions between different sources and fuse them together for the final adverse event prediction. Experimental results on a new multi-sourced ADR prediction dataset show that PADR outperforms state-of-the-art drug-based baselines. Moreover, the case and ablation studies also illustrate the effectiveness of our proposed fusion strategies and the reasonableness of each module design.

Deterministic learning-based neural identification and knowledge fusion

Recent deterministic learning methods have achieved locally-accurate identification of unknown system dynamics. However, the locally-accurate identification means that the neural networks can only capture the local dynamics knowledge along the system trajectory. In order to capture a broader knowledge region, this article investigates the knowledge fusion problem of deterministic learning, that is, the integration of different knowledge regions along different individual trajectories. Specifically, two kinds of knowledge fusion schemes are systematically introduced: an online fusion scheme and an offline fusion scheme. The online scheme can be viewed as an extension of distributed cooperative learning control to cooperative neural identification for sampled-data systems. By designing an auxiliary information transmission strategy to enable the neural network to receive information learned from other tasks while learning its own task, it is proven that the weights of all localized RBF networks exponentially converge to their common true/ideal values. The offline scheme can be regarded as a knowledge distillation strategy, in which the fused network is obtained by offline training through the knowledge learned from all individual system trajectories via deterministic learning. A novel weight fusion algorithm with low computational complexity is proposed based on the least squares solution under subspace constraints. Simulation studies show that the proposed fusion schemes can successfully integrate the knowledge regions of different individual trajectories while maintaining the learning performance, thereby greatly expanding the knowledge region learned from deterministic learning.

Identification of mango variety using near infrared spectroscopy

<p>The structure of the proposed framework is separated into three stages: i) foundation deduction, ii) component extraction, and iii) preparing and characterization. At first, K-implies grouping methods were carried out for foundation de- duction. The second step applies color, texture, and shape-based feature extraction methods. Finally, a “merging” fusion feature is analyzed with a C4.5, support vector machine (SVM), and K-nearest neighbors (KNN). Overall, the recognition system produces an adequate performance accuracy with 97.89, 94.60, and 90.25 percent values by utilizing C4.5, SVM, and KNN, respectively. The experimentation points out that the proposed fusion scheme can significantly support accurately recognizing various fruits and vegetables.</p>

Indirect tire pressure monitoring method based on the fusion of time and frequency domain analysis

Monitoring tire pressure conditions and issuing timely warnings of tire deflation are important to ensure driving safety and avoid unnecessary fuel consumption. Although installing pressure sensors is a direct and accurate monitoring method, it may raise system cost and complexity. The indirect monitoring method with no reliance on pressure sensors is attractive for economical cars. This paper proposes a fusion-based indirect tire pressure monitoring method using existing on-board wheel speed sensors. The screening basis of reasonable wheel speed signals was determined by data mining of typical driving scenarios. Three typical signal distortions were effectively corrected to ensure the signal quality. Tire deflation analysis methods based on time and frequency domain signals were designed and fused using fuzzy logic. Validation results of the proposed two analysis methods and their fusion strategy showed that the proposed fusion strategy can achieve more comprehensive tire deflation monitoring with fast response and high accuracy.

An interactive opposition matrix and regulatory gene fusion strategy for evolutionary multitasking

Evolutionary multitasking algorithms (EMT) study how to solve multiple optimization tasks simultaneously by evolutionary computation, and investigate how knowledge sharing can accelerate the convergence of individual tasks, meaning that useful knowledge gained in solving one task can be used to solve other tasks. However, as the evolutionary search continues, the learnability among tasks may decrease, leading to a decrease in the efficiency of knowledge transfer and affecting the population evolution. To solve this problem, a new multifactorial evolutionary algorithm (MFEA-VOM) is proposed in this paper, which applies to three strategies, namely, implicit conversion strategy, opposition matrix strategy, and regulatory gene fusion strategy. The implicit conversion strategy is applied to minimize the threat of negative knowledge migration and reduce the impact caused by negative knowledge migration. The proposed opposition matrix strategy explores more unknown areas of the population and improves the exploration ability of the population by further exploring and utilizing the unified search space, transforming the parent individuals into an appropriate task through mapping relationships, and reducing the gap between tasks. The proposed regulatory gene fusion strategy is applied to the reproduction of individuals to produce better individuals applicable to the task, submitting the efficiency of knowledge transfer. Through a comprehensive experimental analysis of the EMT optimization problem, the experimental results demonstrate the better performance of MFEA-VOM compared to other EMT algorithms.

Suppression of Self-Noise and Environmental Disturbances by Fusion in Fiber-Optic Gyroscopes for Precision Measurement

We present a study of noise suppression by fusion in fiber-optic gyroscopes based on the source-sharing architecture. A theoretical model is proposed for the detailed analysis of noise mechanism. The importance of the modulation parameters is revealed, and the optimal modulation strategies are given. Through simulations and experiments, we demonstrate that the proposed fusion scheme not only suppresses self-noise including relative intensity noise and thermal phase noise, but also achieves complete elimination of environmental disturbances. Up to 5.8-fold suppression of self-noise and 5.2-fold reduction of bias instability is obtained in the experimental apparatus when compared with the performance before fusion. In the applications of fiber-optic gyroscopes for precision measurement, the proposed schemes effectively suppress the noises at the system level.

V-SLAM ENHANCED INS/GNSS FUSION SCHEME FOR LANE LEVEL VEHICULAR NAVIGATION APPLICATIONS IN DYNAMIC ENVIRONMENT

Abstract. With the development of different sensors, such as global navigation satellite system (GNSS), inertial measurement unit (IMU), LiDAR, radar and camera, more localization information is available for autonomous vehicular applications. However, each sensor has its limitations in different circumstances. For example, visual Simultaneous Localization and Mapping (SLAM) easily loses tracking in an open sky area where accurate GNSS measurements can be obtained. Sensors can complement each other by integrated their information in a multi-sensor fusion scheme. In this study, we proposed a visual-SLAM enhanced INS/GNSS localization fusion scheme for a high dynamic environment. Oriented FAST and rotated BRIEF (ORB) SLAM are used to pre-process image sequences from monocular camera, rescaled and refreshed after applying GNSS measurements, and convert to position and velocity information, which can provide updates to the system. The performance of the fusion system was verified through two field tests at different speed ranges (about 30–60 km/s), using a reliable reference system as ground-truth to assess the accuracy of the proposed localization fusion scheme. The results indicated that the proposed system could improve the navigation accuracy compared to INS/GNSS integration scheme and achieve which-lane level or even where-in-lane level.

Improving Road Segmentation by Combining Satellite Images and LiDAR Data with a Feature-Wise Fusion Strategy

Numerous deep learning techniques have been explored in pursuit of achieving precise road segmentation; nonetheless, this task continues to present a significant challenge. Exposing shadows and the obstruction of objects are the most important difficulties associated with road segmentation using optical image data alone. By incorporating additional data sources, such as LiDAR data, the accuracy of road segmentation can be improved in areas where optical images are insufficient to segment roads properly. The missing information in spectral data due to the object blockage and shadow effect can be compensated by the integration of 2D and 3D information. This study proposes a feature-wise fusion strategy of optical images and point clouds to enhance the road segmentation performance of a deep learning model. For this purpose, high-resolution satellite images and airborne LiDAR point cloud collected over Florida, USA, were used. Eigenvalue-based and geometric 3D property-based features were calculated based on the LiDAR data. These optical images and LiDAR-based features were used together to train, end-to-end, a deep residual U-Net architecture. In this strategy, the high-level features generated from optical images were concatenated with the LiDAR-based features before the final convolution layer. The consistency of the proposed strategy was evaluated using ResNet backbones with a different number of layers. According to the obtained results, the proposed fusion strategy improved the prediction capacity of the U-Net models with different ResNet backbones. Regardless of the backbone, all models showed enhancement in prediction statistics by 1% to 5%. The combination of optical images and LiDAR point cloud in the deep learning model has increased the prediction performance and provided the integrity of road geometry in woodland and shadowed areas.

Evidential transformer for pavement distress segmentation

AbstractDistress segmentation assigns each pixel of a pavement image to one distress class or background, which provides a simplified representation for distress detection and measurement. Even though remarkably benefiting from deep learning, distress segmentation still faces the problems of poor calibration and multimodel fusion. This study has proposed a deep neural network by combining the Dempster–Shafer theory (DST) and a transformer network for pavement distress segmentation. The network, called the evidential segmentation transformer, uses its transformer backbone to obtain pixel‐wise features from input images. The features are then converted into pixel‐wise mass functions by a DST‐based evidence layer. The pixel‐wise masses are utilized for performing distress segmentation based on the pignistic criterion. The proposed network is iteratively trained by a new learning strategy, which represents uncertain information of ambiguous pixels by mass functions. In addition, an evidential fusion strategy is proposed to fuse heterogeneous transformers with different distress classes. Experiments using three public data sets (Pavementscape, Crack500, and CrackDataset) show that the proposed networks achieve state‐of‐the‐art accuracy and calibration on distress segmentation, which allows for measuring the distress shapes more accurately and stably. The proposed fusion strategy combines heterogeneous transformers while remaining a performance not less than those of the individual networks on their respective data sets. Thus, the fusion strategy makes it possible to use the existing networks to build a more general and accurate one for distress segmentation.

Spatiotemporal dynamic graph convolutional network for traffic speed forecasting

Accurate traffic speed forecasting is challenging because of complex spatiotemporal correlations of traffic data. Some studies have recognized that correlations among sensors change gradually and have constructed dynamic graphs to reflect this change. However, they ignored the temporal dependencies among dynamic graphs, resulting in an inability to capture the deep dynamic dependencies among sensors. Few studies have explored the hybrid interaction patterns of static and dynamic graphs, hindering the adequate exploration of spatial dependencies. Therefore, a novel deep learning model is proposed in this study to address these problems and produce accurate traffic speed forecasting. First, a new graph generation method is proposed to capture the deep dynamic dependencies among sensors, which exploits historical information of dynamic graphs to develop temporal dependencies among dynamic graphs. Then, a new fusion strategy is proposed to investigate the hybrid interaction patterns of static and dynamic graphs, which effectively mines hidden information in graphs to fully extract spatial dependencies. Finally, a new spatiotemporal network architecture is proposed, which unifies the proposed graph generation method and fusion strategy into a consistent framework and yields the final forecasting results. Experimental results on two real-world traffic datasets indicate that the proposed model outperforms state-of-the-art models.

Estimation of Respiratory Rate during Biking with a Single Sensor Functional Near-Infrared Spectroscopy (fNIRS) System

The employment of wearable systems for continuous monitoring of vital signs is increasing. However, due to substantial susceptibility of conventional bio-signals recorded by wearable systems to motion artifacts, estimation of the respiratory rate (RR) during physical activities is a challenging task. Alternatively, functional Near-Infrared Spectroscopy (fNIRS) can be used, which has been proven less vulnerable to the subject's movements. This paper proposes a fusion-based method for estimating RR during bicycling from fNIRS signals recorded by a wearable system. Firstly, five respiratory modulations are extracted, based on amplitude, frequency, and intensity of the oxygenated hemoglobin concentration (O2Hb) signal. Secondly, the dominant frequency of each modulation is computed using the fast Fourier transform. Finally, dominant frequencies of all modulations are fused, based on averaging, to estimate RR. The performance of the proposed method was validated on 22 young healthy subjects, whose respiratory and fNIRS signals were simultaneously recorded during a bicycling task, and compared against a zero delay Fourier domain band-pass filter. The comparison between results obtained by the proposed method and band-pass filtering indicated the superiority of the former, with a lower mean absolute error (3.66 vs. 11.06 breaths per minute, p<0.05). The proposed fusion strategy also outperformed RR estimations based on the analysis of individual modulation. This study orients towards the practical limitations of traditional bio-signals for RR estimation during physical activities.