Hybrid Fusion Scheme Research Articles

FusionNGFPE: An Image Fusion Approach Driven by Non-global Fuzzy Pre-enhancement Framework

The majority of prevailing image fusion methods employ a global strategy, often resulting in a reduction of contrast. This study addresses this issue by proposing a novel image fusion approach called FusionNGFPE, specifically designed for the structural characteristics of infrared (IR) imagery. The approach introduces a contrast equalization algorithm based on the Fourth-order Partial Differential Equation (FPDE) to enhance background regions effectively. Considering the inherent differences between IR and visible (VIS) images, we developed a hybrid fusion strategy that combines the Expectation Maximization (EM) algorithm and Principal Component Analysis (PCA). Comparative analysis with state-of-the-art fusion methods shows that our proposed algorithm achieves superior performance in both qualitative and quantitative evaluations. To further demonstrate the practical significance of FusionNGFPE, we integrated this fusion framework into the RGBT target tracking task using the VOT-RGBT and OTCBVS datasets. Extensive comparative experiments confirm that the FusionNGFPE framework integrates seamlessly with the tracking task, significantly improving tracking accuracy across diverse scenarios.

MMDTA: A Multimodal Deep Model for Drug-Target Affinity with a Hybrid Fusion Strategy.

The prediction of the drug-target affinity (DTA) plays an important role in evaluating molecular druggability. Although deep learning-based models for DTA prediction have been extensively attempted, there are rare reports on multimodal models that leverage various fusion strategies to exploit heterogeneous information from multiple different modalities of drugs and targets. In this study, we proposed a multimodal deep model named MMDTA, which integrated the heterogeneous information from various modalities of drugs and targets using a hybrid fusion strategy to enhance DTA prediction. To achieve this, MMDTA first employed convolutional neural networks (CNNs) and graph convolutional networks (GCNs) to extract diverse heterogeneous information from the sequences and structures of drugs and targets. It then utilized a hybrid fusion strategy to combine and complement the extracted heterogeneous information, resulting in the fused modal information for predicting drug-target affinity through the fully connected (FC) layers. Experimental results demonstrated that MMDTA outperformed the competitive state-of-the-art deep learning models on the widely used benchmark data sets, particularly with a significantly improved key evaluation metric, Root Mean Square Error (RMSE). Furthermore, MMDTA exhibited excellent generalization and practical application performance on multiple different data sets. These findings highlighted MMDTA's accuracy and reliability in predicting the drug-target binding affinity. For researchers interested in the source data and code, they are accessible at http://github.com/dldxzx/MMDTA.

SMIN: Semi-Supervised Multi-Modal Interaction Network for Conversational Emotion Recognition

Conversational emotion recognition is a crucial research topic in human-computer interactions. Due to the heavy annotation cost and inevitable label ambiguity, collecting large amounts of labeled data is challenging and expensive, which restricts the performance of current fully-supervised methods in this domain. To address this problem, researchers attempt to distill knowledge from unlabeled data via semi-supervised learning. However, most of these semi-supervised methods ignore multimodal interactive information, although recent works have proven that such interactive information is essential for emotion recognition. To this end, we propose a novel framework to seamlessly integrate semi-supervised learning with multimodal interactions, called ‘`Semi-supervised Multi-modal Interaction Network (SMIN)’'. SMIN contains two well-designed semi-supervised modules, ‘`Intra-modal Interactive Module (IIM)’' and ‘`Cross-modal Interactive Module (CIM)’' to learn intra- and cross-modal interactions. These two modules leverage additional unlabeled data to extract emotion-salient representations. To capture additional contextual information, we utilize the hierarchical recurrent networks followed with the hybrid fusion strategy to integrate multimodal features. These multimodal features are further utilized for conversational emotion recognition. Experimental results on four benchmark datasets (i.e., IEMOCAP, MELD, CMU-MOSI and CMU-MOSEI) demonstrate that SMIN succeeds over existing state-of-the-art strategies on emotion recognition.

Let the Data Choose: Flexible and Diverse Anchor Graph Fusion for Scalable Multi-View Clustering

In the past few years, numerous multi-view graph clustering algorithms have been proposed to enhance the clustering performance by exploring information from multiple views. Despite the superior performance, the high time and space expenditures limit their scalability. Accordingly, anchor graph learning has been introduced to alleviate the computational complexity. However, existing approaches can be further improved by the following considerations: (i) Existing anchor-based methods share the same number of anchors across views. This strategy violates the diversity and flexibility of multi-view data distribution. (ii) Searching for the optimal anchor number within hyper-parameters takes much extra tuning time, which makes existing methods impractical. (iii) How to flexibly fuse multi-view anchor graphs of diverse sizes has not been well explored in existing literature. To address the above issues, we propose a novel anchor-based method termed Flexible and Diverse Anchor Graph Fusion for Scalable Multi-view Clustering (FDAGF) in this paper. Instead of manually tuning optimal anchor with massive hyper-parameters, we propose to optimize the contribution weights of a group of pre-defined anchor numbers to avoid extra time expenditure among views. Most importantly, we propose a novel hybrid fusion strategy for multi-size anchor graphs with theoretical proof, which allows flexible and diverse anchor graph fusion. Then, an efficient linear optimization algorithm is proposed to solve the resultant problem. Comprehensive experimental results demonstrate the effectiveness and efficiency of our proposed framework. The source code is available at https://github.com/Jeaninezpp/FDAGF.

Multimodal transformer augmented fusion for speech emotion recognition.

Speech emotion recognition is challenging due to the subjectivity and ambiguity of emotion. In recent years, multimodal methods for speech emotion recognition have achieved promising results. However, due to the heterogeneity of data from different modalities, effectively integrating different modal information remains a difficulty and breakthrough point of the research. Moreover, in view of the limitations of feature-level fusion and decision-level fusion methods, capturing fine-grained modal interactions has often been neglected in previous studies. We propose a method named multimodal transformer augmented fusion that uses a hybrid fusion strategy, combing feature-level fusion and model-level fusion methods, to perform fine-grained information interaction within and between modalities. A Model-fusion module composed of three Cross-Transformer Encoders is proposed to generate multimodal emotional representation for modal guidance and information fusion. Specifically, the multimodal features obtained by feature-level fusion and text features are used to enhance speech features. Our proposed method outperforms existing state-of-the-art approaches on the IEMOCAP and MELD dataset.

Multimodal Sentiment Analysis Using BiGRU and Attention-Based Hybrid Fusion Strategy

Multimodal Sentiment Analysis Using BiGRU and Attention-Based Hybrid Fusion Strategy

Research on the Application of Multi-Source Information Fusion in Multiple Gait Pattern Transition Recognition.

Multi-source information fusion technology is a kind of information processing technology which comprehensively processes and utilizes multi-source uncertain information. It is an effective scheme to solve complex pattern recognition and improve classification performance. This study aims to improve the accuracy and robustness of exoskeleton gait pattern transition recognition in complex environments. Based on the theory of multi-source information fusion, this paper explored a multi-source information fusion model for exoskeleton gait pattern transition recognition in terms of two aspects of multi-source information fusion strategy and multi-classifier fusion. For eight common gait pattern transitions (between level and stair walking and between level and ramp walking), we proposed a hybrid fusion strategy of multi-source information at the feature level and decision level. We first selected an optimal feature subset through correlation feature extraction and feature selection algorithm, followed by the feature fusion through the classifier. We then studied the construction of a multi-classifier fusion model with a focus on the selection of base classifier and multi-classifier fusion algorithm. By analyzing the classification performance and robustness of the multi-classifier fusion model integrating multiple classifier combinations with a number of multi-classifier fusion algorithms, we finally constructed a multi-classifier fusion model based on D-S evidence theory and the combination of three SVM classifiers with different kernel functions (linear, RBF, polynomial). Such multi-source information fusion model improved the anti-interference and fault tolerance of the model through the hybrid fusion strategy of feature level and decision level and had higher accuracy and robustness in the gait pattern transition recognition, whose average recognition accuracy for eight gait pattern transitions reached 99.70%, which increased by 0.15% compared with the highest average recognition accuracy of the single classifier. Moreover, the average recognition accuracy in the absence of different feature data reached 97.47% with good robustness.

A hybrid fusion precipitation bias correction approach for Yin‐He global spectral model

In this article, we explored the performance of several fusion strategies for bias correction of precipitation on U-Net-based-models and proposed a simple but efficient hybrid fusion strategy. The geopotential, vertical velocity, specific humidity and 3-h cumulative precipitation from Yin-He global spectral model (YHGSM) re-forecast products are used as multiple correction factors, and the 3-h cumulative precipitation calculated from ERA5 are used as labels. Experimental results reveal that, our hybrid fusion strategy can reduce the number of parameters by about 40% and obtain better fraction skill score (FSS) and threat score (TS). For TS of 0.1, 3.0, 10.0 and 20. 0 mm, U-Net with hybrid fusion improves 31.8%, 97.0%, 314.0% and 576.0% than YHGSM, improves 1.3%, 3.0%, 3.6% and 10.5% than the classical U-Net. This hybrid fusion strategy should provide a feasible approach to utilize multiple information inputs more efficiently in geophysics field.

An Enhanced Smartphone Indoor Positioning Scheme with Outlier Removal Using Machine Learning

In smartphone indoor positioning, owing to the strong complementarity between pedestrian dead reckoning (PDR) and WiFi, a hybrid fusion scheme of them is drawing more and more attention. However, the outlier of WiFi will easily degrade the performance of the scheme, to remove them, many researches have been proposed such as: improving the WiFi individually or enhancing the scheme. Nevertheless, due to the inherent received signal strength (RSS) variation, there still exist some unremoved outliers. To solve this problem, this paper proposes the first outlier detection and removal strategy with the aid of Machine Learning (ML), so called WiFi-AGNES (Agglomerative Nesting), based on the extracted positioning characteristics of WiFi when the pedestrian is static. Then, the paper proposes the second outlier detection and removal strategy, so called WiFi-Chain, based on the extracted positioning characteristics of WiFi, PDR, and their complementary characteristics when the pedestrian is walking. Finally, a hybrid fusion scheme is proposed, which integrates the two proposed strategies, WiFi, PDR with an inertial-navigation-system-based (INS-based) attitude heading reference system (AHRS) via Extended Kalman Filter (EKF), and an Unscented Kalman Filter (UKF). The experiment results show that the two proposed strategies are effective and robust. With WiFi-AGNES, the minimum percentage of the maximum error (MaxE) is reduced by 66.5%; with WiFi-Chain, the MaxE of WiFi is less than 4.3 m; further the proposed scheme achieves the best performance, where the root mean square error (RMSE) is 1.43 m. Moreover, since characteristics are universal, the proposed scheme integrated the two characteristic-based strategies also possesses strong robustness.

Bi-LSTM Network for Multimodal Continuous Human Activity Recognition and Fall Detection

This paper presents a framework based on multi-layer bi-LSTM network (bidirectional Long Short-Term Memory) for multimodal sensor fusion to sense and classify daily activities’ patterns and high-risk events such as falls. The data collected in this work are continuous activity streams from FMCW radar and three wearable inertial sensors on the wrist, waist, and ankle. Each activity has a variable duration in the data stream so that the transitions between activities can happen at random times within the stream, without resorting to conventional fixed-duration snapshots. The proposed bi-LSTM implements soft feature fusion between wearable sensors and radar data, as well as two robust hard-fusion methods using the confusion matrices of both sensors. A novel hybrid fusion scheme is then proposed to combine soft and hard fusion to push the classification performances to approximately 96% accuracy in identifying continuous activities and fall events. These fusion schemes implemented with the proposed bi-LSTM network are compared with conventional sliding window approach, and all are validated with realistic “leaving one participant out” (L1PO) method (i.e. testing subjects unknown to the classifier). The developed hybrid-fusion approach is capable of stabilizing the classification performance among different participants in terms of reducing accuracy variance of up to 18.1% and increasing minimum, worst-case accuracy up to 16.2%.

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

Multimodal emotion recognition has gained traction in affective computing research community to overcome the limitations posed by the processing a single form of data and to increase recognition robustness. In this study, a novel emotion recognition system is introduced, which is based on multiple modalities including facial expressions, galvanic skin response (GSR) and electroencephalogram (EEG). This method follows a hybrid fusion strategy and yields a maximum one-subject-out accuracy of 81.2% and a mean accuracy of 74.2% on our bespoke multimodal emotion dataset (LUMED-2) for 3 emotion classes: sad, neutral and happy. Similarly, our approach yields a maximum one-subject-out accuracy of 91.5% and a mean accuracy of 53.8% on the Database for Emotion Analysis using Physiological Signals (DEAP) for varying numbers of emotion classes, 4 in average, including angry, disgust, afraid, happy, neutral, sad and surprised. The presented model is particularly useful in determining the correct emotional state in the case of natural deceptive facial expressions. In terms of emotion recognition accuracy, this study is superior to, or on par with, the reference subject-independent multimodal emotion recognition studies introduced in the literature.

A Hybrid Fusion Strategy for the Land Vehicle Navigation Using MEMS INS, Odometer and GNSS

How to limit the drifts of the navigation errors in an inertial navigation system (INS) with low-cost sensors is one of the main challenges for the land vehicle navigations. In this paper, we present a novel hybrid navigation strategy to integrate the Micro-Electric-Mechanic-systems (MEMS) INS, odometer (OD) and global navigation satellite systems (GNSS), with aim to enhance the positioning accuracy of the inertial system during GNSS outages. To accurately estimate the INS error states, the neural network (NN) is proposed to mimic the velocity of the navigation frame with the data from the MEMS INS, odometer, as well as the non-holonomic constraints (NHC). The long short-term memory (LSTM) NN is adopted in our approach due to its ability to adaptively use the data in the past. The road tests are conducted with two different MEMS IMUs to verify the proposed navigation strategy. Comparing to the traditional integrated MEMS INS/OD/GNSS system based on the extended Kalman filtering (EKF), our hybrid approach provides over 60% improvements in terms of the root mean square (RMS) and maximum horizontal position errors during GNSS outages.

Deriving a Forest Cover Map in Kyrgyzstan Using a Hybrid Fusion Strategy

Forests have potential economic value and play a significant role in maintaining ecological balance. Considering its outdated and incomplete forest statistics, the Kyrgyzstan Republic urgently needs a forest cover map for assessing its current forest resources and assisting national policies on improving rural livelihood and sustainability. This study adopted a hybrid fusion strategy to develop a forest cover map for the Kyrgyzstan Republic with improved accuracy. The fusion strategy uses the merits of the GlobeLand30 in 2010 and the USGS TreeCover2010, the benefits of auxiliary geographic information, and the advantages of the stacking learning method in classification. Additionally, we explored the influence of different forest definitions, based on the tree cover percentage value in the USGS TreeCover2010, on the accuracy of forest cover. Results suggested that the accuracy of our model can be improved significantly by including auxiliary geographic features and feeding the optimal size of training samples. Thereafter, using our model, forest cover maps were derived at different tree cover threshold values in the USGS TreeCover2010. Importantly, the forest cover map at the tree cover threshold value of 40% was determined as the most accurate one with the kappa value of 0.89, whose spatial extent constitutes about 2.4% of the entire territory. This estimated forest cover percentage suggests a low estimation of forest resources based on rigorous definition, which can be valuable for reviewing and amending the current national forest policies.

Intelligent Control of Bulk Tobacco Curing Schedule Using LS-SVM- and ANFIS-Based Multi-Sensor Data Fusion Approaches.

The bulk tobacco flue-curing process is followed by a bulk tobacco curing schedule, which is typically pre-set at the beginning and might be adjusted by the curer to accommodate the need for tobacco leaves during curing. In this study, the controlled parameters of a bulk tobacco curing schedule were presented, which is significant for the systematic modelling of an intelligent tobacco flue-curing process. To fully imitate the curer’s control of the bulk tobacco curing schedule, three types of sensors were applied, namely, a gas sensor, image sensor, and moisture sensor. Feature extraction methods were given forward to extract the odor, image, and moisture features of the tobacco leaves individually. Three multi-sensor data fusion schemes were applied, where a least squares support vector machines (LS-SVM) regression model and adaptive neuro-fuzzy inference system (ANFIS) decision model were used. Four experiments were conducted from July to September 2014, with a total of 603 measurement points, ensuring the results’ robustness and validness. The results demonstrate that a hybrid fusion scheme achieves a superior prediction performance with the coefficients of determination of the controlled parameters, reaching 0.9991, 0.9589, and 0.9479, respectively. The high prediction accuracy made the proposed hybrid fusion scheme a feasible, reliable, and effective method to intelligently control over the tobacco curing schedule.

Distributed Joint Probabilistic Data Association Filter With Hybrid Fusion Strategy

This paper investigates the problem of distributed multitarget tracking (MTT) over a large-scale sensor network, consisting of low-cost sensors. Each local sensor runs a joint probabilistic data association filter to obtain local estimates and communicates with its neighbors for information fusion. The conventional fusion strategies, i.e., consensus on measurement (CM) and consensus on information (CI), are extended to MTT scenarios. This means that data association uncertainty and sensor fusion problems are solved simultaneously. Motivated by the complementary characteristics of these two different fusion approaches, a novel distributed MTT algorithm using a hybrid fusion strategy, e.g., a mix of CM and CI, is proposed. A distributed counting algorithm is incorporated into the tracker to provide the knowledge of the total number of sensor nodes. The new algorithm developed shows advantages in preserving boundedness of local estimates, guaranteeing global convergence to the optimal centralized version and being implemented without requiring no global information, compared with other fusion approaches. Simulations clearly demonstrate the characteristics and tracking performance of the proposed algorithm.

An improved multimodal medical image fusion scheme based on hybrid combination of nonsubsampled contourlet transform and stationary wavelet transform

AbstractThis research proposes an improved hybrid fusion scheme for non‐subsampled contourlet transform (NSCT) and stationary wavelet transform (SWT). Initially, the source images are decomposed into different sub‐bands using NSCT. The locally weighted sum of square of the coefficients based fusion rule with consistency verification is used to fuse the detailed coefficients of NSCT. The SWT is employed to decompose approximation coefficients of NSCT into different sub‐bands. The entropy of square of the coefficients and weighted sum‐modified Laplacian is employed as the fusion rules with SWT. The final output is obtained using inverse NSCT. The proposed research is compared with existing fusion schemes visually and quantitatively. From the visual analysis, it is observed that the proposed scheme retained important complementary information of source images in a better way. From the quantitative comparison, it is seen that this scheme gave improved edge information, clarity, contrast, texture, and brightness in the fused image.

Score-based Fusion Schemes for Plant Identification from Multi-organ Images

This paper describes some fusion techniques for achieving high accuracy species identification from images of different plant organs. Given a series of different image organs such as branch, entire, flower, or leaf, we firstly extract confidence scores for each single organ using a deep convolutional neural network. Then, various late fusion approaches including conventional transformation-based approaches (sum rule, max rule, product rule), a classification-based approach (support vector machine), and our proposed hybrid fusion model are deployed to determine the identity of the plant of interest. For single organ identification, two schemes are proposed. The first scheme uses one Convolutional neural network (CNN) for each organ while the second one trains one CNN for all organs. Two famous CNNs (AlexNet and Resnet) are chosen in this paper. We evaluate the performances of the proposed method in a large number of images of 50 species which are collected from two primary resources: PlantCLEF 2015 dataset and Internet resources. The experiment exhibits the dominant results of the fusion techniques compared with those of individual organs. At rank-1, the highest species identification accuracy of a single organ is 75.6% for flower images, whereas by applying fusion technique for leaf and flower, the accuracy reaches to 92.6%. We also compare the fusion strategies with the multi-column deep convolutional neural networks (MCDCNN) [1]. The proposed hybrid fusion scheme outperforms MCDCNN in all combinations. It obtains from + 3.0% to + 13.8% improvement in rank-1 over MCDCNN method. The evaluation datasets as well as the source codes are publicly available.
 Keywords: Plant identification, Convolutional neural network, Deep learning, Fusion.

A novel hybrid score level and decision level fusion scheme for cancelable multi-biometric verification

In spite of the benefits of biometric-based authentication systems, there are few concerns raised because of the sensitivity of biometric data to outliers, low performance caused due to intra-class variations and privacy invasion caused by information leakage. To address these issues, we propose a hybrid fusion framework where only the protected modalities are combined to fulfill the requirement of secrecy and performance improvement. This paper presents a method to integrate cancelable modalities utilizing mean-closure weighting (MCW) score level and Dempster-Shafer (DS) theory based decision level fusion for iris and fingerprint to mitigate the limitations in the individual score or decision fusion mechanisms. The proposed hybrid fusion scheme incorporates the similarity scores from different matchers corresponding to each protected modality. The individual scores obtained from different matchers for each modality are combined using MCW score fusion method. The MCW technique achieves the optimal weight for each matcher involved in the score computation. Further, DS theory is applied to the induced scores to output the final decision. The rigorous experimental evaluations on three virtual databases indicate that the proposed hybrid fusion framework outperforms over the component level or individual fusion methods (score level and decision level fusion). As a result, we achieve (48%,66%), (72%,86%) and (49%,38%) of performance improvement over unimodal cancelable iris and unimodal cancelable fingerprint verification systems for Virtual_A, Virtual_B and Virtual_C databases, respectively. Also, the proposed method is robust enough to the variability of scores and outliers satisfying the requirement of secure authentication.

DPD-DFF: A dual phase distributed scheme with double fingerprint fusion for fast and accurate identification in large databases

Nowadays, many companies and institutions need fast and reliable identification systems that are able to deal with very large databases. Fingerprints are among the most used biometric traits for identification. In the current literature there are fingerprint matching algorithms that are focused on efficiency, whilst others are based on accuracy. In this paper we propose a flexible dual phase identification method, called DPD-DFF, that combines two fingers and two matchers within a hybrid fusion scheme to obtain both fast and accurate results. Different alternatives are designed to find a trade-off between runtime and accuracy that can be further tuned with a single parameter. The experiments show that DPD-DFF obtains very competitive results in comparison with the state-of-the-art score fusion techniques, especially when dealing with large databases or impostor fingerprints.

Kernel regression in mixed feature spaces for spatio-temporal saliency detection

Spatio-temporal saliency detection has attracted lots of research interests due to its competitive performance on wide multimedia applications. For spatio-temporal saliency detection, existing bottom-up algorithms often over-simplify the fusion strategy, which results in the inferior performance than the human vision system. In this paper, a novel bottom-up spatio-temporal saliency model is proposed to improve the accuracy of attentional region estimation in videos through fully exploiting the merit of fusion. In order to represent the space constructed by several types of features such as location, appearance and temporal cues extracted from video, kernel regression in mixed feature spaces (KR-MFS) including three approximation entity-models is proposed. Using KR-MFS, a hybrid fusion strategy which considers the combination of spatial and temporal saliency of each individual unit and incorporates the impacts from the neighboring units is presented and embedded into the spatio-temporal saliency model. The proposed model has been evaluated on the publicly available dataset. Experimental results show that the proposed spatio-temporal saliency model can achieve better performance than the state-of-the-art approaches.