Feature-level Fusion Research Articles

Centrifugal Pump Cavitation Fault Diagnosis Based on Feature-Level Multi-Source Information Fusion

In nuclear power systems, centrifugal pumps often need to operate under extreme conditions. However, accurately determining the cavitation status of centrifugal pumps under such extreme conditions is challenging. To improve the recognition accuracy of the three statuses of non-cavitation, incipient cavitation, and severe cavitation while improving the anti-interference capability of the monitoring system, this study extracted cavitation features from centrifugal pumps’ motor current and vibration signals under three different operational conditions. It fused the features using feature-level multi-source information fusion (MSIF) based on the backpropagation neural network (BPNN) or support vector machine (SVM) to construct a cavitation status recognition model and analyzed the results to compare with those of recognition without information fusion. The results show that, compared with one signal source, MSIF can significantly improve the recognition accuracy of cavitation statuses. Combined current and pump casing axial monitoring based on the BPNN is the optimal scheme, with an overall recognition accuracy of 97.3% for all operational conditions, compared to 73.9% for the single current signal and 89.3% for the single casing axial vibration signal. These research results can guide the monitoring of cavitation statuses in practical engineering, as well as timely intervention at incipient cavitations to reduce structural damage to centrifugal pumps and prolong service life.

A Multi-group similarity-decoding-based method for deep model ensembling applied in the microcalcification classification on digital mammograms

Classifying mammograms scored at Breast Imaging-Reporting and Data System (BI-RADS ) 4 is crucial for the early detection of breast cancer. But it is particularly challenging because microcalcifications (MCs) vary greatly in shape and distribution. Traditional machine learning and deep models are either time-consuming or poorly generalized. Recently, deep ensemble learning, which can improve generalization, has been developed rapidly. However, most ensembling strategies are based on weighted average and often sensitive to biased learners. To achieve data-adaption, they employ complex techniques to implement dynamic weight calculation. In this paper, we proposed a novel multi-group similarity-decoding-based method (MG-SDM) which enables a more robust and easier ensembling of different deep models. Firstly, the multi-group (MG) strategy is designed to transform model ensembling into a process of feature-level fusion, which could alleviate the impact of a single model on overall performance. Specifically, individual features of different models are grouped based on the position, and feature fusion is performed within each group. Secondly, the similarity-decoding-based method (SDM) is developed to fuse features. All individual features are vectorized through linear transformation. Then their similarities with a learnable latent embedding, which is introduced to denote most task-related features, are calculated using inner product. Finally, the similarity values among features within same group are normalized to be weights for a subsequent weighted summation operation. Extensive experiments on the clinical dataset demonstrate the superiority of the proposed over other state-of-the-art methods.

A Bayesian CNN-based fusion framework of sensor fault diagnosis

Sensors equipped on the high-speed train provide large amounts of data which contributes to its state monitoring. However, it is challenging to distinguish whether the fault originates from the mechanical component or the sensors themselves. The main difficulties lie in the biased amount of normal and fault data as well as the deficiency of multi-source data’s inherent correlation. In this paper, we propose a Bayesian convolutional neural networks (CNN)-based fusion framework to enhance the ability to identify sensor errors. The framework utilizes wavelet time–frequency maps to extract abnormal features, employs a Bayesian CNN to obtain spatial features from a single sensor, integrates multi-source features via bidirectional long short-term memory network and enhances the acquired spatial and temporal features using an attention mechanism. The enhanced information finally generated leads to precise identification of the sensor faults. The proposed feature-level fusion framework and the associated attention mechanism facilitate discovering the inherent correlation and filtering of irrelevant information. Results indicate that our proposed method achieves 95.4% in terms of accuracy, which outperforms methods relying on feature extraction with single-source sensors by 7.8%.

Combined CNN LSTM with attention for speech emotion recognition based on feature-level fusion

Combined CNN LSTM with attention for speech emotion recognition based on feature-level fusion

Fusion recognition of palmprint and palm vein based on modal correlation.

Biometric authentication prevents losses from identity misuse in the artificial intelligence (AI) era. The fusion method integrates palmprint and palm vein features, leveraging their stability and security and enhances counterfeiting prevention and overall system efficiency through multimodal correlations. However, most of the existing multi-modal palmprint and palm vein feature extraction methods extract only feature information independently from different modalities, ignoring the importance of the correlation between different modal samples in the class to the improvement of recognition performance. In this study, we addressed the aforementioned issues by proposing a feature-level joint learning fusion approach for palmprint and palm vein recognition based on modal correlations. The method employs a sparse unsupervised projection algorithm with a "purification matrix" constraint to enhance consistency in intra-modal features. This minimizes data reconstruction errors, eliminating noise and extracting compact, and discriminative representations. Subsequently, the partial least squares algorithm extracts high grayscale variance and category correlation subspaces from each modality. A weighted sum is then utilized to dynamically optimize the contribution of each modality for effective classification recognition. Experimental evaluations conducted for five multimodal databases, composed of six unimodal databases including the Chinese Academy of Sciences multispectral palmprint and palm vein databases, yielded equal error rates (EER) of 0.0173%, 0.0192%, 0.0059%, 0.0010%, and 0.0008%. Compared to some classical methods for palmprint and palm vein fusion recognition, the algorithm significantly improves recognition performance. The algorithm is suitable for identity recognition in scenarios with high security requirements and holds practical value.

A Unified Multi-Modality Fusion Framework for Deep Spatio-Temporal-Spectral Feature Learning in Resting-State fMRI Denoising.

Resting-state functional magnetic resonance imaging (rs-fMRI) is a commonly used functional neuroimaging technique to investigate the functional brain networks. However, rs-fMRI data are often contaminated with noise and artifacts that adversely affect the results of rs-fMRI studies. Several machine/deep learning methods have achieved impressive performance to automatically regress the noise-related components decomposed from rs-fMRI data, which are expressed as the pairs of a spatial map and its associated time series. However, most of the previous methods individually analyze each modality of the noise-related components and simply aggregate the decision-level information (or knowledge) extracted from each modality to make a final decision. Moreover, these approaches consider only the limited modalities making it difficult to explore class-discriminative spectral information of noise-related components. To overcome these limitations, we propose a unified deep attentive spatio-spectral-temporal feature fusion framework. We first adopt a learnable wavelet transform module at the input-level of the framework to elaborately explore the spectral information in subsequent processes. We then construct a feature-level multi-modality fusion module to efficiently exchange the information from multi-modality inputs in the feature space. Finally, we design confidence-based voting strategies for decision-level fusion at the end of the framework to make a robust final decision. In our experiments, the proposed method achieved remarkable performance for noise-related component detection on various rs-fMRI datasets.

A Fast Iris Liveness Detection for Embedded Systems using Textural Feature Level Fusion Algorithm

Iris recognition is a widely used biometric authentication technique due to its high accuracy and uniqueness. However, iris recognition systems are susceptible to attacks using fake or synthetic iris images, causing a serious security threat. To address this issue, this paper presents a fast iris liveness detection method specifically designed for embedded systems. The proposed method utilizes a textural feature level fusion algorithm using Local Binary Pattern (LBP) and Gray-Level Co-Occurrence Matrix (GLCM) to distinguish between live and printed iris images. LBP captures texture information, while GLCM characterizes the statistical properties of the iris images. By combining these complementary features, the proposed method enhances the discrimination capability and robustness against presentation attacks. Furthermore, to enable real-time and efficient implementation, the proposed liveness detection is optimized and implemented for embedded systems. Experimental results on benchmark datasets demonstrate the effectiveness of the proposed method in accurately detecting iris liveness. The proposed fast iris liveness detection is implemented and optimized on C++ which can be complied and deployed in various embedded devices for iris recognition systems on real-world applications, such as access control, biometric authentication, and surveillance systems.

Land Suitability Assessment Based on Feature-Level Fusion of Sentinel-1 and Sentinel-2 Imagery: A Case Study of the Honam Region of Iran

Land Suitability Assessment Based on Feature-Level Fusion of Sentinel-1 and Sentinel-2 Imagery: A Case Study of the Honam Region of Iran

Hybridization of Deep Sequential Network for Emotion Recognition Using Unconstraint Video Analysis

The reliable way to discern human emotions in various circumstances has been proven to be through facial expressions. Facial expression recognition (FER) has emerged as a research topic to identify various essential emotions in the present exponential rise in research for emotion detection. Happiness is one of these basic emotions everyone may experience, and facial expressions are better at detecting it than other emotion-measuring methods. Most techniques have been designed to recognize various emotions to achieve the highest level of general precision. Maximizing the recognition accuracy for a particular emotion is challenging for researchers. Some techniques exist to identify a single happy mood recorded in unrestricted video. Still, they are all limited by the processing of extreme head posture fluctuations that they need to consider, and their accuracy still needs to be improved. This research proposes a novel hybrid facial emotion recognition using unconstraint video to improve accuracy. Here, a Deep Belief Network (DBN) with long short-term memory (LSTM) is employed to extract dynamic data from the video frames. The experiments conducted uses decision-level and feature-level fusion techniques are applied unconstrained video dataset. The outcomes show that the proposed hybrid approach may be more precise than some existing facial expression models.

Global Clue-Guided Cross-Memory Quaternion Transformer Network for Multisource Remote Sensing Data Classification.

Multisource remote sensing data classification is a challenging research topic, and how to address the inherent heterogeneity between multimodal data while exploring their complementarity is crucial. Existing deep learning models usually directly adopt feature-level fusion designs, most of which, however, fail to overcome the impact of heterogeneity, limiting their performance. As such, a multimodal joint classification framework, called global clue-guided cross-memory quaternion transformer network (GCCQTNet), is proposed for multisource data i.e., hyperspectral image (HSI) and synthetic aperture radar (SAR)/light detection and ranging (LiDAR) classification. First, a three-branch structure is built to extract the local and global features, where an independent squeeze-expansion-like fusion (ISEF) structure is designed to update the local and global representations by considering the global information as an agent, suppressing the negative impact of multimodal heterogeneity layer by layer. A cross-memory quaternion transformer (CMQT) structure is further constructed to model the complex inner relationships between the intramodality and intermodality features to capture more discriminative fusion features that fully characterize multimodal complementarity. Finally, a cross-modality comparative learning (CMCL) structure is developed to impose the consistency constraint on global information learning, which, in conjunction with a classification head, is used to guide the end-to-end training of GCCQTNet. Extensive experiments on three public multisource remote sensing datasets illustrate the superiority of our GCCQTNet with regards to other state-of-the-art methods.

A fault diagnosis method based on feature-level fusion of multi-sensor information for rotating machinery

Traditionally, most fault diagnosis work on rotating machinery is carried out on single sensor datasets. However, the single feature source may suffer from missing or inaccurate features, which is especially sluggish for fault diagnosis tasks under noise interference. Feature-level fusion of multi-sensor information can obtain more comprehensive and abundant feature information, while improving the feature discrimination. Therefore, through feature-level fusion of multi-sensor information, a parallel multi-scale attentional convolutional neural network (PMSACNN) is proposed in this paper to achieve rotating machinery fault diagnosis. A dilated wide convolutional layer is designed to extract the short-time features of signals with noise by performing sparse sampling on them. The multi-scale structure is constructed to capture the diversity feature information of signals, and the feature-level stitching of multi-sensor information is realized by the parallel input mechanism. Feature fusion is achieved by adaptively correcting the importance of different channel features by using channel attention. The global averaging pooling operation is introduced to reduce the number of parameters and improve the efficiency of the model operation. The effectiveness of PMSACNN is verified by using the bearing dataset acquired from the mechanical comprehensive diagnosis simulation platform. The experimental results indicate that the proposed method outperforms the existing methods of this field in terms of fault diagnosis accuracy and noise immunity, which can improve the reliability and safety of rotating machinery.

Multi-biometric fusion for enhanced human authentication in information security

In the digital age, information security has taken on greater importance, necessitating the development of strong and trustworthy human authentication techniques. Traditional single-factor authentication schemes, such passwords or PINs, are prone to a number of security flaws. Multi-biometric fusion approaches have become a possible remedy to these restrictions. In comparison to other fusion techniques, feature level fusion is a very popular technique. Features are taken from every biometric attribute in this fusion. Then, the retrieved features are concatenated to create a high dimension final feature vector. In this study, we present a novel method for feature-level fusion employing optimal feature level fusion. The essential features are chosen and identified using a Binary chimp optimized adaptive kernel support vector machine (BCO-AKSVM). According to f1-score (99 %), recall (98 %), accuracy (97 %), precision (88 %), and time computation (1000 ms), the experimental results of the suggested method are analyzed. When compared to current methods, it can show that the proposed BCO-AKSVM achieves the highest performance of information security in human biometric authentication. Multi-biometric fusion is anticipated to play a significant role as technology develops in maintaining secure access control and safeguarding sensitive information in numerous sectors.

MASKED AND UNMASKED FACE RECOGNITION ON UNCONSTRAINED FACIAL IMAGES USING HAND-CRAFTED METHODS

In this study, the face recognition task is applied on masked and unmasked faces using hand-crafted methods. Due to COVID-19 and masks, facial identification from unconstrained images became a hot topic. To avoid COVID-19, most people use masks outside. In many cases, typical facial recognition technology is useless. The majority of contemporary advanced face recognition methods are based on deep learning, which primarily relies on a huge number of training examples, however, masked face recognition may be investigated using hand-crafted approaches at a lower computing cost than using deep learning systems. A low-cost system is intended to be constructed for recognizing masked faces and compares its performance to that of face recognition systems that do not use masks. The proposed method fuses hand-crafted methods using feature-level fusion strategy. This study compares the performance of masked and unmasked face recognition systems. The experiments are undertaken on two publicly accessible datasets for masked face recognition: Masked Labeled Faces in the Wild (MLFW) and Cross-Age Labeled Faces in the Wild (CALFW). The best accuracy is achieved as 94.8% on MLFW dataset. The rest of the results on different train and test sets from CALFW and MLFW datasets are encouraging compared to the state-of-the-art models.

Multisource Information Fusion for Autoformer: Soft Sensor Modeling of FeO Content in Iron Ore Sintering Process

As a key thermal-state indicator of the iron ore sintering process, the content of ferrous oxide (FeO) in the finished sinter is directly related to product quality. Based on the massive data of sintering process, the data-driven soft sensor model provides a good choice for real-time FeO content detection. However, the complex characteristics of the data, including dynamics, nonlinearity, and multi-source heterogeneity, are still the main obstacles to improving the modeling accuracy. To solve this problem, a multi-source information fusion Autoformer (MIF-Autoformer) model is introduced in this paper. Firstly, feature-level information fusion and data-level information fusion are implemented based on the MIF strategy. Then, the comprehensive information of the sintering process is fed to the downstream Autoformer model in a serial manner, which not only improves the information capacity, but also provides additional prior information about the FeO content grade. This is helpful for Autoformer to capture the complex temporal distributions in the sintering process. Finally, the proposed model is applied to a real sintering plant. Experimental results show that the hybrid image features provided by the MIF strategy have a general optimization effect on different competitive models, and MIF-Autoformer exhibits the lowest prediction error on the test set.

Stock market prediction-COVID-19 scenario with lexicon-based approach

Stock market forecasting remains a difficult problem in the economics industry due to its incredible stochastic nature. The creation of such an expert system aids investors in making investment decisions about a certain company. Due to the complexity of the stock market, using a single data source is insufficient to accurately reflect all of the variables that influence stock fluctuations. However, predicting stock market movement is a challenging undertaking that requires extensive data analysis, particularly from a big data perspective. In order to address these problems and produce a feasible solution, appropriate statistical models and artificially intelligent algorithms are needed. This paper aims to propose a novel stock market prediction by the following four stages; they are, preprocessing, feature extraction, improved feature level fusion and prediction. The input data is first put through a preparation step in which stock, news, and Twitter data (related to the COVID-19 epidemic) are processed. Under the big data perspective, the input data is taken into account. These pre-processed data are then put through the feature extraction, The improved aspect-based lexicon generation, PMI, and n-gram-based features in this case are derived from the news and Twitter data, while technical indicator-based features are derived from the stock data. The improved feature-level fusion phase is then applied to the extracted features. The ensemble classifiers, which include DBN, CNN, and DRN, were proposed during the prediction phase. Additionally, a SI-MRFO model is suggested to enhance the efficiency of the prediction model by adjusting the best classifier weights. Finally, SI-MRFO model’s effectiveness compared to the existing models with regard to MAE, MAPE, MSE and MSLE. The SI-MRFO accomplished the minimal MAE rate for the 90th learning percentage is approximately 0.015 while other models acquire maximum ratings.

YOLO-MS: Multispectral Object Detection via Feature Interaction and Self-Attention Guided Fusion

Object detection is essential for an autonomous driving sensing system. Since the light condition is changed in unconstrained scenarios, the detection accuracy based on visible images can be greatly degraded. Although the detection accuracy can be improved by fusing visible and infrared images, existing multispectral object detection (MOD) algorithms suffer from inadequate inter-modal interaction and a lack of global dependence in the fusion approach. Thus, we propose a multispectral object detection framework called YOLO-MS by designing a Feature Interaction and Self-Attention Fusion Network (FISAFN) as the backbone network. Within the FISAFN, correlations between two modalities are extracted by the Feature Interaction Module (FIM) for reconstructing the information components of each modality and enhancing capability of information exchange. To filter redundant features and enhance complementary features, long-range information dependence between two modalities are established by using a Self-Attention Feature Fusion Module (SAFFM). Thus, a better information richness of the fused features can be achieved. Experimental results on the FLIR-aligned dataset and the MFD dataset demonstrate that the proposed YOLO-MS performs favorably against state-of-the-art approaches, including feature-level fusion and pixel-level fusion. And further, the proposed YOLO-MS possesses good detection performance under diverse scene conditions.

MMAF-Net: Multi-view multi-stage adaptive fusion for multi-sensor 3D object detection

In this paper, we propose a 3D object detection method called MMAF-Net that is based on the multi-view and multi-stage adaptive fusion of RGB images and LiDAR point cloud data. This is an end-to-end architecture, which combines the characteristics of RGB images, the front view of point clouds based on reflection intensity, and the bird’s eye view of point clouds. It also adopts a multi-stage fusion approach of “data-level fusion + feature-level fusion” to fully exploit the strength of multimodal information. Our proposed method addresses key challenges found in current 3D object detection methods for autonomous driving, including insufficient feature extraction from multimodal data, rudimentary fusion techniques, and sensitivity to distance and occlusion. To ensure the comprehensive integration of multimodal information, we present a series of targeted fusion methods. Firstly, we propose a novel input form that encodes dense point cloud reflectivity information into the image to enhance its representational power. Secondly, we design the Region Attention Adaptive Fusion module utilizing an attention mechanism to guide the network in adaptively adjusting the importance of different features. Finally, we extend the 2D DIOU (Distance Intersection over Union) loss function to 3D and develop a joint regression loss based on 3D_DIOU and SmoothL1 to optimize the similarity between detected and ground truth boxes. The experimental results on the KITTI dataset demonstrate that MMAF-Net effectively addresses the challenges posed by highly obscured or crowded scenes while maintaining real-time performance and improving the detection accuracy of smaller and more difficult objects that are occluded at far distances.

Data and models for stance and premise detection in COVID-19 tweets: Insights from the Social Media Mining for Health (SMM4H) 2022 shared task

The COVID-19 pandemic has sparked numerous discussions on social media platforms, with users sharing their views on topics such as mask-wearing and vaccination. To facilitate the evaluation of neural models for stance detection and premise classification, we organized the Social Media Mining for Health (SMM4H) 2022 Shared Task 2. This competition utilized manually annotated posts on three COVID-19-related topics: school closures, stay-at-home orders, and wearing masks. In this paper, we extend the previous work and present newly collected data on vaccination from Twitter to assess the performance of models on a different topic. To enhance the accuracy and effectiveness of our evaluation, we employed various strategies to aggregate tweet texts with claims, including models with feature-level (early) fusion and dual-view architectures from the SMM4H 2022 Task 2 leaderboard. Our primary objective was to create a valuable dataset and perform an extensive experimental evaluation to support future research in argument mining in the health domain.

Multi-view domain-adaptive representation learning for EEG-based emotion recognition

Current research suggests that there exist certain limitations in EEG emotion recognition, including redundant and meaningless time-frames and channels, as well as inter- and intra-individual differences in EEG signals from different subjects. To deal with these limitations, a Cross-attention-based Dilated Causal Convolutional Neural Network with Domain Discriminator (CADD-DCCNN) for multi-view EEG-based emotion recognition is proposed to minimize individual differences and automatically learn more discriminative emotion-related features. First, differential entropy (DE) features are obtained from the raw EEG signals using short-time Fourier transform (STFT). Second, each channel of the DE features is regarded as a view, and the attention mechanisms are utilized at different views to aggregate the discriminative affective information at the level of the time-frame of EEG. Then, a dilated causal convolutional neural network is employed to distill nonlinear relationships among different time frames. Next, a feature-level fusion is used to fuse features from multiple channels, aiming to explore the potential complementary information among different views and enhance the representational ability of the feature. Finally, to minimize individual differences, a domain discriminator is employed to generate domain-invariant features, which projects data from both the different domains into the same data representation space. We evaluated our proposed method on two public datasets, SEED and DEAP. The experimental results illustrate that our CADD-DCCNN method outperforms the SOTA methods.

Three-dimensional hybrid fusion networks for current-based bearing fault diagnosis

Abstract Intelligent fault diagnosis (IFD) techniques commonly use vibration-based measurements to perform health monitoring of critical rotating components in industrial systems. However, these vibration-based approaches may be limited in cost-sensitive applications, because the installation of vibration sensors is inconvenient and vibration sensors are expensive. Considering the difficulties of IFD using only current-related information from the motor current signal (MCS), this paper proposes a three-dimensional hybrid-fusion neural network (3D-HFN) that can automatically perform both data- and feature-level fusion of multi-phase current signals for MCS-based IFD of the rolling bearing. The 3D-HFN consists of the multivariate variational mode decomposition (MVMD) and an improved three-dimensional convolution neural network (3D-CNN). Firstly, MVMD is proposed to process multi-phase current signals, which adaptively acquire several intrinsic mode functions with mode-alignment properties. Subsequently, signal-to-image conversion and 3D stacking methods are used to construct 3D-like data in the current-phase dimension, which can fully preserve the interaction relationship between different phases using data-level fusion. Finally, an improved 3D-CNN with multiscale feature fusion and the smooth maximum unit is proposed to learn the 3D-like data and identify different health conditions for the rolling bearing. An open-source dataset with composite bearing faults is used to validate the merits of the proposed method. Experimental results show that the proposed approach has achieved more reliable diagnosis performance than other hand-crafted or 2D/3D-CNN-based algorithms in MCS-based IFD of the rolling bearing.