Multichannel Data Fusion Research Articles

3-D Facial Expression Recognition via Attention-Based Multichannel Data Fusion Network

Facial expression has long been recognized as containing meaningful nonverbal affective cues for decoding human emotions. Recently, multimodal 2-D + 3-D fusion method has shown significant potential in facial expression recognition (FER) due to its fine-grained face descriptions in various spatial channels. However, current work mainly relies on feature- or even score-level fusion to find emotion cues spread in different channels and may miss key information due to lack of focus. To this end, we propose an attention-based multichannel data fusion network (AMDFN) to better preserve and find such key facial cues. More specifically, we first map a 3-D face scan into multichannel images and then fuse them in a ResNet18 backbone to get layered emotion features. Second, we leverage a layer attention model to explore the dependencies between features of different layers to learn discriminative affective cues for effective emotion recognition. Our comprehensive experiments on two widely used datasets (i.e., Facescape and Bosphorus) have verified the performance of our approach compared to several state-of-the-art rivals.

Intelligent Fault Diagnosis of Multichannel Motor–Rotor System Based on Multimanifold Deep Extreme Learning Machine

Nowadays, the measurement technology of multichannel information fusion provides a solid research foundation for digital and intelligent fault diagnosis of mechatronics equipment. To implement the rapid fusion of multichannel data and intelligent diagnosis, a new fault diagnosis method for multichannel motor–rotor system via multimanifold deep extreme learning machine (MDELM) algorithm is first proposed in this article. Specifically, the designed MDELM algorithm is divided into two main components: 1) unsupervised self-taught feature extraction via the designed extreme learning machine based-modified sparse filtering feature extractor; 2) semisupervised fault classification via the designed MELM classifier with multimanifold constraints to mine the intraclass and interclass discriminant feature information. Experimental and industrial data from motor–rotor system demonstrates the superiority of the proposed method and algorithms. Compared with other fault diagnosis methods, the proposed MDELM algorithm has better learning efficiency, and it is more suitable for intelligent diagnosis of multichannel data fusion.

Separation of multiple local-damage-related components from vibration data using Nonnegative Matrix Factorization and multichannel data fusion

The problem of local damage detection in case of analyzing vibration signal from rotating machines is mostly related to the detection of periodic impulsive components. Depending on various features of the signal, this task can be relatively simple in some cases (e.g. for an impulsive component in the presence of Gaussian noise). However, in our case multiple impulsive components occupying with the overlapping frequency bands are present. For all components, the impulses can be periodic. In this article, authors present a novel methodology based on data fusion from multichannel vibration data from heavy-duty industrial gearbox operating in the driving station of a belt conveyor. The proposed method is based on the factorization of spectrograms using Generalized Hierarchical Alternating Least Squares Nonnegative Matrix Factorization with Beta-Divergence (later referred to as β-HALS NMF). Partial information obtained from the factorization is fused into a single data set for each impulsive component present in the signal. Finally, Griffin-Lim algorithm is used to estimate the complex phase layer of artificial spectrograms allowing to recover the near-perfect time series of each impulsive component extracted from the signal. This method has been tested on four-channel vibration data.

A Multichannel Data Fusion Method Based on Multiple Deep Belief Networks for Intelligent Fault Diagnosis of Main Reducer

Aiming at the problems of poor efficiency of the intelligent fault diagnosis method of the main reducer and the poor effectiveness of multichannel data fusion, this paper proposes a multichannel data fusion method based on deep belief networks and random forest fusion for fault diagnosis. Multiple deep belief networks (MDBNs) are constructed to obtain deep representative features from multiple modalities of multichannel data. Random forest can fuse deep representative features achieved from MDBNs to construct the model of multiple deep belief networks fusion (MDBNF). The proposed method is applied to fault diagnosis of the main reducer and evaluation of the performance. Multiple deep belief network model fusions (MD BN F) are constructed to improve the multichannel data fusion effect. Single sensory data, multichannel data, and two intelligent models based on support vector machine and deep belief networks are used as comparison in the experiments. The results indicate that the classification accuracy of the test set collected by sensor 1 and sensor 2 is 88.35% and 88.73%, respectively. The comparison results show that the method has good convergence. The data fusion of the proposed diagnostic model can effectively improve the correlation between the collected vibration signals and the failure mode, thereby improving the diagnostic performance by nearly 8%, representing improved diagnostic accuracy.

Audio Signal Detection and Enhancement Based on Linear CMOS Array and Multi-Channel Data Fusion

An audio signal detection system based on laser speckle and multi-channel data fusion is presented. A linear CMOS array is used as the detector, which owns a fast line rate and suitable sensing size. The signals from the pixels are selected and fused to enhance the reconstructed signal. The reconstructed audio signals are evaluated with a segmental SNR (SegSNR) algorithm. The experimental results of three categories of audio sources (single voice audio, conversation and music) show that data fusion can improve the SegSNR scores. Especially, direct phase-error based filtering (pbf) fusion gives a nearly 3.0 dB increase and obtains another 1.0 dB increase with the combination of single channel process. The experimental results show that the fusion algorithms are not sensitive to audio types and the performance of multi-channel data fusion is not weakened with the increase of measuring distance. This feature has potential applications in remote sensing. The intelligibility of the fused audio signals is evaluated with normalized subband envelope correlation (NSEC) algorithm and the evaluation results shows that fusion can also enhance the intelligibility of the recovered signal.

Combining Chrominance Features and Fast ICA for Noncontact Imaging Photoplethysmography

Video-based noncontact detection of heart rate has a wide range of applications in the field of medicine and health. However, this method is susceptible to noise interference, making it difficult to effectively extract blood volume pulse (BVP) signals. To overcome this problem, a new method of noncontact heart rate estimation that can suppress noise interference is proposed in this paper. First, the established data acquisition system conducts video collection, and the captured videos are divided into multiple small regions. Subsequently, the initial signals of BVP are extracted in accordance with the chrominance features extracted through multi-channel data fusion. The BVP signals are separated using the FastICA algorithm. The kurtosis value and signal-to-noise ratios of the power spectrum of the separated signals are analyzed to determine the effective separation component. Results show that this method can extract and process pulse signals, effectively suppressing non-periodic interference. The experiment also proves that the method has good consistency with the measurement of pulse oximeter and has good stability and accuracy in the detection of heart rate of the human body.

A Novel Deep Learning Method for Intelligent Fault Diagnosis of Rotating Machinery Based on Improved CNN-SVM and Multichannel Data Fusion.

Intelligent fault diagnosis methods based on deep learning becomes a research hotspot in the fault diagnosis field. Automatically and accurately identifying the incipient micro-fault of rotating machinery, especially for fault orientations and severity degree, is still a major challenge in the field of intelligent fault diagnosis. The traditional fault diagnosis methods rely on the manual feature extraction of engineers with prior knowledge. To effectively identify an incipient fault in rotating machinery, this paper proposes a novel method, namely improved the convolutional neural network-support vector machine (CNN-SVM) method. This method improves the traditional convolutional neural network (CNN) model structure by introducing the global average pooling technology and SVM. Firstly, the temporal and spatial multichannel raw data from multiple sensors is directly input into the improved CNN-Softmax model for the training of the CNN model. Secondly, the improved CNN are used for extracting representative features from the raw fault data. Finally, the extracted sparse representative feature vectors are input into SVM for fault classification. The proposed method is applied to the diagnosis multichannel vibration signal monitoring data of a rolling bearing. The results confirm that the proposed method is more effective than other existing intelligence diagnosis methods including SVM, K-nearest neighbor, back-propagation neural network, deep BP neural network, and traditional CNN.

Recognizing Multidimensional Engagement of E-Learners Based on Multi-Channel Data in E-Learning Environment

Despite recent advances in MOOC, the current e-learning systems have advantages of alleviating barriers by time differences, and geographically spatial separation between teachers and students. However, there has been a 'lack of supervision' problem that e-learner's learning unit state(LUS) can't be supervised automatically. In this paper, we present a fusion framework considering three channel data sources: 1) videos/images from a camera, 2) eye movement information tracked by a low solution eye tracker and 3) mouse movement. Based on these data modalities, we propose a novel approach of multi-channel data fusion to explore the learning unit state recognition. We also propose a method to build a learning state recognition model to avoid manually labeling image data. The experiments were carried on our designed online learning prototype system, and we choose CART, Random Forest and GBDT regression model to predict e-learner's learning state. The results show that multi-channel data fusion model have a better recognition performance in comparison with single channel model. In addition, a best recognition performance can be reached when image, eye movement and mouse movement features are fused.

Color Edge Detection Based on Data Fusion Technology in Presence of Gaussian Noise

A color edge detection method was developed to correctly reproduce distinct, continuous edges based on the multichannel data fusion technique and the gradient direction information contained in the image. Since the proposed techniques for edge detection are very sensitive to noise, subprefiltering or prefiltering algorithms are generally adopted and very critical. A new nonlinear prefilter is used to reduce noise in the R, G, and, B components of the image. The method preserves edges, corners and fine image details, smoothes Gaussian noise and does not require any a priori knowledge.

Automatic registration of ERS and SPOT multisensor images in a data fusion context

This paper deals with an automatic registration method of multidate and multisensor remote sensing images as preprocessing in a multichannel consensual segmentation scheme. This two step method consists in a geometrical correction followed by residual translation compensation, performed by detecting the crosscorrelation maximum between multidate images. This second step is preceded by an edge detection for prefiltered European remote sensing satellite (ERS) and satellite pour l’observation de la terre (SPOT) multisensor images. The registration of these two kinds of images constitutes, indeed, the unavoidable first part of a multichannel data fusion scheme that requires point to point matching.