Sequential Data Processing Research Articles

Deep neural network-based approach for processing sequential data

Increasing volumes of offline/online data has thrown diverse challenges. Availability of the data has made the researchers to urge for new paradigms to analyze data to gain useful insights from it. Such analysis can be helpful for predictions in various domains such as medical informatics, cyber security, fraud detection etc. Further, problem of overloaded information is that it leads to over-choice, especially for users in web applications. Machine learning algorithms construct models that can learn from the data without being explicitly programmed, to perform predictions/decisions. However, the limitations of machine learning have paved a way to deep learning. Unlike the task specific algorithms of machine learning, deep learning learns feature representations from data itself. These traits enabled deep learning to achieve futuristic results for exceptionally complex tasks of various domains. Particularly in health care, natural language processing, information retrieval, computer vision etc. Area of research in this work focused on processing sequential data in the domains of ambient air quality management and recommender systems. Air quality data analysis is one of the domains where deep learning models have outperformed the traditional machine learning models. Hence, this research aims to build a model to predict ambient air quality using Amplified Recurrent Neural Networks for a particular geographical area. Recommender Systems (RS) have become a part of all the web applications these days. The main objective of RS is to asist users by avoiding information overload through generating personalized suggestions. Deep learning-based RS models have gained more prominence these days. Consequently, this research aims to develop deep learning-based recommenders/personal assistance/advisory systems to exploit the information systems and to capture its user’s preferences. The primary goals of the objectives were successfully accomplished with promising results.

Data driven governing equations approximations using attention based multistep neural networks

During the past decade, deep learning has represented the biggest research trend in the field of machine learning, which provides new powerful tools to interrogate high dimensional time series data in a way that has not been possible before. With the recent success in natural language processing, one would expect widespread adaptation to problems like time series forecasting and classification. After all, both involve processing sequential data. However, to this point, research on their adaptation to time series problems has remained limited. Recently, a multi-step time-stepping scheme without the need of direct access to temporal gradients has been proposed, which can accurately identify nonlinear dynamical systems from time series data. In this paper, we combined an attention mechanism with a deep model in a multi-step time-stepping scheme to perform nonlinear system identification and forecasting tasks. To our knowledge, this is the first paper to use attention based models to deal with nonlinear system identification and forecasting problems. The attention weights on rows select those variables that are helpful for forecasting, which can enhance the information across multiple time steps to capture temporal information. The experiment results indicate that the attention based model in a multi-step time-stepping scheme has better identification and prediction performance for nonlinear time series identification and forecasting problems.

Global piston restoration of segmented mirrors with recurrent neural networks

Recurrent neural networks are usually used for processing sequential data. They have been employed in this paper to deal with the sequence of diffraction subimages created by every intersection from a segmented mirror. Every subimage is first processed by a convolutional neural network that extracts a set of features from each of them. It was attained superior prediction accuracy with the recurrent approach than with convolution layers alone. Furthermore, a consistency test was added to detect wrong predictions before computing the global piston values. The final system predicts global piston values with rms = 7.34 nm, high reliability, and capture range of [ − 21λ, 21λ]. Atmospheric seeing, polishing and tip-tilt residual errors were also added in the simulations.

Enhancing Top-Down Proteomics Data Analysis by Combining Deconvolution Results through a Machine Learning Strategy.

Top-down mass spectrometry (MS) is a powerful tool for the identification and comprehensive characterization of proteoforms arising from alternative splicing, sequence variation, and post-translational modifications. However, the complex data set generated from top-down MS experiments requires multiple sequential data processing steps to successfully interpret the data for identifying and characterizing proteoforms. One critical step is the deconvolution of the complex isotopic distribution that arises from naturally occurring isotopes. Multiple algorithms are currently available to deconvolute top-down mass spectra, resulting in different deconvoluted peak lists with varied accuracy compared to true positive annotations. In this study, we have designed a machine learning strategy that can process and combine the peak lists from different deconvolution results. By optimizing clustering results, deconvolution results from THRASH, TopFD, MS-Deconv, and SNAP algorithms were combined into consensus peak lists at various thresholds using either a simple voting ensemble method or a random forest machine learning algorithm. For the random forest algorithm, which had better predictive performance, the consensus peak lists on average could achieve a recall value (true positive rate) of 0.60 and a precision value (positive predictive value) of 0.78. It outperforms the single best algorithm, which achieved a recall value of only 0.47 and a precision value of 0.58. This machine learning strategy enhanced the accuracy and confidence in protein identification during database searches by accelerating the detection of true positive peaks while filtering out false positive peaks. Thus, this method shows promise in enhancing proteoform identification and characterization for high-throughput data analysis in top-down proteomics.

Averaged Bi-LSTM networks for RUL prognostics with non-life-cycle labeled dataset

LSTM network is an effective RNN model to predict the system RUL for its superior performance in sequential data processing. Usually, networks trained by life-cycle labeled dataset would possess similar RUL predicting accuracies, because the network training algorithm could ensure the network optimality for the whole training dataset. However, for networks trained by non-life-cycle labeled samples, the network uncertainty caused by different training conditions could lead to degradation prediction uncertainty for some local points. Further, the RUL predicting results that are computed by these uncertain local points may shows relatively large differences. Therefore, in order to obtain an accurate RUL prediction with networks trained by non-life-cycle labeled samples, our paper proposes a novel network model averaging method to deal with the network uncertainty. What is more, to learn the temporal correlation information of training samples sufficiently, we adopt the Bi-LSTM network to illustrate the application of the proposed network model averaging method. Finally, degradation values of Graphite/LiCoO2 battery are used to verify the effectiveness of the proposed method. The results show that the proposed method could improve the RUL prediction accuracy and reduce the prediction error.

An arsenal of tools based on Kendrick mass defects to process congested electrospray ionization high-resolution mass spectra of polymers with multiple charging.

Electrospray ionization (ESI) favors the multiple charging of high molecular weight polymer samples and allows their high-resolution mass analysis in the low-mass range. It also induces the detection of numerous ion series at different charge states with different adducts complicating the interpretation of the mass spectrum which should be facilitated by an appropriate data processing. An arsenal of tools based on the Kendrick mass defect (KMD) is proposed to process congested ESI high-resolution mass spectra of poly(propylene oxide) (PPO) samples. The combination of regular, charge-dependent, and resolution-enhanced KMD plots in addition to a "remainders" plot and a new three-dimensional plot offers unrivaled capabilities of filtering for any minor series among thousands of points. The sequential data processing is conducted using Kendo, a spreadsheet developed in-house for an advanced KMD analysis. The charge-state distribution is easily evaluated by counting the parallel lines in a regular KMD plot. A charge-dependent resolution-enhanced KMD plot instantly reveals the variation of adducted ions at a given charge state, helping the user to choose the best analytical conditions. Ion series at different charge states from PPO oligomers carrying different end-groups are also efficiently extracted using several combinations of KMD and remainders plots and assigned using a new simulator tool. The innovative combination of existing and new KMD-related plots, selection tools, and simulator all combined in a single spreadsheet dramatically facilitates the processing and interpretation of complex ESI mass spectral data. The presented tools may be extended to any other class of homo-, co- and terpolymers.

Speech Emotion Recognition Using Scalogram Based Deep Structure

Speech Emotion Recognition (SER) is an important part of speech-based Human-Computer Interface (HCI) applications. Previous SER methods rely on the extraction of features and training an appropriate classifier. However, most of those features can be affected by emotionally irrelevant factors such as gender, speaking styles and environment. Here, an SER method has been proposed based on a concatenated Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). The CNN can be used to learn local salient features from speech signals, images, and videos. Moreover, the RNNs have been used in many sequential data processing tasks in order to learn long-term dependencies between the local features. A combination of these two gives us the advantage of the strengths of both networks. In the proposed method, CNN has been applied directly to a scalogram of speech signals. Then, the attention-mechanism-based RNN model was used to learn long-term temporal relationships of the learned features. Experiments on various data such as RAVDESS, SAVEE, and Emo-DB demonstrate the effectiveness of the proposed SER method.

Classification with a disordered dopant-atom network in silicon.

Classification is an important task at which both biological and artificial neural networks excel1,2. In machine learning, nonlinear projection into a high-dimensional feature space can make data linearly separable3,4, simplifying the classification of complex features. Such nonlinear projections are computationally expensive in conventional computers. A promising approach is to exploit physical materials systems that perform this nonlinear projection intrinsically, because of their high computational density5, inherent parallelism and energy efficiency6,7. However, existing approaches either rely on the systems' time dynamics, which requires sequential data processing and therefore hinders parallel computation5,6,8, or employ large materials systems that are difficult to scale up7. Here we use a parallel, nanoscale approach inspired by filters in the brain1 and artificial neural networks2 to perform nonlinear classification and feature extraction. We exploit the nonlinearity of hopping conduction9-11 through an electrically tunable network of boron dopant atoms in silicon, reconfiguring the network through artificial evolution to realize different computational functions. We first solve the canonical two-input binary classification problem, realizing all Boolean logic gates12 up to room temperature, demonstrating nonlinear classification with the nanomaterial system. We then evolve our dopant network to realize feature filters2 that can perform four-input binary classification on the Modified National Institute of Standards and Technology handwritten digit database. Implementation of our material-based filters substantially improves the classification accuracy over that of a linear classifier directly applied to the original data13. Our results establish a paradigm of silicon-based electronics for small-footprint and energy-efficient computation14.

Spatio-temporal SRU with global context-aware attention for 3D human action recognition

3D action recognition has attracted much attention in machine learning fields in recent years, and recurrent neural networks (RNNs) have been widely used for 3D action recognition due to their efficiency in processing sequential data. However, in order to achieve good performance, traditional RNN architectures are usually time-consuming for the training and inference process. To address the problem, a global context-aware attention spatio-temporal SRU (GCA-ST-SRU) method is proposed and applied for 3D action recognition in this paper, through extending the original simple recurrent unit (SRU) algorithm to joint spatio-temporal domain with an attention mechanism. First, deep neural networks were employed to learn the features of skeleton joints at each frame, and then these new high-level feature sequences were classified using the GCA-ST-SRU method which can learn the spatio-temporal dependence between different joints in the same frame and pay more attention to informative joints. Extensive experiments were conducted on the UT-Kinect and SBU-Kinect Interaction datasets to evaluate the effectiveness of the proposed method. Compared with several existing algorithms including SRU, long short-term memory (LSTM), spatio-temporal LSTM (ST-LSTM) and global context-aware attention LSTM (GCA-LSTM), our method has exhibited better performance in classification accuracy and computational efficiency. The experimental results demonstrate the effectiveness and practicability of our algorithm. Compared to the methods with similar performance, our algorithms can reduce training time and improve the inference speed, and thus it achieves a balance between speed and accuracy.

Online Reweighted Least Squares Robust PCA

The letter deals with the problem known as robust principal component analysis (RPCA), that is, the decomposition of a data matrix as the sum of a low-rank matrix component and a sparse matrix component. After expressing the low-rank matrix component in factorized form, we develop a novel online RPCA algorithm that is based entirely on reweighted least squares recursions and is appropriate for sequential data processing. The proposed algorithm is fast, memory optimal and, as corroborated by indicative empirical results on simulated data and a video processing application, competitive to the state-of-the-art in terms of estimation performance.

Real-Time Intended Knee Joint Motion Prediction by Deep-Recurrent Neural Networks

Human-assisting intelligent systems demand certain methods to precisely predict motorized limb joint angles. This paper presents the application of deep-recurrent neural networks (RNNs), which is a type of neural network for processing sequential data, for predicting the knee joint angle in real-time. This model is created based on a combination of electromyographic (EMG) signals, (with electrodes being placed on three leg muscles), and inertial measurements of the upper and lower legs. The data collected from different subjects when they performed different gaits were used to construct the model, which was evaluated in a real-time setting. The proposed RNN model based on fusion information contains a balance between computational complexity and prediction accuracy. Results on a microcontroller show that, within a predicted horizon of 50 ms, the model has a low prediction error of ±2.93 degrees.

Attention mechanism enhanced LSTM with residual architecture and its application for protein-protein interaction residue pairs prediction

BackgroundRecurrent neural network(RNN) is a good way to process sequential data, but the capability of RNN to compute long sequence data is inefficient. As a variant of RNN, long short term memory(LSTM) solved the problem in some extent. Here we improved LSTM for big data application in protein-protein interaction interface residue pairs prediction based on the following two reasons. On the one hand, there are some deficiencies in LSTM, such as shallow layers, gradient explosion or vanishing, etc. With a dramatic data increasing, the imbalance between algorithm innovation and big data processing has been more serious and urgent. On the other hand, protein-protein interaction interface residue pairs prediction is an important problem in biology, but the low prediction accuracy compels us to propose new computational methods.ResultsIn order to surmount aforementioned problems of LSTM, we adopt the residual architecture and add attention mechanism to LSTM. In detail, we redefine the block, and add a connection from front to back in every two layers and attention mechanism to strengthen the capability of mining information. Then we use it to predict protein-protein interaction interface residue pairs, and acquire a quite good accuracy over 72%. What’s more, we compare our method with random experiments, PPiPP, standard LSTM, and some other machine learning methods. Our method shows better performance than the methods mentioned above.ConclusionWe present an attention mechanism enhanced LSTM with residual architecture, and make deeper network without gradient vanishing or explosion to a certain extent. Then we apply it to a significant problem– protein-protein interaction interface residue pairs prediction and obtain a better accuracy than other methods. Our method provides a new approach for protein-protein interaction computation, which will be helpful for related biomedical researches.

Attention augmentation with multi-residual in bidirectional LSTM

Recurrent neural networks (RNNs) have been proven to be efficient in processing sequential data. However, the traditional RNNs have suffered from the gradient diminishing problem until the advent of Long Short-Term Memory (LSTM). However, LSTM is weak in capturing long-time dependency in sequential data due to the inadequacy of memory capacity in LSTM cells. To address this challenge, we propose an Attention-augmentation Bidirectional Multi-residual Recurrent Neural Network (ABMRNN) to overcome the deficiency. We propose an algorithm which integrates both past and future information at every time step with omniscient attention model. The multi-residual mechanism has also been leveraged in the proposed model targeting the pattern of the relationship between current time step and further distant time steps instead of only one previous time step. The results of experiments show that our model outperforms the traditional statistical classifiers and other existing RNN architectures.

Deep Learning Based Prediction of Autism Spectrum Disorder using Codon Encoding of Gene Sequences

The development of computational tools to recognize Autism Spectrum Disorder (ASD) originated by genetic mutations is vital to the development of disease-specific targeted therapies. Identifying genes causing the genetically transmitted ASD is still a challenging task. As genomics data is dependent on domain specific experts for identifying efficient features and extracting hand-crafted attributes involves much time, an alternate effective solution is the need of the hour. The rapid developments in the design of deep architecture models have led to the broad application of these models in a variety of research areas and they have shown considerable success in sequential data processing tasks. The primary goal of this work is to classify the ASD gene sequences by employing a Deep Neural Network based model. This in turn will enable effective genetic diagnoses of this disease and facilitate the targeted genetic testing of individuals. This work utilizes codon encoding and one hot encoding technique to transform the mutated gene sequences which are exploited for self learning the features by deep network. Experiments showed that the performance of the proposed model was better than that of the conventional Multilayer Perceptron with promising accuracy of 77.8%, 80.1% and 81.2% for three different datasets.

A Sequential Data Analysis Approach to Detect Emergent Leaders in Small Groups

This paper addresses the problem of predicting emergent leaders (ELs) in small groups, that is, meetings. This is a long-lasting research problem for social and organizational psychology and a relevant problem that recently gained momentum in social computing. Toward this goal, we propose a novel method, which analyzes the temporal dependencies of the audio-visual data by applying unsupervised deep learning generative models (feature learning). To the best of our knowledge, this is the first attempt that sequential data processing is performed for EL detection. Feature learning results in a single feature vector per a given time interval and all feature vectors representing a participant are aggregated using novel fusion techniques. Finally, the EL detection is performed using the state-of-the-art single and multiple kernel learning algorithms. The proposed method shows (significantly) improved results compared to the state-of-the-art methods and it can be adapted to analyze various small group interactions given that it is a general approach.

Adversarial Dropout for Recurrent Neural Networks

Successful application processing sequential data, such as text and speech, requires an improved generalization performance of recurrent neural networks (RNNs). Dropout techniques for RNNs were introduced to respond to these demands, but we conjecture that the dropout on RNNs could have been improved by adopting the adversarial concept. This paper investigates ways to improve the dropout for RNNs by utilizing intentionally generated dropout masks. Specifically, the guided dropout used in this research is called as adversarial dropout, which adversarially disconnects neurons that are dominantly used to predict correct targets over time. Our analysis showed that our regularizer, which consists of a gap between the original and the reconfigured RNNs, was the upper bound of the gap between the training and the inference phases of the random dropout. We demonstrated that minimizing our regularizer improved the effectiveness of the dropout for RNNs on sequential MNIST tasks, semi-supervised text classification tasks, and language modeling tasks.

Electrode regulating system modeling in electrical smelting furnace using recurrent neural network with attention mechanism

Electrical smelting furnace (ESF) is the primary equipment to produce steer, nonferrous metals and other materials. In smelting process, it is beneficial to keep the smelting current stable within a reasonable range to improve product quality and reduce energy consumption. ESF is characterized by nonlinearity, strong coupling and time variation, which makes it difficult to establish accurate mathematical models. Therefore, we propose a data-driven recurrent neural network (RNN) using gated recurrent unit (GRU) with attention mechanism not only to establish the relationship between the electrode position and the smelting current but also to reveal the internal dynamic variation between three-phase currents for the electrode regulating system. Our proposed model is tested on the actual production data collected from a fused magnesium furnace (one kind of ESF) in Liaoning Province of China. Numerical results show RNN excels at processing sequential data and describing inner changes of the dynamic system; GRU alleviates long-term dependency problems in industrial big data; attention mechanism can identify and put emphasis on the crucial points containing key information in long sequential data. The results indicate that the proposed model is effective and feasible for the modeling of electrode regulating system.

One-pass person re-identification by sketch online discriminant analysis

Person re-identification (re-id) is to match people across disjoint camera views in a multi-camera system, and re-id has been an important technology applied in smart city in recent years. However, the majority of existing person re-id methods assumes all data samples are available in advance for training. However, in a real-world scenario person images detected from multi-camera system are coming sequentially, and thus these methods are not designed for processing sequential data in an online way. While there is a few work on discussing online re-id, most of them require considerable storage of all passed labelled data samples that have been ever observed. In this work, we present an one-pass person re-id model that adapts the re-id model based on each newly observed data and no passed data are required for each update. More specifically, we develop a Sketch online Discriminant Analysis (SoDA) by embedding sketch processing into Fisher discriminant analysis (FDA). SoDA can efficiently keep the main data variations of all passed samples in a low rank matrix when processing sequential data samples, and estimate the approximate within-class variance (i.e. within-class covariance matrix) from the sketch data information. We provide theoretical analysis on the effect of the estimated approximate within-class covariance matrix. In particular, we derive upper and lower bounds on the Fisher discriminant score (i.e. the quotient between between-class variation and within-class variation after feature transformation) in order to investigate how the optimal feature transformation learned by SoDA sequentially approximates the offline FDA that is learned on all observed data. Extensive experimental results have shown the effectiveness of our SoDA and empirically support our theoretical analysis.

Interpreting Recurrent Neural Networks Behaviour via Excitable Network Attractors

Machine learning provides fundamental tools both for scientific research and for the development of technologies with significant impact on society. It provides methods that facilitate the discovery of regularities in data and that give predictions without explicit knowledge of the rules governing a system. However, a price is paid for exploiting such flexibility: machine learning methods are typically black boxes where it is difficult to fully understand what the machine is doing or how it is operating. This poses constraints on the applicability and explainability of such methods. Our research aims to open the black box of recurrent neural networks, an important family of neural networks used for processing sequential data. We propose a novel methodology that provides a mechanistic interpretation of behaviour when solving a computational task. Our methodology uses mathematical constructs called excitable network attractors, which are invariant sets in phase space composed of stable attractors and excitable connections between them. As the behaviour of recurrent neural networks depends both on training and on inputs to the system, we introduce an algorithm to extract network attractors directly from the trajectory of a neural network while solving tasks. Simulations conducted on a controlled benchmark task confirm the relevance of these attractors for interpreting the behaviour of recurrent neural networks, at least for tasks that involve learning a finite number of stable states and transitions between them.

Nonlinear Dynamic Soft Sensor Modeling With Supervised Long Short-Term Memory Network

Soft sensor has been extensively utilized in industrial processes for prediction of key quality variables. To build an accurate virtual sensor model, it is very significant to model the dynamic and nonlinear behaviors of process sequential data properly. Recently, a long short-term memory (LSTM) network has shown great modeling ability on various time series, in which basic LSTM units can handle data nonlinearities and dynamics with a dynamic latent variable structure. However, the hidden variables in the basic LSTM unit mainly focus on describing the dynamics of input variables, which lack representation for the quality data. In this paper, a supervised LSTM (SLSTM) network is proposed to learn quality-relevant hidden dynamics for soft sensor application, which is composed of basic SLSTM unit at each sampling instant. In the basic SLSTM unit, the quality and input variables are simultaneously utilized to learn the dynamic hidden states, which are more relevant and useful for quality prediction. The effectiveness of the proposed SLSTM network is demonstrated on a penicillin fermentation process and an industrial debutanizer column.