Dependencies In Sequential Data Research Articles

Transformers for cardiac patient mortality risk prediction from heterogeneous electronic health records

With over 17 million annual deaths, cardiovascular diseases (CVDs) dominate the cause of death statistics. CVDs can deteriorate the quality of life drastically and even cause sudden death, all the while inducing massive healthcare costs. This work studied state-of-the-art deep learning techniques to predict increased risk of death in CVD patients, building on the electronic health records (EHR) of over 23,000 cardiac patients. Taking into account the usefulness of the prediction for chronic disease patients, a prediction period of six months was selected. Two major transformer models that rely on learning bidirectional dependencies in sequential data, BERT and XLNet, were trained and compared. To our knowledge, the presented work is the first to apply XLNet on EHR data to predict mortality. The patient histories were formulated as time series consisting of varying types of clinical events, thus enabling the model to learn increasingly complex temporal dependencies. BERT and XLNet achieved an average area under the receiver operating characteristic curve (AUC) of 75.5% and 76.0%, respectively. XLNet surpassed BERT in recall by 9.8%, suggesting that it captures more positive cases than BERT, which is the main focus of recent research on EHRs and transformers.

Characterization of anomalous diffusion through convolutional transformers

The results of the Anomalous Diffusion Challenge (AnDi Challenge) (Muñoz-Gil G et al 2021 Nat. Commun. 12 6253) have shown that machine learning methods can outperform classical statistical methodology at the characterization of anomalous diffusion in both the inference of the anomalous diffusion exponent α associated with each trajectory (Task 1), and the determination of the underlying diffusive regime which produced such trajectories (Task 2). Furthermore, of the five teams that finished in the top three across both tasks of the AnDi Challenge, three of those teams used recurrent neural networks (RNNs). While RNNs, like the long short-term memory network, are effective at learning long-term dependencies in sequential data, their key disadvantage is that they must be trained sequentially. In order to facilitate training with larger data sets, by training in parallel, we propose a new transformer based neural network architecture for the characterization of anomalous diffusion. Our new architecture, the Convolutional Transformer (ConvTransformer) uses a bi-layered convolutional neural network to extract features from our diffusive trajectories that can be thought of as being words in a sentence. These features are then fed to two transformer encoding blocks that perform either regression (Task 1 1D) or classification (Task 2 1D). To our knowledge, this is the first time transformers have been used for characterizing anomalous diffusion. Moreover, this may be the first time that a transformer encoding block has been used with a convolutional neural network and without the need for a transformer decoding block or positional encoding. Apart from being able to train in parallel, we show that the ConvTransformer is able to outperform the previous state of the art at determining the underlying diffusive regime (Task 2 1D) in short trajectories (length 10–50 steps), which are the most important for experimental researchers.

Comparative Analysis of Hidden Markov Model and Bidirectional Long Short-Term Memory for POS Tagging in Eastern Armenian

This scientific article introduces a novel POS tagging system specifically developed for the Eastern Armenian language. The primary objective of the study is to conduct a comparative analysis of two well-established methods for part-of-speech (POS) tagging: the hidden Markov model (HMM) coupled with the Viterbi algorithm, and an artificial neural network in the form of a recurrent neural network. The study places particular emphasis on the Eastern Armenian language and employs the ArmSpeech-POS Eastern Armenian part-of-speech tagged corpus for conducting comprehensive experiments and evaluations. POS tagging is a fundamental task in natural language processing (NLP) that involves assigning grammatical tags to words in a given text. Accurate POS tagging is crucial for various NLP applications, including machine translation, information retrieval, and sentiment analysis. The Viterbi algorithm is a well-established probabilistic method that utilizes a hidden Markov model (HMM) to determine the most likely sequence of POS tags. On the other hand, RNNs, a type of deep learning model, can capture complex patterns and dependencies in sequential data. Experimental results indicate that both methods achieve reasonable accuracy in POS tagging for Eastern Armenian. However, the RNN outperforms the Viterbi algorithm, exhibiting higher accuracy rates. This can be attributed to the RNN’s ability to capture long-range dependencies and learn intricate linguistic patterns. The article concludes by discussing the implications of the study’s findings and potential areas for further research. It emphasizes the significance of accurate POS tagging for improving NLP applications in Eastern Armenian and suggests exploring advanced neural network architectures and incorporating linguistic features to enhance POS tagging performance.

Partially Recurrent Network With Highway Connections

It is difficult to train deep recurrent neural networks (RNNs) to learn the complex dependencies in sequential data, because of gradient problems and learning conflicts. To address these learning problems, this article proposes a partially recurrent network (PR-Net), which consists of the partially recurrent (PR) layers with highway connections. The proposed PR layer explicitly arranges the neurons into a memory module and an output module, which are implemented by highway networks. The memory module places memory on the information highway, to maintain long short-term memory (LSTM). The output module places external input on the information highway, to approximate the complex transition function from one step to the next. Because of the highway connections in the output module, the PR layer can pass gradient information across layers easily, to allow the training of the PR-Net with multiple stacked PR layers. Furthermore, the explicit separation of the memory module and output module can reduce the learning burden of RNNs that which neurons are responsible for memory and which are for output. With a comparable number of parameters, sublayers in the recurrent layer, and stacked recurrent layers, the proposed PR-Net outperformed the recurrent highway network and LSTM on three sequence learning benchmarks, significantly.

Human activity recognition based on hybrid learning algorithm for wearable sensor data

Human Activity Recognition (HAR), based on sensor devices and the Internet of Things (IoT), attracted many researchers since it has diversified applications in health sectors, smart environments, and entertainment. HAR has emerged as one of the important health monitoring applications and it necessitates the constant usage of smartphones, smartwatches, and wearable devices to capture patients' daily activities. To predict multiple human activities, deep learning (DL)-based methods have been successfully applied to time-series data that are generated by smartphones and wearable sensors. Although DL-based approaches were deployed in activity recognition, they still have encountered a few issues when working with time-series data. Those issues could be managed with the proposed methodology. This work proposed a couple of Hybrid Learning Algorithms (HLA) to build comprehensive classification methods for HAR using wearable sensor data. The aim of this work is to make use of the Convolution Memory Fusion Algorithm(CMFA) and Convolution Gated Fusion Algorithm(CGFA) that model learns both local features and long-term and gated-term dependencies in sequential data. Feature extraction has been enhanced with the deployment of various filter sizes. They are used to capture different local temporal dependencies, and thus the enhancement is implemented. This Amalgam Learning Model has been deployed on the WISDM dataset, and the proposed models have achieved 97.76%, 94.98% for smartwatch and smartphone of CMFA, 96.91%, 84.35% for smartwatch and smartphone of CGFA. Experimental results show that these models demonstrated greater accuracy than other existing deep neural network frameworks.

Comparing two machine learning approaches in predicting lupus hospitalization using longitudinal data

Systemic lupus erythematosus (SLE) is a heterogeneous autoimmune disease characterized by flares ranging from mild to life-threatening. Severe flares and complications can require hospitalizations, which account for most of the direct costs of SLE care. This study investigates two machine learning approaches in predicting SLE hospitalizations using longitudinal data from 925 patients enrolled in a multicenter electronic health record (EHR)-based lupus cohort. Our first Differential approach accounts for the time dependencies in sequential data by introducing additional lagged variables between consecutive time steps. We next evaluate the performance of LSTM, a state-of-the-art deep learning model designed for time series. Our experimental results demonstrate that both methods can effectively predict lupus hospitalizations, but each has its strengths and limitations. Specifically, the Differential approach can be integrated into any non-temporal machine learning algorithms and is preferred for tasks with short observation periods. On the contrary, the LSTM model is desirable for studies utilizing long observation intervals attributing to its capability in capturing long-term dependencies embedded in the longitudinal data. Furthermore, the Differential approach has more options in handling class imbalance in the underlying data and delivers stable performance across different prognostic horizons. LSTM, on the other hand, demands more class-balanced training data and outperforms the Differential approach when there are sufficient positive samples facilitating model training. Capitalizing on our experimental results, we further study the optimal length of patient monitoring periods for different prediction horizons.

Using MLSTM and Multioutput Convolutional LSTM Algorithms for Detecting Anomalous Patterns in Streamed Data of Unmanned Aerial Vehicles

In this article, we present a comparative study of two existing deep learning tools that are used in a novel way to detect anomalies in the streamed data of the unmanned aerial vehicle (UAV). Detecting anomalies is very vital to predict potential faults that are caused by hardware and software faults and may prevent the UAV from hazardous accidents. Therefore, we suggest using multiple long short-term memory and multioutput convolutional long short-term memory (LSTM) to detect anomalies in UAV data. LSTM networks attracted many researchers in several domains, as it is a useful tool for learning dynamic temporal patterns and long-range dependencies in sequential data, which cannot be achieved using traditional neural networks. However, utilizing multiple LSTM networks would result in too much redundancy. The redundancy issue can be solved by incorporating one convolutional LSTM (ConvLSTM) network with multiple outputs. The ConvLSTM is suitable for analyzing multivariate temporal data; due to its convolutional architecture and the advantage of preserving the benefits of the LSTM networks. We evaluated and compared the two approaches using well-known indicators such as the detection rate, the false alarm rate, the precision, and the F.score indicators. The two methods exhibited promising results in predicting different types of faults, for instance (sensor-impulse and sensor-cut). However, the multioutput ConvLSTM was faster in training and testing phases, and its results were superior in predicting (sensor-stuck and sensor-drift) faults.

Attentional networks for music generation

Realistic music generation has always remained as a challenging problem as it may lack structure or rationality. In this work, we propose a deep learning based music generation method in order to produce old style music particularly JAZZ with rehashed melodic structures utilizing a Bi-directional Long Short Term Memory (Bi-LSTM) Neural Network with attention. Owing to the success in modelling long-term temporal dependencies in sequential data and its success in case of videos, Bi-LSTMs with attention serves as a natural choice and early utilization in music generation. We validate in our experiments that Bi-LSTMs with attention are able to preserve the richness and technical nuances of the music performed.

Rail Break Prediction and Cause Analysis Using Imbalanced In-Service Train Data

Timely detection and identification of rail breaks are crucial for safety and reliability of railway networks. This article proposes a new deep learning-based approach using the daily monitoring data from in-service trains. A time-series generative adversarial network (TimeGAN) is employed to mitigate the problem of data imbalance and preserve the temporal dynamics for generating synthetic rail breaks. A feature-level attention-based bidirectional recurrent neural network (AM-BRNN) is proposed to enhance feature extraction and capture two-direction dependencies in sequential data for accurate prediction. The proposed approach is implemented on a three-year dataset collected from a section of railroads (up to 350 km) in Australia. A real-life validation is carried out to evaluate the prediction performance of the proposed model, where historical data are used to train the model and future “unseen” rail breaks along the whole track section are used for testing. The results show that the model can successfully predict nine out of 11 rail breaks three months ahead of time with a false prediction of nonbreak of 8.2%. Predicting rail breaks three months ahead of time will provide railroads enough time for maintenance planning. Given the prediction results, a Shapley additive explanations (SHAP) method is employed to perform a cause analysis for individual rail break. The results of cause analysis can assist railroads to plan appropriate maintenance to prevent rail breaks.

Deep Learning Methods for Vessel Trajectory Prediction Based on Recurrent Neural Networks

Data-driven methods open up unprecedented possibilities for maritime surveillance using Automatic Identification System (AIS) data. In this work, we explore deep learning strategies using historical AIS observations to address the problem of predicting future vessel trajectories with a prediction horizon of several hours. We propose novel sequence-to-sequence vessel trajectory prediction models based on encoder-decoder recurrent neural networks (RNNs) that are trained on historical trajectory data to predict future trajectory samples given previous observations. The proposed architecture combines Long Short-Term Memory (LSTM) RNNs for sequence modeling to encode the observed data and generate future predictions with different intermediate aggregation layers to capture space-time dependencies in sequential data. Experimental results on vessel trajectories from an AIS dataset made freely available by the Danish Maritime Authority show the effectiveness of deep-learning methods for trajectory prediction based on sequence-to-sequence neural networks, which achieve better performance than baseline approaches based on linear regression or on the Multi-Layer Perceptron (MLP) architecture. The comparative evaluation of results shows: i) the superiority of attention pooling over static pooling for the specific application, and ii) the remarkable performance improvement that can be obtained with labeled trajectories, i.e., when predictions are conditioned on a low-level context representation encoded from the sequence of past observations, as well as on additional inputs (e.g., port of departure or arrival) about the vessel's high-level intention, which may be available from AIS.

Enhanced Deep Hierarchical Long Short-Term Memory and Bidirectional Long Short-Term Memory for Tamil Emotional Speech Recognition using Data Augmentation and Spatial Features

Neural networks have become increasingly popular for language modelling and within these large and deep models, overfitting, and gradient remains an important problem that heavily influences the model performance. As long short-term memory (LSTM) and bidirectional long short-term memory (BILSTM) individually solve long-term dependencies in sequential data, the combination of both LSTM and BILSTM in hierarchical gives added reliability to minimise the gradient, overfitting, and long learning issues. Hence, this paper presents four different architectures such as the Enhanced Deep Hierarchal LSTM & BILSTM (EDHLB), EDHBL, EDHLL & EDHBB has been developed. The experimental evaluation of a deep hierarchical network with spatial and temporal features selects good results for four different models. The average accuracy of EDHLB is 92.12%, EDHBL is 93.13, EDHLL is 94.14% & EDHBB is 93.19% and the accuracy level obtained for the basic models such as the LSTM, which is 74% and BILSTM, which is 77%. By evaluating all the models, EDHBL performs better than other models, with an average efficiency of 94.14% and a good accuracy rate of 95.7%. Moreover, the accuracy for the collected Tamil emotional dataset, such as happiness, fear, anger, sadness, and neutral emotions indicates 100% accuracy in a cross-fold matrix. Emotions such as disgust show around 80% efficiency. Lastly, boredom shows 75% accuracy. Moreover, the training time and evaluation time utilised by EDHBL is less when compared with the other models. Therefore, the experimental analysis shows EDHBL as superior to the other models on the collected Tamil emotional dataset. When compared with the basic models, it has attained 20% more efficiency.

Approximate LSTMs for Time-Constrained Inference: Enabling Fast Reaction in Self-Driving Cars

The need to recognize long-term dependencies in sequential data, such as video streams, has made long short-term memory (LSTM) networks a prominent artificial intelligence model for many emerging applications. However, the high computational and memory demands of LSTMs introduce challenges in their deployment on latency-critical systems such as self-driving cars, which are equipped with limited computational resources on-board. In this article, we introduce a progressive inference computing scheme that combines model pruning and computation restructuring leading to the best possible approximation of the result given the available latency budget of the target application. The proposed methodology enables mission-critical systems to make informed decisions even in early stages of the computation, based on approximate LSTM inference, meeting their specifications on safety and robustness. Our experiments on a state-of-the-art driving model for autonomous vehicle navigation demonstrate that the proposed approach can yield outputs with similar quality of result compared to a faithful LSTM baseline, up to 415× faster (198× on average, 76× geo. mean).

Temporal Pyramid Recurrent Neural Network

Learning long-term and multi-scale dependencies in sequential data is a challenging task for recurrent neural networks (RNNs). In this paper, a novel RNN structure called temporal pyramid RNN (TP-RNN) is proposed to achieve these two goals. TP-RNN is a pyramid-like structure and generally has multiple layers. In each layer of the network, there are several sub-pyramids connected by a shortcut path to the output, which can efficiently aggregate historical information from hidden states and provide many gradient feedback short-paths. This avoids back-propagating through many hidden states as in usual RNNs. In particular, in the multi-layer structure of TP-RNN, the input sequence of the higher layer is a large-scale aggregated state sequence produced by the sub-pyramids in the previous layer, instead of the usual sequence of hidden states. In this way, TP-RNN can explicitly learn multi-scale dependencies with multi-scale input sequences of different layers, and shorten the input sequence and gradient feedback paths of each layer. This avoids the vanishing gradient problem in deep RNNs and allows the network to efficiently learn long-term dependencies. We evaluate TP-RNN on several sequence modeling tasks, including the masked addition problem, pixel-by-pixel image classification, signal recognition and speaker identification. Experimental results demonstrate that TP-RNN consistently outperforms existing RNNs for learning long-term and multi-scale dependencies in sequential data.

Sentiment Analysis Using Gated Recurrent Neural Networks

Text sentiment analysis is an important and challenging task. Sentiment analysis of customer reviews is a common problem faced by companies. It is a machine learning problem made demanding due to the varying nature of sentences, different lengths of the paragraphs of text, contextual understanding, sentiment ambiguity and the use of sarcasm and comparatives. Traditional approaches to sentiment analysis use the tally or recurrence of words in a text which are allotted sentiment values by some expert. These strategies overlook the order of words and the complex different meanings they can communicate. Hence, RNNs were introduced that are effective yet challenging to train. Bi-GRUs and Bi-LSTM architectures are a recent form of RNNs which can store information about long-term dependencies in sequential data. In this work, we attempted a survey of different deep learning techniques that have been applied to sentiment classification and analysis. We have implemented the baseline models for LSTM, GRU and Bi-LSTM and Bi-GRU on an Amazon review dataset.

Dynamic temporal residual network for sequence modeling

The long short-term memory (LSTM) network with gating mechanism has been widely used in sequence modeling tasks including handwriting and speech recognition. As an LSTM network can be unfolded along the temporal dimension and its temporal depth is equal to the length of the input feature sequence, the introduction of gating might not be sufficient to completely model the dynamic temporal dependencies in sequential data. Inspired by the residual learning in ResNet, this paper proposes a dynamic temporal residual network (DTRN) by incorporating residual learning into an LSTM network along the temporal dimension. DTRN involves two networks: Its primary network consists of modified LSTM units with weighted shortcut connections for adjacent temporal outputs, while its secondary network generates dynamic weights for the shortcut connections. To validate the performance of DTRN, we conduct experiments on three commonly used public handwriting recognition datasets (IFN/ENIT, IAM and Rimes) and one speech recognition dataset (TIMIT). The experimental results show that the proposed DTRN has outperformed previously reported methods.

Hybrid Deep Learning Architectures for Multimodal Data Fusion in Healthcare Diagnostics

Integrating multimodal data, such as medical imaging, electronic health records (EHRs), and genomic data, is critical for comprehensive healthcare diagnostics. However, these data sources' heterogeneity and high dimensionality present challenges in developing robust and accurate diagnostic models. This paper proposes a hybrid deep learning architecture that combines Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformer models to achieve efficient multimodal data fusion for healthcare diagnostics. The proposed architecture leverages CNNs for extracting spatial features from image data, RNNs for capturing temporal dependencies in sequential data, and Transformers for cross-modality attention and fusion. A comprehensive evaluation of benchmark healthcare datasets, such as MIMIC-III, ChestX-ray14, and UK Biobank, demonstrates the model's superior diagnostic accuracy, interpretability, and generalization compared to existing methods. This study highlights the potential of hybrid deep learning architectures for improving diagnostic precision, enabling early disease detection, and facilitating personalized treatment strategies in real-world clinical settings. Future work will focus on enhancing model interpretability and reducing computational complexity for more practical deployment.

T -Exponential Memory Networks for Question-Answering Machines.

Recent advances in deep learning have brought to the fore models that can make multiple computational steps in the service of completing a task; these are capable of describing long-term dependencies in sequential data. Novel recurrent attention models over possibly large external memory modules constitute the core mechanisms that enable these capabilities. Our work addresses learning subtler and more complex underlying temporal dynamics in language modeling tasks that deal with sparse sequential data. To this end, we improve upon these recent advances by adopting concepts from the field of Bayesian statistics, namely, variational inference. Our proposed approach consists in treating the network parameters as latent variables with a prior distribution imposed over them. Our statistical assumptions go beyond the standard practice of postulating Gaussian priors. Indeed, to allow for handling outliers, which are prevalent in long observed sequences of multivariate data, multivariate t -exponential distributions are imposed. On this basis, we proceed to infer corresponding posteriors; these can be used for inference and prediction at test time, in a way that accounts for the uncertainty in the available sparse training data. Specifically, to allow for our approach to best exploit the merits of the t -exponential family, our method considers a new t -divergence measure, which generalizes the concept of the Kullback-Leibler divergence. We perform an extensive experimental evaluation of our approach, using challenging language modeling benchmarks, and illustrate its superiority over existing state-of-the-art techniques.