Short-term Memory Layers Research Articles

Towards a new intelligent traffic system based on deep learning and data integration

<span lang="EN-US">Time series forecasting is an important technique to study the behavior of temporal data in order to forecast the future values, which is widely applied in intelligent traffic systems (ITS). In this paper, several deep learning models were designed to deal with the multivariate time series forecasting problem for the purpose of long-term predicting traffic volume. Simulation results showed that the best forecasts are obtained with the use of two hidden long short-term memory (LSTM) layers: the first with 64 neurons and the second with 32 neurons. Over 93% of the forecasts were made with less than ±2.0% error. The analysis of variances is mainly due to peaks in some extreme conditions. For this purpose, the data was then merged between two different sources: electromagnetic loops and cameras. Data fusion is based on a calibration of the reliability of the sources according to the visibility conditions and time of the day. The integration results were then compared with the real data to prove the improvement of the prediction results in peak periods after the data fusion step.</span><p> </p>

PPG Signals-Based Blood-Pressure Estimation Using Grid Search in Hyperparameter Optimization of CNN-LSTM.

Researchers commonly use continuous noninvasive blood-pressure measurement (cNIBP) based on photoplethysmography (PPG) signals to monitor blood pressure conveniently. However, the performance of the system still needs to be improved. Accuracy and precision in blood-pressure measurements are critical factors in diagnosing and managing patients' health conditions. Therefore, we propose a convolutional long short-term memory neural network (CNN-LSTM) with grid search ability, which provides a robust blood-pressure estimation system by extracting meaningful information from PPG signals and reducing the complexity of hyperparameter optimization in the proposed model. The multiparameter intelligent monitoring for intensive care III (MIMIC III) dataset obtained PPG and arterial-blood-pressure (ABP) signals. We obtained 75,226 signal segments, with 60,180 signals allocated for training data, 12,030 signals allocated for the validation set, and 15,045 signals allocated for the test data. During training, we applied five-fold cross-validation with a grid-search method to select the best model and determine the optimal hyperparameter settings. The optimized configuration of the CNN-LSTM layers consisted of five convolutional layers, one long short-term memory (LSTM) layer, and two fully connected layers for blood-pressure estimation. This study successfully achieved good accuracy in assessing both systolic blood pressure (SBP) and diastolic blood pressure (DBP) by calculating the standard deviation (SD) and the mean absolute error (MAE), resulting in values of 7.89 ± 3.79 and 5.34 ± 2.89 mmHg, respectively. The optimal configuration of the CNN-LSTM provided satisfactory performance according to the standards set by the British Hypertension Society (BHS), the Association for the Advancement of Medical Instrumentation (AAMI), and the Institute of Electrical and Electronics Engineers (IEEE) for blood-pressure monitoring devices.

From IMU Measurement Sequence to Velocity Estimate Sequence: An Effective and Efficient Data-Driven Inertial Odometry Approach

We propose a novel deep learning-based inertial odometry framework to improve both localization accuracy and efficiency. The measured 3D accelerations and angular velocities are constructed as two independent channel 2D features. In each channel, the 3 axes of accelerations or angular velocities constitute the height dimension, whereas the temporal axis represents the width dimension. In the neural network module, we first employ 2D convolutional operations to convert the input features to 1D series with 4-fold downsampling along the temporal axis. We then apply the convolutional attention module to refine the generated 1D feature map and long short-term memory (LSTM) layers to generate the hidden state for each refined time index, which is followed by two separate 2-layer multilayer perceptrons (MLPs) to regress the velocity and the uncertainty. During backpropagation, we use the negative log-likelihood loss to optimize the velocity and the uncertainty objectives. Benefiting from the velocity sequence objective in a single network forward propagation procedure, we also add the average squared distance between the ground-truth relative position and that of the prediction to minimize the location error. Extensive experiments on both the RoNIN dataset and the real scenario collected data in the CUHK campus show higher localization accuracy with 10 times higher efficiency of the proposed framework than the comparing methods.

Prediction of the tensile strength of tablets using LSTM networks on compression profiles

This publication's objective was to predict the tensile strength of tablets using an analysis of process data comprising compression pressure, sampling timestamps, and punch positions. A recurrent neuronal network, specifically designed with Long Short-Term Memory layers, was utilized to accommodate the time-series characteristics of the data. A dataset from 344 tablet compression cycles was employed for model training, after which the model demonstrated a predictive ability with a coefficient of determination of 0.954 on test data from 804 tableting cycles. The foundational database incorporated data from both pure substances and mixtures consisting of up to four components compressed at various compression pressures and with three different tablet masses. Interestingly, the prediction errors did not exhibit any significant correlation with specific materials, mixtures, maximum compression pressures, or tablet weights. With the aid of the model, it was possible to calculate the entire tabletability profile of twelve substances from just a single compression process each. Models of this nature bear promising potential for future application in the research and development of formulations as well as in production processes to predict tensile strength.

Neural network architectures for optical channel nonlinear compensation in digital subcarrier multiplexing systems.

In this work, we propose to use various artificial neural network (ANN) structures for modeling and compensation of intra- and inter-subcarrier fiber nonlinear interference in digital subcarrier multiplexing (DSCM) optical transmission systems. We perform nonlinear channel equalization by employing different ANN cores including convolutional neural networks (CNN) and long short-term memory (LSTM) layers. First, we develop a fiber nonlinearity compensation for DSCM systems based on a fully-connected network across all subcarriers. In subsequent steps, and borrowing from the perturbation analysis of fiber nonlinearity, we gradually upgrade proposed designs towards modular structures with better performance-complexity advantages. Our study shows that putting proper macro structures in design of ANN nonlinear equalizers in DSCM systems can be crucial in development of practical solutions for future generations of coherent optical transceivers.

A two-channel deep network based model for improving ultra-short-term prediction of wind power via utilizing multi-source data

Data-driven predictions of wind turbine power outputs have received numerous discussions with using turbine data from the supervisory control and data acquisition (SCADA) system itself. It is of a high curiosity on studying the feasibility of extending the prediction capability with additionally considering data of other sources and the data-driven modeling principle of doing so. To respond such problem and introduce more innovative prediction techniques, this paper presents a pioneering attempt of studying a two-channel deep network modeling method for wind power predictions which leverage both the wind farm data and farm geoinformation. Novelty can be profiled as follows: 1) To accommodate multi-source data in the input, a new high-dimensional input form of two components, a tensor and a graph, is developed; 2) To advance predictions with such input, a deep graph attention convolutional recurrent (GACR) method, which develops one novel deep network channel with stacking multiple graph convolution and long short term memory (GCN-LSTM) layers for engineering high-level latent features from high-dimensional inputs and another classical feature selection channel for directly engaging valuable wind turbine attributes, is proposed. Comprehensive computational experiments are conducted to verify the value of such modeling development by comparing it with a set of competitive benchmarking models. An ablation study is also conducted to explain the necessity and value of network modules in the proposed method. A new state-of-the-art prediction performance is achieved.

HAR-DeepConvLG: Hybrid deep learning-based model for human activity recognition in IoT applications

Smartphones and wearable devices have built-in sensors that can collect multivariant time-series data that can be used to recognize human activities. Research on human activity recognition (HAR) has gained significant attention in recent years due to its growing demand in various application domains. As wearable sensor-aided devices and the Internet of Things (IoT) became more common, great attention has been paid to the HAR ubiquitous computing and mobile computing. To infer human activity data from a massive amount of multivariant data generated by different wearable devices, in this study an innovative deep learning–based model named HAR-DeepConvLG is proposed. It includes three convolution layers and a squeezing and excitation (SE) block, which are employed to precisely learn and extract the spatial representation data from the collected raw sensor data. The extracted features are used as input of three parallel paths, each of which includes a long short-term memory (LSTM) layer connected in sequence with a gated recurrent unit (GRU) layer to learn temporal representation. The three paths are connected in parallel to avoid the vanishing gradient problem. Finally, to evaluate the effectiveness of the proposed model, experiments were conducted on four widely utilized HAR datasets. Additionally, the model’s performance was compared to several state-of-the-art deep learning models, which further validated its effectiveness. The experimental results show that the proposed HAR-DeepConvLG model performs better than the existing HAR deep learning–based models, achieving a competitive classification accuracy.

Deep learning system for classification of ploidy status using time-lapse videos

To develop a spatiotemporal model for de prediction of euploid and aneuploid embryos using time-lapse videos from 10-115 hours after insemination (hpi). Retrospective study. The research used an end-to-end approach to develop an automated artificial intelligence system capable of extracting features from images and classifying them, considering spatiotemporal dependencies. A convolutional neural network extracted the most relevant features from each video frame. A bidirectional long short-term memory layer received this information and analyzed the temporal dependencies, obtaining a low-dimensional feature vector that characterized each video. A multilayer perceptron classified them into 2 groups, euploid and noneuploid. The model performance in accuracy fell between 0.6170 and 0.7308. A multi-input model with a gate recurrent unit module performed better than others; the precision (or positive predictive value) is 0.8205 for predicting euploidy. Sensitivity, specificity, F1-Score and accuracy are 0.6957, 0.7813, 0.7042, and 0.7308, respectively. This article proposes an artificial intelligence solution for prioritizing euploid embryo transfer. We can highlight the identification of a noninvasive method for chromosomal status diagnosis using a deep learning approach that analyzes raw data provided by time-lapse incubators. This method demonstrated potential automation of the evaluation process, allowing spatial and temporal information to encode.

Efficient intent classification and entity recognition for university administrative services employing deep learning models

The design and implementation of a domain specific conversational agent requires efficient Natural Language Understanding (NLU). The task is harder when multiple languages have to be supported, and training datasets can be beneficial. This work focuses on the development of an intelligent system, an automated multilingual customer service conversational agent (chatbot) for university students, which supports both Greek and English and combines Intent Classification or Intent Extraction (IE) and Named Entity Recognition (NER) to understand the content (i.e. type of actions conveyed and respective entities) of users' messages. We focus on the development of the fundamental tasks required by a conversational agent to provide customer services in the education industry and manage requests with instant responses and increased customer satisfaction. Instead of handling IE and NER separately, as it is common in the related work, we develop a joint model that combines Bidirectional Long Short-Term Memory (BiLSTM) and Conditional Random Fields (CRF) layers and generates outputs both for IE and NER. We introduce a novel, open access dataset for customer services in education industry, the UniWay dataset, that has been used for training and evaluating our model, comprises students' questions in English and Greek about essential information related to their studies. A comparative evaluation of the proposed model versus state-of-the-art standalone and joint model solutions in UniWay and xSID datasets, results in improvement of the performance for the IE task up to 1.4% and it is on par with the state-of-the-art for the NER task. These results justify the intuition that closed domains can benefit from less sophisticated architectures, but less costly in terms of computational and memory resources, that jointly resolve multiple NLU tasks.

Epiphany: predicting Hi-C contact maps from 1D epigenomic signals

Recent deep learning models that predict the Hi-C contact map from DNA sequence achieve promising accuracy but cannot generalize to new cell types and or even capture differences among training cell types. We propose Epiphany, a neural network to predict cell-type-specific Hi-C contact maps from widely available epigenomic tracks. Epiphany uses bidirectional long short-term memory layers to capture long-range dependencies and optionally a generative adversarial network architecture to encourage contact map realism. Epiphany shows excellent generalization to held-out chromosomes within and across cell types, yields accurate TAD and interaction calls, and predicts structural changes caused by perturbations of epigenomic signals.

Multitask learning of health state assessment and remaining useful life prediction for sensor-equipped machines

Prognostics and health management (PHM) uses data collected through sensors to monitor the states of sensor-equipped machines and provide maintenance decisions. PHM includes two essential tasks, i.e., health state (HS) assessment and remaining useful life (RUL) prediction. In existing works, the two tasks are often conducted separately without considering the relationships between HS and RUL. In this paper, we propose a multitask deep learning model for simultaneously assessing the HS and predicting the RUL of a machine; the model consists of four modules: a shared module, a multigate mixture-of-experts (MMOE) layer, an HS module, and an RUL module. In the model, a shared module including a bidirectional long short-term memory (BiLSTM) layer, an encoding layer, and a sampling layer is used to extract the shared information of the two tasks. Then, the MMOE layer is built to identify different information according to the two tasks. In the output layer, the HS module with an attention mechanism is used to evaluate the HS of the studied machine. Moreover, the RUL module predicts the RUL and constructs RUL prediction intervals to quantify uncertainty. Finally, the proposed model outperforms the state-of-the-art benchmark models and is validated on a public dataset of engines.

Speech emotion detection based on MFCC and CNN-LSTM architecture

Emotion detection techniques have been applied to multiple cases mainly from facial image features and vocal audio features, of which the latter aspect is disputed yet not only due to the complexity of speech audio processing but also the difficulties of extracting appropriate features. Part of the SAVEE and RAVDESS datasets are selected and combined as the dataset, containing seven sorts of common emotions (i.e. happy, neutral, sad, anger, disgust, fear, and surprise) and thousands of samples. Based on the Librosa package, this paper processes the initial audio input into waveplot and spectrum for analysis and concentrates on multiple features including MFCC as targets for feature extraction. The hybrid CNN-LSTM architecture is adopted by virtue of its strong capability to deal with sequential data and time series, which mainly consists of four convolutional layers and three long short-term memory layers. As a result, the architecture achieved an accuracy of 61.07% comprehensively for the test set, among which the detection of anger and neutral reaches a performance of 75.31% and 71.70% respectively. It can also be concluded that the classification accuracy is dependent on the properties of emotion to some extent, with frequently-used and distinct-featured emotions having less probability to be misclassified into other categories. Emotions like surprise whose meaning depends on the specific context are more likely to confuse with positive or negative emotions, and negative emotions also have a possibility to get mixed with each other.

Topological Similarity and Centrality Driven Hybrid Deep Learning for Temporal Link Prediction

Several real-world phenomena, including social, communication, transportation, and biological networks, can be efficiently expressed as graphs. This enables the deployment of graph algorithms to infer information from such complex network interactions to enhance graph applications’ accuracy, including link prediction, node classification, and clustering. However, the large size and complexity of the network data limit the efficiency of the learning algorithms in making decisions from such graph datasets. To overcome these limitations, graph embedding techniques are usually adopted. However, many studies not only assume static networks but also pay less attention to preserving the network topological and centrality information, which information is key in analyzing networks. In order to fill these gaps, we propose a novel end-to-end unified Topological Similarity and Centrality driven Hybrid Deep Learning model for Temporal Link Prediction (TSC-TLP). First, we extract topological similarity and centrality-based features from the raw networks. Next, we systematically aggregate these topological and centrality features to act as inputs for the encoder. In addition, we leverage the long short-term memory (LSTM) layer to learn the underlying temporal information in the graph snapshots. Lastly, we impose topological similarity and centrality constraints on the model learning to enforce preserving of topological structure and node centrality role of the input graphs in the learned embeddings. The proposed TSC-TLP is tested on 3 real-world temporal social networks. On average, it exhibits a 4% improvement in link prediction accuracy and a 37% reduction in MSE on centrality prediction over the best benchmark.

Boosting multiple sclerosis lesion segmentation through attention mechanism

Magnetic resonance imaging is a fundamental tool to reach a diagnosis of multiple sclerosis and monitoring its progression. Although several attempts have been made to segment multiple sclerosis lesions using artificial intelligence, fully automated analysis is not yet available. State-of-the-art methods rely on slight variations in segmentation architectures (e.g. U-Net, etc.). However, recent research has demonstrated how exploiting temporal-aware features and attention mechanisms can provide a significant boost to traditional architectures. This paper proposes a framework that exploits an augmented U-Net architecture with a convolutional long short-term memory layer and attention mechanism which is able to segment and quantify multiple sclerosis lesions detected in magnetic resonance images. Quantitative and qualitative evaluation on challenging examples demonstrated how the method outperforms previous state-of-the-art approaches, reporting an overall Dice score of 89% and also demonstrating robustness and generalization ability on never seen new test samples of a new dedicated under construction dataset.

Hybrid Deep Learning and Discrete Wavelet Transform-Based ECG Biometric Recognition for Arrhythmic Patients and Healthy Controls.

The intrinsic and liveness detection behavior of electrocardiogram (ECG) signals has made it an emerging biometric modality for the researcher with several applications including forensic, surveillance and security. The main challenge is the low recognition performance with datasets of large populations, including healthy and heart-disease patients, with a short interval of an ECG signal. This research proposes a novel method with the feature-level fusion of the discrete wavelet transform and a one-dimensional convolutional recurrent neural network (1D-CRNN). ECG signals were preprocessed by removing high-frequency powerline interference, followed by a low-pass filter with a cutoff frequency of 1.5 Hz for physiological noises and by baseline drift removal. The preprocessed signal is segmented with PQRST peaks, while the segmented signals are passed through Coiflets' 5 Discrete Wavelet Transform for conventional feature extraction. The 1D-CRNN with two long short-term memory (LSTM) layers followed by three 1D convolutional layers was applied for deep learning-based feature extraction. These combinations of features result in biometric recognition accuracies of 80.64%, 98.81% and 99.62% for the ECG-ID, MIT-BIH and NSR-DB datasets, respectively. At the same time, 98.24% is achieved when combining all of these datasets. This research also compares conventional feature extraction, deep learning-based feature extraction and a combination of these for performance enhancement, compared to transfer learning approaches such as VGG-19, ResNet-152 and Inception-v3 with a small segment of ECG data.

Exploring Tactile Temporal Features for Object Pose Estimation during Robotic Manipulation.

Dexterous robotic manipulation tasks depend on estimating the state of in-hand objects, particularly their orientation. Although cameras have been traditionally used to estimate the object's pose, tactile sensors have recently been studied due to their robustness against occlusions. This paper explores tactile data's temporal information for estimating the orientation of grasped objects. The data from a compliant tactile sensor were collected using different time-window sample sizes and evaluated using neural networks with long short-term memory (LSTM) layers. Our results suggest that using a window of sensor readings improved angle estimation compared to previous works. The best window size of 40 samples achieved an average of 0.0375 for the mean absolute error (MAE) in radians, 0.0030 for the mean squared error (MSE), 0.9074 for the coefficient of determination (R2), and 0.9094 for the explained variance score (EXP), with no enhancement for larger window sizes. This work illustrates the benefits of temporal information for pose estimation and analyzes the performance behavior with varying window sizes, which can be a basis for future robotic tactile research. Moreover, it can complement underactuated designs and visual pose estimation methods.

Estimating Whole-Body Walking Motion from Inertial Measurement Units at Wrist and Heels Using Deep Learning

A recurrent-neural-network-based deep-learning model was developed to estimate the three-axis joint angles of an entire body with 17 bones during walking from three inertial measurement units (IMUs) — one each on the left wrist and heels. In this model, the acceleration and angular velocity of the previous 49 frames and current frame were considered as inputs. The architecture comprises two hidden layers (two long short-term memory layers) and a dense layer. The performance of the model was evaluated using the National Institute of Advanced Industrial Science and Technology (AIST) Gait Database 2019 public dataset. Consequently, the root mean squared error of each joint angle was less than 12.28°. A comparison of the estimation results of the same model with IMUs at the pelvis and shanks revealed that the proposed model is advantageous in terms of balanced measurement accuracy and ease of use in realizing whole-body motion capture. Although the accuracy of the model was better than those of previous models in estimating the general whole-body motion from six IMUs, it was worse than that of a previous model in estimating only the lower-limb motion from three IMUs attached to the pelvis and shanks during walking. In the proposed model, IMUs are attached to the left wrist and heels, and whole-body motion can be easily captured using a smartwatch and smart shoes.

Machine learning and domain adaptation to monitor yoghurt fermentation using ultrasonic measurements

In manufacturing environments, real-time monitoring of yoghurt fermentation is required to maintain an optimal production schedule, ensure product quality, and prevent the growth of pathogenic bacteria. Ultrasonic sensors combined with machine learning models offer the potential for non-invasive process monitoring. However, methods are required to ensure the models are robust to changing ultrasonic measurement distributions as a result of changing process conditions. As it is unknown when these changes in distribution will occur, domain adaptation methods are needed that can be applied to newly acquired data in real-time. In this work, yoghurt fermentation processes are monitored using non-invasive ultrasonic sensors. Furthermore, a transmission based method is compared to an industrially-relevant non-transmission method which does not require the sound wave to travel through the fermenting yoghurt. Three machine learning algorithms were investigated including fully-connected neural networks, fully-connected neural networks with long short-term memory layers, and convolutional neural networks with long short-term memory layers. Three real-time domain adaptation strategies were also evaluated, namely; feature alignment, prediction alignment, and feature removal. The most accurate method (mean squared error of 0.008 to predict pH during fermentation) was non-transmission based and used convolutional neural networks with long short-term memory layers, and a combination of all three domain adaption methods.

Long short-term memory with activation on gradient

As the number of long short-term memory (LSTM) layers increases, vanishing/exploding gradient problems exacerbate and have a negative impact on the performance of the LSTM. In addition, the ill-conditioned problem occurs in the training process of LSTM and adversely affects its convergence. In this work, a simple and effective method of the gradient activation is applied to the LSTM, while empirical criteria for choosing gradient activation hyperparameters are found. Activating the gradient refers to modifying the gradient with a specific function named the gradient activation function. Moreover, different activation functions and different gradient operations are compared to prove that the gradient activation is effective on LSTM. Furthermore, comparative experiments are conducted, and their results show that the gradient activation alleviates the above problems and accelerates the convergence of the LSTM. The source code is publicly available at https://github.com/LongJin-lab/ACT-In-NLP.

Automatic modulation classification using bimodal parallel multichannel deep learning framework for spatial multiplexing MIMO system

In recent years, automatic modulation classification (AMC) has proved its importance for the military as well as civil applications and deep learning (DL) based AMC has attracted wide attention. But existing methods neglect to consider the advantages of both multimodality and complementarities simultaneously in a single DL framework for multiple-input multiple-output (MIMO) system. To mitigate this, bimodal multichannel configurable DL-based AMC has been presented for the MIMO system under perfect channel state information with zero forcing equalizer. The proposed DL framework consists of two parallel structures of multichannel convolutional layers in which one multichannel structure is fed with in-phase/quadrature (I/Q) as first modal information while another multichannel structure accepts amplitude/phase as second modal information. Features extracted from this parallel structure then pass through long short-term memory (LSTM) layers for further extracting temporal information effectively. Finally, classification is accomplished through fully connected layers. Simulation results manifest the robustness of the proposed framework that achieves an average accuracy of about 0.6% to 12% higher compared to the state-of-the-art DL models. Simulations also illustrate the impact of antenna diversities with spatial multiplexing on the classification.