Memory Neural Network Research Articles

Epilepsy Prediction using a Combined LSTM – XGBoost System on EEG Signals

Ranking 4th in the list of most common neurological diseases, Epilepsy – a severe chronic disorder that causes recurrent and unprovoked seizures, affects over 1% of the world population. One of the most preliminary and commonly used mechanisms to test the presence of epilepsy in patients is the electroencephalogram (EEG). EEG – an instrument capable of recording the electrical activity in the brain. The EEG data are capable of revealing information, unique to a patient with episodes of seizure. In this article a system capable of detecting such information is proposed, using neural networks and machine learning algorithms, which can be utilized in the automation process of epilepsy detection. The proposed system utilizes the Long Short-Term Memory (LSTM) neural network algorithm and the eXtreme Gradient Boosting (XGB) algorithm, to classify the channels of the EEG data. The system produces an average accuracy of 96.2% in the LSTM channel classification models and an ensemble classification of the LSTM classifications using XGB, producing an average accuracy of 98.5%. Data encoding is employed in the system, which improves the efficiency and performance of the system by exhibiting a classification duration of 31s/sample.

Meta Learning-Based MIMO Detectors: Design, Simulation, and Experimental Test

Deep neural networks (NNs) have exhibited considerable potential for efficiently balancing the performance and complexity of multiple-input and multiple-output (MIMO) detectors. However, existing NN-based MIMO detectors are difficult to be deployed in practical systems because of their slow convergence speed and low robustness in new environments. To address these issues systematically, we propose a receiver framework that enables efficient online training by leveraging the following simple observation: although NN parameters should adapt to channels, not all of them are channel-sensitive. In particular, we use a deep unfolded NN structure that represents iterative algorithms in signal detection and channel decoding modules as multi layer deep feed forward networks. An expectation propagation (EP) module, called EPNet, is established for signal detection by unfolding the EP algorithm and rendering the damping factors trainable. An unfolded turbo decoding module, called TurboNet, is used for channel decoding. This component decodes the turbo code, where trainable NN units are integrated into the traditional max-log-maximum <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a posteriori</i> decoding procedure. We demonstrate that TurboNet is robust for channels and requires only one off-line training. Therefore, only a few damping factors in EPNet must be re-optimized online. An online training mechanism based on meta learning is then developed. Here, the optimizer, which is implemented by long short-term memory NNs, is trained to update damping factors efficiently by using a small training set such that they can quickly adapt to new environments. Simulation results indicate that the proposed receiver significantly outperforms traditional receivers and that the online learning mechanism can quickly adapt to new environments. Furthermore, an over-the-air platform is presented to demonstrate the significant robustness of the proposed receiver in practical deployment.

Neural network event forecasting for robots with continuous training

Introduction: High hopes for a significant expansion of human capabilities in various fields of activity are pinned on the creation and use of highly intelligent robots. To achieve this level of robot intelligence, it is necessary to successfully solve the problems of predicting the external environment and the state of the robots themselves. Solutions based on recurrent neural networks with controlled elements are promising neural network forecasting systems. Purpose: Search for appropriate neural network structures for predicting events. Development of approaches to controlling the associative call of information from a neural network memory. Methods: Computer simulation of recurrent neural networks with controlled elements and various structures of layers. Results: An improved method of neural network event forecasting with continuous robot training has been developed. This method allows you to predict events on either long or short samples of time series. In order to improve the forecasting accuracy, new rules have been proposed for controlling the associative call of information from the neural network memory. A software system has been developed which implements the proposed method and supports the emulation of neural networks with various layer structures. The possibilities of recurrent neural networks with linear or spiral layer structures are analyzed using the example of urban traffic flow forecasting. The gain of the proposed method in comparison with the ARIMA model for the MAPE indicator is from 4.1 to 7.4%. Among the studied neural network structures, the spiral structures have shown the highest accuracy, and linear structures have shown the lowest accuracy. Practical relevance: The results of the study can be used to improve the accuracy of event forecasting for intelligent robots.

Real‐time regional seismic damage assessment framework based on long short‐term memory neural network

AbstractEffective post‐earthquake response requires a prompt and accurate assessment of earthquake‐induced damage. However, existing damage assessment methods cannot simultaneously meet these requirements. This study proposes a framework for real‐time regional seismic damage assessment that is based on a Long Short‐Term Memory (LSTM) neural network architecture. The proposed framework is not specially designed for individual structural types, but offers rapid estimates at regional scale. The framework is built around a workflow that establishes high‐performance mapping rules between ground motions and structural damage via region‐specific models. This workflow comprises three main parts—namely, region‐specific database generation, LSTM model training and verification, and model utilization for damage prediction. The influence of various LSTM architectures, hyperparameter selection, and dataset resampling procedures are systematically analyzed. As a testbed for the established framework, a case study is performed on the Tsinghua University campus buildings. The results demonstrate that the developed LSTM framework can perform damage assessment in real time at regional scale with high prediction accuracy and acceptable variance.

Decoding Kinematic Information From Primary Motor Cortex Ensemble Activities Using a Deep Canonical Correlation Analysis.

The control of arm movements through intracortical brain–machine interfaces (BMIs) mainly relies on the activities of the primary motor cortex (M1) neurons and mathematical models that decode their activities. Recent research on decoding process attempts to not only improve the performance but also simultaneously understand neural and behavioral relationships. In this study, we propose an efficient decoding algorithm using a deep canonical correlation analysis (DCCA), which maximizes correlations between canonical variables with the non-linear approximation of mappings from neuronal to canonical variables via deep learning. We investigate the effectiveness of using DCCA for finding a relationship between M1 activities and kinematic information when non-human primates performed a reaching task with one arm. Then, we examine whether using neural activity representations from DCCA improves the decoding performance through linear and non-linear decoders: a linear Kalman filter (LKF) and a long short-term memory in recurrent neural networks (LSTM-RNN). We found that neural representations of M1 activities estimated by DCCA resulted in more accurate decoding of velocity than those estimated by linear canonical correlation analysis, principal component analysis, factor analysis, and linear dynamical system. Decoding with DCCA yielded better performance than decoding the original FRs using LSTM-RNN (6.6 and 16.0% improvement on average for each velocity and position, respectively; Wilcoxon rank sum test, p < 0.05). Thus, DCCA can identify the kinematics-related canonical variables of M1 activities, thus improving the decoding performance. Our results may help advance the design of decoding models for intracortical BMIs.

An Attitude Prediction Method for Autonomous Recovery Operation of Unmanned Surface Vehicle.

The development of launch and recovery technology is key for the application to the unmanned surface vehicle (USV). Also, a launch and recovery system (L&RS) based on a pneumatic ejection mechanism has been developed in our previous study. To improve the launch accuracy and reduce the influence of the sea waves, we propose a stacking model of one-dimensional convolutional neural network and long short-term memory neural network predicting the attitude of the USV. The data from experiments by “Jinghai VII” USV developed by Shanghai University, China, under levels 1–4 sea conditions are used to train and test the network. The results show that the stabilized platform with the proposed prediction method can keep the launching angle of the launching mechanism constant by regulating the pitching joint and rotation joint under the random influence from the wave. Finally, the efficiency and effectiveness of the L&RS are demonstrated by the successful application in actual environments.

On Neural Architectures for Astronomical Time-series Classification with Application to Variable Stars

Abstract Despite the utility of neural networks (NNs) for astronomical time-series classification, the proliferation of learning architectures applied to diverse data sets has thus far hampered a direct intercomparison of different approaches. Here we perform the first comprehensive study of variants of NN-based learning and inference for astronomical time series, aiming to provide the community with an overview on relative performance and, hopefully, a set of best-in-class choices for practical implementations. In both supervised and self-supervised contexts, we study the effects of different time-series-compatible layer choices, namely the dilated temporal convolutional neural network (dTCNs), long-short term memory NNs, gated recurrent units and temporal convolutional NNs (tCNNs). We also study the efficacy and performance of encoder-decoder (i.e., autoencoder) networks compared to direct classification networks, different pathways to include auxiliary (non-time-series) metadata, and different approaches to incorporate multi-passband data (i.e., multiple time series per source). Performance—applied to a sample of 17,604 variable stars (VSs) from the MAssive Compact Halo Objects (MACHO) survey across 10 imbalanced classes—is measured in training convergence time, classification accuracy, reconstruction error, and generated latent variables. We find that networks with recurrent NNs generally outperform dTCNs and, in many scenarios, yield to similar accuracy as tCNNs. In learning time and memory requirements, convolution-based layers perform better. We conclude by discussing the advantages and limitations of deep architectures for VS classification, with a particular eye toward next-generation surveys such as the Legacy Survey of Space and Time, the Roman Space Telescope, and Zwicky Transient Facility.

Deep-learning GIS hybrid approach in precipitation modeling based on spatio-temporal variables in the coastal zone of Turkey

It is clearly known that precipitation is essential for fauna and flora. Studies have shown that location and temporal factors have an effect on precipitation. Accurate prediction of precipitation is very important for water resource management, and artificial intelligence methods are frequently used to make such predictions. In this study, the deep-learning and geographic information system (GIS) hybrid approach based on spatio-temporal variables was applied in order to model the amount of precipitation on Turkey’s coastline. Information about latitude, longitude, altitude, distance to the sea, and aspect was taken from meteorological stations, and these factors were utilized as spatial variables. The change in monthly precipitation was taken into account as a temporal variable. Artificial intelligence methods such as Gaussian process regression, support vector regression, the Broyden-Fletcher-Goldfarb-Shanno artificial neural network, M5, random forest, and long short-term memory (LSTM) were used. According to the results of the study, in which different input variable alternatives were also evaluated, LSTM was the most successful method for predicting precipitation with a value of 0.93 R. The study shows that the amount of precipitation can be estimated and a distribution map can be drawn by using spatio-temporal data and the deep-learning and GIS hybrid method at points where the measurement is not performed.

On stability and associative recall of memories in attractor neural networks.

Attractor neural networks such as the Hopfield model can be used to model associative memory. An efficient associative memory should be able to store a large number of patterns which must all be stable. We study in detail the meaning and definition of stability of network states. We reexamine the meanings of retrieval, recognition and recall and assign precise mathematical meanings to each of these terms. We also examine the relation between them and how they relate to memory capacity of the network. We have shown earlier in this journal that orthogonalization scheme provides an effective way of overcoming catastrophic interference that limits the memory capacity of the Hopfield model. It is not immediately apparent whether the improvement made by orthgonalization affects the processes of retrieval, recognition and recall equally. We show that this influence occurs to different degrees and hence affects the relations between them. We then show that the conditions for pattern stability can be split into a necessary condition (recognition) and a sufficient one (recall). We interpret in cognitive terms the information being stored in the Hopfield model and also after it is orthogonalized. We also study the alterations in the network dynamics of the Hopfield network upon the introduction of orthogonalization, and their effects on the efficiency of the network as an associative memory.

Bottleneck Based Gridlock Prediction in an Urban Road Network Using Long Short-Term Memory

The traffic bottlenecks in urban road networks are more challenging to investigate and discover than in freeways or simple arterial networks. A bottleneck indicates the congestion evolution and queue formation, which consequently disturb travel delay and degrade the urban traffic environment and safety. For urban road networks, sensors are needed to cover a wide range of areas, especially for bottleneck and gridlock analysis, requiring high installation and maintenance costs. The emerging widespread availability of GPS vehicles significantly helps to overcome the geographic coverage and spacing limitations of traditional fixed-location detector data. Therefore, this study investigated GPS vehicles that have passed through the links in the simulated gridlock-looped intersection area. The sample size estimation is fundamental to any traffic engineering analysis. Therefore, this study tried a different number of sample sizes to analyze the severe congestion state of gridlock. Traffic condition prediction is one of the primary components of intelligent transportation systems. In this study, the Long Short-Term Memory (LSTM) neural network was applied to predict gridlock based on bottleneck states of intersections in the simulated urban road network. This study chose to work on the Chula-Sathorn SUMO Simulator (Chula-SSS) dataset. It was calibrated with the past actual traffic data collection by using the Simulation of Urban MObility (SUMO) software. The experiments show that LSTM provides satisfactory results for gridlock prediction with temporal dependencies. The reported prediction error is based on long-range time dependencies on the respective sample sizes using the calibrated Chula-SSS dataset. On the other hand, the low sampling rate of GPS trajectories gives high RMSE and MAE error, but with reduced computation time. Analyzing the percentage of simulated GPS data with different random seed numbers suggests the possibility of gridlock identification and reports satisfying prediction errors.

Waveform optimization of a two-axis smooth impact drive mechanism actuator

This paper presents a data-driven method for waveform optimization of a two-axis smooth impact drive mechanism (SIDM) actuator. The actuator was constructed by two piezoelectric elements (PZTs) perpendicularly fixed to an L-shaped base for two-axis positioning. An XY stage was designed and constructed by assembling the two-axis SIDM actuator. The XY stage could position long motion ranges of several millimeters with nanometer-level resolution, and the size was confined to be 20 mm (X) × 20 mm (Y) × 4.5 mm (H). The data-driven method based on the long short-term memory (LSTM) neural networks was used to predict the optimum input voltage waveform of the actuator. With the optimized input voltage waveform, it was verified that the maximum velocity of the stage could be improved about two times.

Ultrashort-Term Power Fluctuation Forecasting Based on the Prediction of the Shipborne Panel Tilt Angle

When a solar ship is navigating in the ocean, the swaying motion of a photovoltaic panel will affect the output power of the photovoltaic (PV) power generation system more frequently and violently. In addition to considering multiple climatic factors, this paper also adopts a ship swaying motion and radiation level of sunlight to establish a suitable calculation model for the output power of photovoltaic systems, which are rarely considered at the same time in previous studies, and also to make ultrashort-term power predictions. Furthermore, this paper proposes a multilayer heterogeneous particle swarm optimization (PSO) algorithm to design the weights and thresholds of a long short-term memory (LSTM) neural network to improve the accuracy of forecasting the changes of a photovoltaic panel’s angle, which is used for accurate power output prediction for the purpose of power planning. The case analysis shows the effectiveness of the algorithm, which provides a more reliable method for designing a power prediction system.

Unsupervised Domain Adaptation based Remaining Useful Life Prediction of Rolling Element Bearings

With the rise of Artificial Intelligence (AI), machine learning techniques are now conquering the research field of Prognostics and Health Management (PHM). Classic deployable prognostic models manipulate large amount of machinery historical data to map the degradation process based on inherent features. Nowadays one of the major challenges in prognostics research is the data deficit problem when historical data is not available or accessible, in enough quantity and variety. In the frames of Transfer Learning, the domain adaptation technique aims to build a model with strong generalization ability which can be transferred to datasets with different distributions. In this paper, a Domain Adversarial Neural Network (DANN) model is combined with a Bidirectional Long Short- Term Memory (Bi-LSTM) neural network for the estimation of the Remaining Useful Life (RUL) of rolling element bearings. The unsupervised domain adaptation is fulfilled using a labelled bearing degradation dataset as the source domain data and an unlabelled dataset captured under different operation conditions as the target domain data for the Bi-LSTM DANN. The proposed method achieves promising results, applied on real bearing vibration data captured on run-to-failure tests, with high prediction accuracy of the bearing RUL compared to un-adapted methods.

Language Translation on Intelligent Navigation System using Image Processing

Visitors traveling to different countries around the world often find it hard to understand and communicate in local languages, because they don't understand it. They can't read the words written on the navigational boards or banners at these new locations. Text detection, extraction, and translation system must, therefore, be built to identify and recognize the text found on the navigation boards. This system proposes and implements a three-stage process that involves detection, extraction, and translation using the concepts of Convolutional Neural Network (CNN) and Long Short Term Memory networks or simply “LSTMs”. The framework has been designed to take into account the need to create a desktop application that extracts the text from images based on traffic navigation boards and then translates it further into a user-understandable language. In this way, the user can grasp the unfamiliar language and roam freely in the unfamiliar terrains.

PM2.5 Forecast Based on a Multiple Attention Long Short-Term Memory (MAT-LSTM) Neural Networks

Air pollution, especially by particulate matter with diameters less than 2.5 μm (PM2.5), is a serious threat to public health. The accurate prediction of PM2.5 concentration is significant for air pollution control and the prevention of health issues. However, accurate prediction and forecasting of PM2.5 has been challenging. In this study, a multiple attention (MAT) mechanism based on multilayer perception, which includes monitoring site attention, time feature attention, and weather attention, was designed to obtain the spatial-temporal and meteorological dependences of PM2.5. A hybrid deep learning method based on MAT long short-term memory (MAT-LSTM) neural networks is proposed to predict PM2.5 concentration. To validate the effectiveness of the proposed model, hourly air pollution, and meteorological measurements were collected at 10 monitoring sites in Hefei City from May 13, 2014 to March 21, 2020. Comparison experiments were performed using the MAT-LSTM model, recurrent neural network (RNN), back propagation (BP), and LSTM neural network models. The comparison of the results demonstrated that the MAT-LSTM model was superior to the baseline models and achieved reductions of 86.44%, 69.64%, and 35.2% in the mean absolute percentage error (MAPE) compared with RNN, BP, and LSTM, respectively. In addition, the PM2.5 forecast experiments were conducted at different time steps from 1 to 12 h. The results show that the proposed model performs well and exhibits an MAPE of 17.86% at the future time step of 1 h, and the prediction performance of the proposed model is satisfactory, even for the next 12-h forecast.

Neural Memory Networks for Seizure Type Classification.

Classification of seizure type is a key step in the clinical process for evaluating an individual who presents with seizures. It determines the course of clinical diagnosis and treatment, and its impact stretches beyond the clinical domain to epilepsy research and the development of novel therapies. Automated identification of seizure type may facilitate understanding of the disease, and seizure detection and prediction have been the focus of recent research that has sought to exploit the benefits of machine learning and deep learning architectures. Nevertheless, there is not yet a definitive solution for automating the classification of seizure type, a task that must currently be performed by an expert epileptologist. Inspired by recent advances in neural memory networks (NMNs), we introduce a novel approach for the classification of seizure type using electrophysiological data. We first explore the performance of traditional deep learning techniques which use convolutional and recurrent neural networks, and enhance these architectures by using external memory modules with trainable neural plasticity. We show that our model achieves a state-of-the-art weighted F1 score of 0.945 for seizure type classification on the TUH EEG Seizure Corpus with the IBM TUSZ preprocessed data. This work highlights the potential of neural memory networks to support the field of epilepsy research, along with biomedical research and signal analysis more broadly.

Air Pollution Prediction Using Recurrent Neural Network, Long Short-Term Memory and Hybrid of Convolutional Neural Network and Long Short-Term Memory Models

Air pollution prediction was not an easy task few years back. With the increasing computation power and wide availability of the datasets, air pollution prediction problem is solved to some extend. Inspired by the deep learning models, in this paper three techniques for air pollution prediction have been proposed. The models used includes recurrent neural network (RNN), Long short-term memory (LSTM) and a hybrid combination of Convolutional neural network (CNN) and LSTM models. These models are tested by comparing MSE loss on air pollution test of Belgium. The validation loss on RNN is 0.0045, LSTM is 0.00441 and CNN and LSTM is 0.0049. The loss on testing dataset for these models are 0.00088, 0.00441 and 0.0049 respectively.

LSTM networks based on attention ordered neurons for gear remaining life prediction

Gear is a commonly-used rotating part in industry, it is of great significance to predict its failure in advance, which is helpful to maintain the health of the whole machine. Firstly, the isometric mapping algorithm is applied to construct the health indicator (HI) based on the statistical characteristics of gear. Then a novel variant of long-short-term memory neural network with attention-guided ordered neurons (LSTM-AON) is constructed to achieve the accurate prediction of gear remaining useful life (RUL). LSTM-AON divides the hierarchy of health characteristic information via attention ordered neurons, so that it can use the sequence information of neurons to improve the predictive performance, which improves the long-term prediction ability and robustness. The experiments show the superiority of the new gear RUL prediction methodology based on LSTM-AON compared to the current prediction methods.

Using Long Short-Term Memory (LSTM) Neural Networks to Predict Emergency Department Wait Time.

Emergency Department (ED) overcrowding is a major global healthcare issue. Many research studies have been conducted to predict ED wait time using various machine learning prediction models to enhance patient experience and improve care efficiency and resource allocation. In this paper, we used Long Short-Term Memory (LSTM) recurrent neural networks to build a model to predict ED wait time in the next 2 hours using a randomly generated patient timestamp dataset of a typical patient hospital journey. Compared with Linear Regression model, the average mean absolute error for the LSTM model is decreased by 9.7% (3 minutes) (p < 0.01). The LSTM model statistically outperforms the LR model, however, both models could be practically useful in ED wait time prediction.

Using Long Short-Term Memory (LSTM) Neural Networks to Predict Emergency Department Wait Time.

Emergency Department (ED) overcrowding is a major global healthcare issue. In this paper, we used Long Short-Term Memory (LSTM) recurrent neural networks to build a model to predict ED wait time in the next 2 hours using a randomly generated patient timestamp dataset of a typical patient hospital journey. Compared with Linear Regression model, the average mean absolute error for the LSTM model is decreased by 15% (3 minutes) (p<0.001). The LSTM model statistically outperforms the LR model, however, both models could be practically useful in ED wait time prediction.