Memory Deep Neural Network Research Articles

Deep Learning for Recognition of Digital Modulations: A Detailed Study

The automatic modulation recognition of the received signal is very attractive in both military and civilian applications. In recent years, deep learning techniques have received much attention since their excellent performance in signal, audio, image and video processing. This paper examines the feasibility of using deep learning algorithms on automatic recognition of the received radio signals' modulation schemes. Modulation recognition has been performed on eight digital modulation types with a signal-to-noise ratio (SNR) from -20dB to 20dB. First, a vanilla neural network is used to classify the type of modulation. Then, convolutional neural networks (CNN) and recurrent neural networks are applied for modulation recognition. These neural networks are widely used in image and signal processing applications. This is followed by designing the other architectures, including densely connected neural network (DenseNet), inception network, recurrent neural network (RNN), long-short term memory network (LSTM), and convolutional long short-term memory deep neural network (CLDNN) for modulation recognition problem, and their results are presented. During this investigation, a basic model is initially considered for each architecture, and then the network performance is studied by adjusting its parameters. The simulation results show that the proposed modified CLDNN model can provide an accuracy of 98% in high SNRs.

Air quality prediction using CNN+LSTM-based hybrid deep learning architecture.

Air pollution prediction based on variables in environmental monitoring data gains further importance with increasing concerns about climate change and the sustainability of cities. Modeling of the complex relationships between these variables by sophisticated methods in machine learning is a promising field. The objectives of this work are to develop a supervised model for the prediction of air pollution by using real sensor data and to transfer the model between cities. The combination of a convolutional neural network and a long short-term memory deep neural network model was proposed to predict the concentration of air pollutants in multiple locations of a city by using spatial-temporal relationships. Two approaches have been adopted: the univariate model contains the information of one pollutant whereas the multivariate model contains the information of all pollutants and meteorology data for prediction. The study was carried out for different pollutants which are in the publicly available data of the cities of Barcelona, Kocaeli, and İstanbul. The hyperparameters of the model (filter, frame, and batch sizes; number of convolutional/LSTM layers and hidden units; learning rate; and parameters for sample selection, pooling, and validation) were tuned to determine the architecture that achieved the lowest test error. The proposed model improved the prediction performance (measured by the root mean square error) by 11-53% for particulate matter, 20-31% for ozone, 9-47% for nitrogenoxides, and 18-46% for sulfurdioxide with respect to the 1-hidden layer long short-term memory networks utilized in the literature. The multivariate model without using meteorological data revealed the best results. Regarding transfer learning, the network weights were transferred from the source city to the target city. The model has more accurate prediction performance with the transfer of the network from Kocaeli to İstanbul as those neighbor cities have similar air pollution and meteorological characteristics.

Implementation of Long-Short Term Memory Neural Network (LSTM) for Predicting The Water Quality Parameters in Sungai Selangor

Predictions of future events must be factored into decision-making. Predictions of water quality are critical to assist authorities in making operational, management, and strategic decisions to keep the quality of water supply monitored under specific criteria. Taking advantage of the good performance of long short-term memory (LSTM) deep neural networks in time-series prediction, the purpose of this paper is to develop and train a Long-Short Term Memory (LSTM) Neural Network to predict water quality parameters in the Selangor River. The primary goal of this study is to predict five (5) water quality parameters in the Selangor River, namely Biochemical Oxygen Demand (BOD), Ammonia Nitrogen (NH3-N), Chemical Oxygen Demand (COD), pH, and Dissolved Oxygen (DO), using secondary data from different monitoring stations along the river basin. The accuracy of this method was then measured using RMSE as the forecast measure. The results show that by using the Power of Hydrogen (pH), the dataset yielded the lowest RMSE value, with a minimum of 0.2106 at station 004 and a maximum of 1.2587 at station 001. The results of the study indicate that the predicted values of the model and the actual values were in good agreement and revealed the future developing trend of water quality parameters, showing the feasibility and effectiveness of using LSTM deep neural networks to predict the quality of water parameters.

Machine Learning in Modeling of Mouse Behavior.

Mouse behavior is a primary outcome in evaluations of therapeutic efficacy. Exhaustive, continuous, multiparametric behavioral phenotyping is a valuable tool for understanding the pathophysiological status of mouse brain diseases. Automated home cage behavior analysis produces highly granulated data both in terms of number of features and sampling frequency. Previously, we demonstrated several ways to reduce feature dimensionality. In this study, we propose novel approaches for analyzing 33-Hz data generated by CleverSys software. We hypothesized that behavioral patterns within short time windows are reflective of physiological state, and that computer modeling of mouse behavioral routines can serve as a predictive tool in classification tasks. To remove bias due to researcher decisions, our data flow is indifferent to the quality, value, and importance of any given feature in isolation. To classify day and night behavior, as an example application, we developed a data preprocessing flow and utilized logistic regression (LG), support vector machines (SVM), random forest (RF), and one-dimensional convolutional neural networks paired with long short-term memory deep neural networks (1DConvBiLSTM). We determined that a 5-min video clip is sufficient to classify mouse behavior with high accuracy. LG, SVM, and RF performed similarly, predicting mouse behavior with 85% accuracy, and combining the three algorithms in an ensemble procedure increased accuracy to 90%. The best performance was achieved by combining the 1DConv and BiLSTM algorithms yielding 96% accuracy. Our findings demonstrate that computer modeling of the home-cage ethome can clearly define mouse physiological state. Furthermore, we showed that continuous behavioral data can be analyzed using approaches similar to natural language processing. These data provide proof of concept for future research in diagnostics of complex pathophysiological changes that are accompanied by changes in behavioral profile.

Day-ahead electricity price prediction applying hybrid models of LSTM-based deep learning methods and feature selection algorithms under consideration of market coupling

The availability of accurate day-ahead electricity price forecasts is pivotal for electricity market participants. In the context of trade liberalisation and market harmonisation in the European markets, accurate price forecasting becomes difficult for electricity market participants to obtain because electricity forecasting requires the consideration of features from ever-growing coupling markets. This study provides a method of exploring the influence of market coupling on electricity price prediction. We apply state-of-the-art long short-term memory (LSTM) deep neural networks combined with feature selection algorithms for electricity price prediction under the consideration of market coupling. LSTM models have a good performance in handling nonlinear and complex problems and processing time series data. In our empirical study of the Nordic market, the proposed models obtain considerably accurate results. The results show that feature selection is essential to achieving accurate prediction, and features from integrated markets have an impact on prediction. The feature importance analysis implies that the German market has a salient role in the price generation of Nord Pool.

Fault detection by an ensemble framework of Extreme Gradient Boosting (XGBoost) in the operation of offshore wind turbines

Offshore wind is a rapidly maturing renewable energy that has presented a large growth over the last decade. This increase in offshore wind capacity has led to the need for more effective monitoring strategies, as currently, Operation and Maintenance (O&M) costs make up to 30% of the overall cost of energy. This study presented a novel data-driven approach to condition monitoring systems by utilizing the existing Supervisory Control And Data Acquisition (SCADA) system and integrating a wide range of machine learning and data mining techniques namely: data pre-processing & re-sampling, anomalies detection & treatment, feature engineering, and hyperparameter optimization, to design a Normal Behaviour Model of the generator for fault detection purposes. An ensemble model of the Extreme Gradient Boosting (XGBoost) framework was successfully developed and critically compared with a Long Short-Term Memory (LSTM) deep learning neural network. The results showed that, in terms of temperature prediction, the proposed methodology captures a high level of accuracy at low computational costs. Moreover, it can be concluded that XGBoost outperformed LSTM in predictive accuracy whilst requiring smaller training times and showcasing a smaller sensitivity to noise that existed in the SCADA database.

Data‐Driven Forecasting of Low‐Latitude Ionospheric Total Electron Content Using the Random Forest and LSTM Machine Learning Methods

AbstractIn this research, we present data‐driven forecasting of ionospheric total electron content (TEC) using the Long‐Short Term Memory (LSTM) deep recurrent neural network method. The random forest machine learning method was used to perform a regression analysis and estimate the variable importance of the input parameters. The input data are obtained from satellite and ground based measurements characterizing the solar‐terrestrial environment. We estimate the relative importance of 34 different parameters, including the solar flux, solar wind density, and speed the three components of interplanetary magnetic field, Lyman‐alpha, the Kp, Dst, and Polar Cap (PC) indices. The TEC measurements are taken with 15‐s cadence from an equatorial GPS station located at Bogota, Columbia (4.7110° N, 74.0721° W). The 2008–2017 data set, including the top five parameters estimated using the random forest, is used for training the machine learning models, and the 2018 data set is used for independent testing of the LSTM forecasting. The LSTM method as applied to forecast the TEC up to 5 h ahead, with 30‐min cadence. The results indicate that very good forecasts with low root mean square (RMS) error (high correlation) can be made in the near future and the RMS errors increase as we forecast further into the future. The data sources are satellite and ground based measurements characterizing the solar‐terrestrial environment.

Prediction of COVID-19 Confirmed Cases after Vaccination: Based on Statistical and Deep Learning Models

In this paper, we analyze and predict the number of daily confirmed cases of coronavirus (COVID-19) based on two statistical models and a deep learning (DL) model; the autoregressive integrated moving average (ARIMA), the generalized autoregressive conditional heteroscedasticity (GARCH), and the stacked long short-term memory deep neural network (LSTM DNN). We find the orders of the statistical models by the autocorrelation function and the partial autocorrelation function, and the hyperparameters of the DL model, such as the numbers of LSTM cells and blocks of a cell, by the exhaustive search. Ten datasets are used in the experiment; nine countries and the world datasets, from Dec. 31, 2019, to Feb. 22, 2021, provided by the WHO. We investigate the effects of data size and vaccination on performance. Numerical results show that performance depends on the used data's dates and vaccination. It also shows that the prediction by the LSTM DNN is better than those of the two statistical models. Based on the experimental results, the percentage improvements of LSTM DNN are up to 88.54% (86.63%) and 90.15% (87.74%) compared to ARIMA and GARCH, respectively, in mean absolute error (root mean squared error). While the performances of ARIMA and GARCH are varying according to the datasets. The obtained results may provide a criterion for the performance ranges and prediction accuracy of the COVID-19 daily confirmed cases.Doi: 10.28991/SciMedJ-2021-0302-7 Full Text: PDF

A Novel Signal Localized Convolution Neural Network for Power Transformer Differential Protection

This paper presents a novel signal localized convolution neural network (SLCNN) for the power transformer differential protection. The distinct signal localization is performed with the convolution process sequentially on the frequency and time coefficients which are obtained from the wavelet decomposition of the differential current signal. The SLCNN is trained with a modified back-propagation algorithm according to SLCNNs architecture. Three power transformer test systems are considered for evaluation of the proposed SLCNN. The training patterns of each transformer are generated for various operating conditions. The SLCNN for each transformer is trained, validated and tested using its corresponding patterns. Then the performance of SLCNN is evaluated through confusion matrix analysis and is also compared with long short-term memory (LSTM) deep neural network, support vector machine (SVM), conventional back-propagation neural network (CBPNN) and conventional biased restraint second harmonic (CBSH) blocking method.

HUMAN ROBOT INTERACTIVE INTENTION PREDICTION USING DEEP LEARNING TECHNIQUES

In this research, we propose a method of human robot interactive intention prediction. The proposed algorithm makes use of a OpenPose library and a Long-short term memory deep learning neural network. The neural network observes the human posture in a time series, then predicts the human interactive intention. We train the deep neural network using dataset generated by us. The experimental results show that, our proposed method is able to predict the human robot interactive intention, providing 92% the accuracy on the testing set.

Wearables and Deep Learning Classify Fall Risk From Gait in Multiple Sclerosis.

Falls are a significant problem for persons with multiple sclerosis (PwMS). Yet fall prevention interventions are not often prescribed until after a fall has been reported to a healthcare provider. While still nascent, objective fall risk assessments could help in prescribing preventative interventions. To this end, retrospective fall status classification commonly serves as an intermediate step in developing prospective fall risk assessments. Previous research has identified measures of gait biomechanics that differ between PwMS who have fallen and those who have not, but these biomechanical indices have not yet been leveraged to detect PwMS who have fallen. Moreover, they require the use of laboratory-based measurement technologies, which prevent clinical deployment. Here we demonstrate that a bidirectional long short-term (BiLSTM) memory deep neural network was able to identify PwMS who have recently fallen with good performance (AUC of 0.88) based on accelerometer data recorded from two wearable sensors during a one-minute walking task. These results provide substantial improvements over machine learning models trained on spatiotemporal gait parameters (21% improvement in AUC), statistical features from the wearable sensor data (16%), and patient-reported (19%) and neurologist-administered (24%) measures in this sample. The success and simplicity (two wearable sensors, only one-minute of walking) of this approach indicates the promise of inexpensive wearable sensors for capturing fall risk in PwMS.

Multi-chromosomal CGP-evolved RNN for signal reconstruction

A multi-chromosomal structure for neuro-evolution of recurrent neural networks (RNNs) with Cartesian genetic programming (CGP) architecture is presented to develop a signal reconstruction model. The group behaviour of multi-chromosomes evolved together is explored in terms of computational and structural complexity, and accurate reconstruction of the corrupted signal. Signal reconstruction modelling is done through prediction of missing samples, and a sliding window mechanism is utilized to recover long segments of missing samples in periodic and aperiodic corrupted signals. The proposed model is compared with the conventional single chromosome-based Cartesian Genetic Programming-evolved Recurrent Neural Network (CGPRNN) architecture as well as state-of-the-art machine learning algorithms, i.e. linear regression, random forest and long short-term memory deep learning neural network. Diverse network architectures of the proposed Multi-Chromosomal Cartesian Genetic Programming-evolved Recurrent Neural Network (MC-CGPRNN) are evolved to explore the robustness of algorithm in terms of optimization of hyper parameters, evolutionary speed and generalization ability. The proposed algorithm is trained and tested on speech signal with both periodic and aperiodic noise. The best trained network is also evaluated on standard music signals achieving remarkable results. Experimental results show that the proposed MC-CGPRNN algorithm achieved SNR of 25.23 dB for speech signal, 29.92 dB for guitar signal and 28.45 dB for flute signal in case of 50% missing samples showing best SNR improvement in comparison with its counterparts for time domain signal reconstruction.

Non-invasive wearable seizure detection using long–short-term memory networks with transfer learning

Objective. The detection of seizures using wearable devices would improve epilepsy management, but reliable detection of seizures in an ambulatory environment remains challenging, and current studies lack concurrent validation of seizures using electroencephalography (EEG) data. Approach. An adaptively trained long–short-term memory deep neural network was developed and trained using a modest number of seizure data sets from wrist-worn devices. Transfer learning was used to adapt a classifier that was initially trained on intracranial electroencephalography (iEEG) signals to facilitate classification of non-EEG physiological datasets comprising accelerometry, blood volume pulse, skin electrodermal activity, heart rate, and temperature signals. The algorithm’s performance was assessed with and without pre-training on iEEG signals and transfer learning. To assess the performance of the seizure detection classifier using long-term ambulatory data, wearable devices were used for multiple months with an implanted neurostimulator capable of recording iEEG signals, which provided independent electrographic seizure detections that were reviewed by a board-certified epileptologist. Main results. For 19 motor seizures from 10 in-hospital patients, the algorithm yielded a mean area under curve (AUC), a sensitivity, and an false alarm rate per day (FAR/day) of 0.98, 0.93, and 2.3, respectively. Additionally, for eight seizures with probable motor semiology from two ambulatory patients, the classifier achieved a mean AUC of 0.97 and an FAR of 2.45 events/day at a sensitivity of 0.9. For all seizure types in the ambulatory setting, the classifier had a mean AUC of 0.82 with a sensitivity of 0.47 and an FAR of 7.2 events/day. Significance. The performance of the algorithm was evaluated using motor and non-motor seizures during in-hospital and ambulatory use. The classifier was able to detect multiple types of motor and non-motor seizures, but performed significantly better on motor seizures.

Derin Öğrenme İle Türkçe Müziklerden Müzik Türü Sınıflandırması

In this study, an architecture called Convolutional Long Short-term memory deep neural network (CLDNN) based on deep learning, which has not been used before in this field, is used for music genre classification. In addition, a new Turkish Music Database consisting of 200 music belonging to various music genres has been created. The classification performance of the proposed architecture and commonly used machine learning methods has been evaluated on this database. In addition, new features are obtained by using Convolutional Neural Network (CNN), which is the first part of this architecture. Both Mel Frequency Cepstrum Coefficients (MFCC) and log mel filterbank energies were used as input to the Convolutional Neural Network to obtain these new features. In addition to these features, many standard features have been obtained by using various toolboxes. The most successful classification results for all methods are achieved when standard features are used together with new features. The best results among the compared classifiers were achieved with 99.5% by using the remaining part of the proposed architecture, Long Short Term Memory (LSTM), together with the Deep Neural Network (DNN) consisting of fully connected layers.

Forecast-Based Consensus Control for DC Microgrids Using Distributed Long Short-Term Memory Deep Learning Models

In a microgrid, renewable energy sources (RES) exhibit stochastic behavior, which affects the microgrid continuous operation. Normally, energy storage systems (ESSs) are installed on the main branches of the microgrids to compensate for the load-supply mismatch. However, their state of charge (SoC) level needs to be balanced to guarantee the continuous operation of the microgrid in case of RES unavailability. This paper proposes a distributed forecast-based consensus control strategy for DC microgrids that balances the SoC levels of ESSs. By using the load-supply forecast of each branch, the microgrid operational continuity is increased while the voltage is stabilized. These objectives are achieved by prioritized (dis)charging of ESSs based on the RES availability and load forecast. Each branch controller integrates a load forecasting unit based on long short-term memory (LSTM) deep neural network that adaptively adjusts the (dis)charging rate of the ESSs to increase the microgrid endurability in the event of temporary generation insufficiencies. Furthermore, due to the large training data requirements of the LSTM models, distributed extended Kalman filter algorithm is used to improve the learning convergence time. The performance of the proposed strategy is evaluated on an experimental 380V DC microgrid hardware-in-the-loop test-bench and the results confirm the achievement of the controller objectives.

State of Charge and State of Energy Estimation for Lithium-Ion Batteries Based on a Long Short-Term Memory Neural Network

State of charge (SOC) and state of energy (SOE) are two crucial battery states which correspond to available capacity in Ah and available energy in Wh, respectively. Both of them play a pivotal role in battery management, however, the joint estimation of the two states was rarely studied. This study investigates a novel data-driven method that can estimate SOC and SOE simultaneously based on a long short-term memory (LSTM) deep neural network. The proposed algorithm is validated with two dynamic driven cycles under various working conditions, such as different temperatures, different battery material and noise interference. The mean absolute error (MAE) of SOC and SOE estimation achieve 0.91% and 1.09% under a fixed temperature condition, 0.63% and 0.64% for a different battery, and 1.32% and 1.19% with noise interference, respectively. The computational burden and network setting are also studied. In addition, the performance of the proposed method is compared with other popular algorithms, including support vector regression (SVR), random forest (RF) and simple recurrent neural network (Simple RNN). The results show that the proposed method obtains higher accuracy and robustness. This study provides a new way of conducting multiple state estimation of batteries using a deep-learning approach.

New Results for Prediction of Chaotic Systems Using Deep Recurrent Neural Networks

Prediction of nonlinear and dynamic systems is a challenging task, however with the aid of machine learning techniques, particularly neural networks, is now possible to accomplish this objective. Most common neural networks used are the multilayer perceptron (MLP) and recurrent neural networks (RNN) using long-short term memory units (LSTM-RNN). In recent years, deep learning neural network models have become more relevant due the improved results they show for various tasks. In this paper the authors compare these neural network models with deep learning neural network models such as long-short term memory deep recurrent neural network (LSTM-DRNN) and gate recurrent unit deep recurrent neural network (GRU-DRNN) when presented with the task of predicting three different chaotic systems such as the Lorenz system, Rabinovich–Fabrikant and the Rossler System. The results obtained show that the deep learning neural network model GRU-DRNN has better results when predicting these three chaotic systems in terms of loss and accuracy than the two other models using less neurons and layers. These results can be very helpful to solve much more complex problems such as the control and synchronization of these chaotic systems.

Predicting stock returns of Tehran exchange using LSTM neural network and feature engineering technique

Prediction is defined as the expression of events that will occur in the future, before they occur, based on scientific and logical principles and rules. Due to the importance of financial markets for economic activists, prediction in this field has received much attention from scholars. Prediction of the stock market, as one of the largest financial markets can be very profitable to the predictors. The dynamic and complexity of the market has added to its appeal to researchers. To date, many researchers have reported good returns for prediction in this market using neural network methods. In this paper, we attempted to obtain better results on Tehran Stock Exchange by using their findings and by applying the Long Short-Term Memory (LSTM) deep neural network. In the area of feature engineering, we have tried to reduce the number of features using AutoEncoder-based feature selection to improve stock returns and reduce prediction error. To evaluate the proposed method, a return measure that is closer to the real world of stock trading was used. Experimental results showed that using the proposed method yielded a better output with a lower error mean.

Real-Time Driving Distraction Recognition Through a Wrist-Mounted Accelerometer.

We propose a method for recognizing driver distraction in real time using a wrist-worn inertial measurement unit (IMU). Distracted driving results in thousands of fatal vehicle accidents every year. Recognizing distraction using body-worn sensors may help mitigate driver distraction and consequently improve road safety. Twenty participants performed common behaviors associated with distracted driving while operating a driving simulator. Acceleration data collected from an IMU secured to each driver's right wrist were used to detect potential manual distractions based on 2-s long streaming data. Three deep neural network-based classifiers were compared for their ability to recognize the type of distractive behavior using F1-scores, a measure of accuracy considering both recall and precision. The results indicated that a convolutional long short-term memory (ConvLSTM) deep neural network outperformed a convolutional neural network (CNN) and recursive neural network with long short-term memory (LSTM) for recognizing distracted driving behaviors. The within-participant F1-scores for the ConvLSTM, CNN, and LSTM were 0.87, 0.82, and 0.82, respectively. The between-participant F1-scores for the ConvLSTM, CNN, and LSTM were 0.87, 0.76, and 0.85, respectively. The results of this pilot study indicate that the proposed driving distraction mitigation system that uses a wrist-worn IMU and ConvLSTM deep neural network classifier may have potential for improving transportation safety.

Discriminative Multi-Stream Postfilters Based on Deep Learning for Enhancing Statistical Parametric Speech Synthesis.

Statistical parametric speech synthesis based on Hidden Markov Models has been an important technique for the production of artificial voices, due to its ability to produce results with high intelligibility and sophisticated features such as voice conversion and accent modification with a small footprint, particularly for low-resource languages where deep learning-based techniques remain unexplored. Despite the progress, the quality of the results, mainly based on Hidden Markov Models (HMM) does not reach those of the predominant approaches, based on unit selection of speech segments of deep learning. One of the proposals to improve the quality of HMM-based speech has been incorporating postfiltering stages, which pretend to increase the quality while preserving the advantages of the process. In this paper, we present a new approach to postfiltering synthesized voices with the application of discriminative postfilters, with several long short-term memory (LSTM) deep neural networks. Our motivation stems from modeling specific mapping from synthesized to natural speech on those segments corresponding to voiced or unvoiced sounds, due to the different qualities of those sounds and how HMM-based voices can present distinct degradation on each one. The paper analyses the discriminative postfilters obtained using five voices, evaluated using three objective measures, Mel cepstral distance and subjective tests. The results indicate the advantages of the discriminative postilters in comparison with the HTS voice and the non-discriminative postfilters.