Convolutional Neural Network Long Short-term Memory Research Articles

Application of stacked and bidirectional long short-term memory deep learning models for wind speed forecasting at an offshore site

ABSTRACT Very short-term offshore wind speed forecasting by application of Stacked long short-term memory (LSTM) and Bidirectional LSTM deep learning models is done in this work. Wind speed data of two different offshore sites located in two different continents are used for testing the models. Performance is measured on the basis of accuracy of forecasting and computational time. The effectiveness of Stacked LSTM and Bidirectional LSTM models is also validated by comparing their performance with convolutional neural network, convolutional neural network-long short-term memory network, multi-layer perceptron, and rolling forecasting auto regressive integrated moving average models. Results of forecasting error confirm that Stacked LSTM model is better than other compared models in forecasting very short-term offshore wind speed. Mean absolute percentage error (MAPE) of wind speed forecasting by Stacked LSTM model is 4.59% at Anholt (Denmark) and 3.62% at Dhanushkodi (India) sites. From comparison of MAPE of Stacked LSTM model with that of eight other latest existing models in literature, it can be concluded that Stacked LSTM model is superior to many other existing models.

Forecasting stock price movement: new evidence from a novel hybrid deep learning model

PurposeThis study explores whether a new machine learning method can more accurately predict the movement of stock prices.Design/methodology/approachThis study presents a novel hybrid deep learning model, Residual-CNN-Seq2Seq (RCSNet), to predict the trend of stock price movement. RCSNet integrates the autoregressive integrated moving average (ARIMA) model, convolutional neural network (CNN) and the sequence-to-sequence (Seq2Seq) long–short-term memory (LSTM) model.FindingsThe hybrid model is able to forecast both linear and non-linear time-series component of stock dataset. CNN and Seq2Seq LSTMs can be effectively combined for dynamic modeling of short- and long-term-dependent patterns in non-linear time series forecast. Experimental results show that the proposed model outperforms baseline models on S&P 500 index stock dataset from January 2000 to August 2016.Originality/valueThis study develops the RCSNet hybrid model to tackle the challenge by combining both linear and non-linear models. New evidence has been obtained in predicting the movement of stock market prices.

A Hybrid Convolutional Neural Network-Long Short Term Memory for Discharge Capacity Estimation of Lithium-Ion Batteries

Abstract Predicting discharge capacities of lithium-ion batteries (LIBs) is essential for safe battery operation in electric vehicles (EVs). In this paper, a convolutional neural network-long short term memory (CNN-LSTM) approach is proposed to estimate the discharge capacity of LIBs. The parameters such as the voltage, current, temperature, and charge/discharge capacity are recorded from a battery management system (BMS) at various stages of the charge–discharge cycles. The experiments are conducted to obtain the data at different cycles, where each cycle is divided into four steps. Each testing cycle comprises charging, rest, discharging, and rest. In the predictive model, the initial layers are convolutional layers that help in feature extraction. Then, the long and short term memory layer is used to retain or forget related information. Finally, the prediction is completed by selecting the corresponding activation function. The evaluation model is established via the multiple train test split method. The lower values of weighted mean squared error suggest that discharge capacity estimation using CNN-LSTM is a reliable method. The CNN-LSTM approach can further be compiled in BMSs of EVs to get real-time status for state of charge and state of health values.

Two-Step Biometrics Using Electromyogram Signal Based on Convolutional Neural Network-Long Short-Term Memory Networks

Electromyogram (EMG) signals cannot be forged and have the advantage of being able to change the registered data as they are characterized by the waveform, which varies depending on the gesture. In this paper, a two-step biometrics method was proposed using EMG signals based on a convolutional neural network–long short-term memory (CNN-LSTM) network. After preprocessing of the EMG signals, the time domain features and LSTM network were used to examine whether the gesture matched, and single biometrics was performed if the gesture matched. In single biometrics, EMG signals were converted into a two-dimensional spectrogram, and training and classification were performed through the CNN-LSTM network. Data fusion of the gesture recognition and single biometrics was performed in the form of an AND. The experiment used Ninapro EMG signal data as the proposed two-step biometrics method, and the results showed 83.91% gesture recognition performance and 99.17% single biometrics performance. In addition, the false acceptance rate (FAR) was observed to have been reduced by 64.7% through data fusion.

A Hybrid CNN-LSTM: A Deep Learning Approach for Consumer Sentiment Analysis Using Qualitative User-Generated Contents

With the fastest growth of information and communication technology (ICT), the availability of web content on social media platforms is increasing day by day. Sentiment analysis from online reviews drawing researchers’ attention from various organizations such as academics, government, and private industries. Sentiment analysis has been a hot research topic in Machine Learning (ML) and Natural Language Processing (NLP). Currently, Deep Learning (DL) techniques are implemented in sentiment analysis to get excellent results. This study proposed a hybrid convolutional neural network-long short-term memory (CNN-LSTM) model for sentiment analysis. Our proposed model is being applied with dropout, max pooling, and batch normalization to get results. Experimental analysis carried out on Airlinequality and Twitter airline sentiment datasets. We employed the Keras word embedding approach, which converts texts into vectors of numeric values, where similar words have small vector distances between them. We calculated various parameters, such as accuracy, precision, recall, and F1-measure, to measure the model’s performance. These parameters for the proposed model are better than the classical ML models in sentiment analysis. Our results analysis demonstrates that the proposed model outperforms with 91.3% accuracy in sentiment analysis.

Hand tremor detection in videos with cluttered background using neural network based approaches.

With the increasing prevalence of neurodegenerative diseases, including Parkinson’s disease, hand tremor detection has become a popular research topic because it helps with the diagnosis and tracking of disease progression. Conventional hand tremor detection algorithms involved wearable sensors. A non-invasive hand tremor detection algorithm using videos as input is desirable but the existing video-based algorithms are sensitive to environmental conditions. An algorithm, with the capability of detecting hand tremor from videos with a cluttered background, would allow the videos recorded in a non-research environment to be used. Clinicians and researchers could use videos collected from patients and participants in their own home environment or standard clinical settings. Neural network based machine learning architectures provide high accuracy classification results in related fields including hand gesture recognition and body movement detection systems. We thus investigated the accuracy of advanced neural network architectures to automatically detect hand tremor in videos with a cluttered background. We examined configurations with different sets of features and neural network based classification models. We compared the performance of different combinations of features and classification models and then selected the combination which provided the highest accuracy of hand tremor detection. We used cross validation to test the accuracy of the trained model predictions. The highest classification accuracy for automatically detecting tremor (vs non tremor) was 80.6% and this was obtained using Convolutional Neural Network-Long Short-Term Memory and features based on measures of frequency and amplitude change.

DeepIPs: comprehensive assessment and computational identification of phosphorylation sites of SARS-CoV-2 infection using a deep learning-based approach

The rapid spread of SARS-CoV-2 infection around the globe has caused a massive health and socioeconomic crisis. Identification of phosphorylation sites is an important step for understanding the molecular mechanisms of SARS-CoV-2 infection and the changes within the host cells pathways. In this study, we present DeepIPs, a first specific deep-learning architecture to identify phosphorylation sites in host cells infected with SARS-CoV-2. DeepIPs consists of the most popular word embedding method and convolutional neural network-long short-term memory network architecture to make the final prediction. The independent test demonstrates that DeepIPs improves the prediction performance compared with other existing tools for general phosphorylation sites prediction. Based on the proposed model, a web-server called DeepIPs was established and is freely accessible at http://lin-group.cn/server/DeepIPs. The source code of DeepIPs is freely available at the repository https://github.com/linDing-group/DeepIPs.

Unified Deep Learning approach for Efficient Intrusion Detection System using Integrated Spatial–Temporal Features

Intrusion detection systems (IDS) differentiate the malicious entries from the legitimate entries in network traffic data and helps in securing the networks. Deep learning algorithms have been greatly employed in the network security field for large scale data in modern cyberspace networks because of their ability to learn the deeply integrated features. However, learning both space and time aspects of system information are very challenging for any individual deep knowledge model. While Convolutional Neural Networks (CNN) effectively acquires the spatial aspects, the Long Short-Term Memory (LSTM) neural networks perform better for temporal features. Integrating the benefits of these models has the potential for improving the large scale IDS. In this paper, a high accurate IDS model is proposed by using a unified model of Optimized CNN (OCNN) and Hierarchical Multi-scale LSTM (HMLSTM) for effective extraction and learning of spatial–temporal features. The proposed IDS model performs the pre-processing, feature extraction through network training and network testing and final classification. In the OCNN–HMLSTM model, the Lion Swarm Optimization (LSO) is used to tune the hyper-parameters of CNN for the optimal configuration of learning spatial features. The HMLSTM learns the hierarchical relationships between the different features and extracts the time features. Lastly, the unified IDS approach utilizes the extracted spatial–temporal features for categorizing the network data. Tests are performed over public IDS datasets namely NSL-KDD, ISCX-IDS and UNSWNB15. Assessing the performance of OCNN–HMLSTM​ against the contemporary IDS methods, the proposed model performs better intrusion detection with high accuracy of above 90% with less false values and better classification coefficients.

Long short term memory–convolutional neural network based deep hybrid approach for solar irradiance forecasting

The volatile behavior of solar energy is the biggest challenge in its successful integration with existing grid systems. Accurate global horizontal irradiance (GHI) forecasting can resolve this issue and lead to early and effective participation in the energy market. This study proposes a new hybrid deep learning model, namely long short term memory–convolutional neural network (LSTM–CNN), for hourly GHI forecasting, which models the spatio-temporal features by integrating the long short term memory (LSTM) and convolutional neural network (CNN) model. The proposed model is trained with the meteorological data of 23 locations of California State, USA, which includes temperature, precipitation, relative humidity, cloud cover, etc., as input parameters. The proposed hybrid LSTM–CNN model firstly uses LSTM to extract the temporal features from time-series solar irradiance data, followed by CNN, which extracts the spatial features from the correlation matrix of several meteorological variables of target and its neighbor location. The prediction accuracy of the developed model is analyzed rigorously by examining the performance for a year, for four seasons and under three sky conditions. Besides, the proposed LSTM–CNN model shows a forecast skill score in a range of about 37%–45% over few standalone models, including smart persistence, support vector machine, artificial neural network, LSTM, CNN and other hybrid models. The findings of the present work suggest that the proposed hybrid LSTM–CNN model is a reliable alternative for short-term GHI prediction due to its high predictive accuracy under diverse climatic, seasonal and sky conditions.

Predictive models for flame evolution using machine learning: A priori assessment in turbulent flames without and with mean shear

Accurate prediction of temporal evolution of turbulent flames represents one of the most challenging problems in the combustion community. In this work, predictive models for turbulent flame evolution were proposed based on machine learning with long short-term memory (LSTM) and convolutional neural network-long short-term memory (CNN-LSTM). Two configurations without and with mean shear are considered, i.e., turbulent freely propagating premixed combustion and turbulent boundary layer premixed combustion, respectively. The predictions of the LSTM and CNN-LSTM models were validated against the direct numerical simulation (DNS) data to assess the model performance. Particularly, the statistics of the fuel (CH4 for the freely propagating flames and H2 for the boundary layer flames) mass fraction and reaction rate were examined in detail. It was found that generally the performance of the CNN-LSTM model is better than that of the LSTM model. This is because that the CNN-LSTM model extracts both the spatial and temporal features of the flames while the LSTM model only extracts the temporal feature of the flames. The errors of the models mainly occur in regions with large scalar gradients. The correlation coefficient of the mass fraction from the DNS and that from the CNN-LSTM model is larger than 0.99 in various flames. The correlation coefficient of the reaction rate from the DNS and that from the CNN-LSTM model is larger than 0.93 in the freely propagating flames and 0.99 in the boundary layer flames. Finally, the profiles of the DNS values and predictions conditioned on axial distance were examined, and it was shown that the predictions of the CNN-LSTM model agree well with the DNS values. The LSTM model failed to accurately predict the evolution of boundary layer flames while the CNN-LSTM model could accurately predict the evolution of both freely propagating and boundary layer flames. Overall, this study shows the promising performance and the applicability of the proposed CNN-LSTM model, which will be applied to turbulent flames a posteriori in future work.

An Effective Pomegranate Fruit Classification Based On CNN-LSTM Deep Learning Models

Objectives: To employ a deep learning technique that would sort the fruits into normal and abnormal based on the features such as fruit colour, number of fruit spots, and shape of the fruit. Methods: A combined CNN LSTM deep learning model is applied to classify a set of 6519 fruits into two classes namely normal and abnormal. The dataset is an excel file with each record consisting of 9 features. Convolutional Neural Networks (CNN) are applied for deep feature extraction and Long-Short Term Memory (LSTM) is used to detect the class based on extracted features. Findings: The proposed system achieved an accuracy of 98.17%, specificity of 98.65%, sensitivity of 97.77%, and an F1-score of 98.39%. Novelty: The sensitivity of disease detection was less with lesser availability of enhanced detection methods for detecting disease in earlier stages. The issue with these various existing algorithms is that the accuracy was reduced since some sources of errors were not eliminated. Deep Learning delivers methodologies, approaches, and functionalities that can help to resolve analytic and predictive analysis accurately. Keywords: Deep Learning; CNN LSTM; Classification; Hyperparameters

Comparative study of machine learning methods for COVID-19 transmission forecasting

Within the recent pandemic, scientists and clinicians are engaged in seeking new technology to stop or slow down the COVID-19 pandemic. The benefit of machine learning, as an essential aspect of artificial intelligence, on past epidemics offers a new line to tackle the novel Coronavirus outbreak. Accurate short-term forecasting of COVID-19 spread plays an essential role in improving the management of the overcrowding problem in hospitals and enables appropriate optimization of the available resources (i.e., materials and staff).This paper presents a comparative study of machine learning methods for COVID-19 transmission forecasting. We investigated the performances of deep learning methods, including the hybrid convolutional neural networks-Long short-term memory (LSTM-CNN), the hybrid gated recurrent unit-convolutional neural networks (GAN-GRU), GAN, CNN, LSTM, and Restricted Boltzmann Machine (RBM), as well as baseline machine learning methods, namely logistic regression (LR) and support vector regression (SVR). The employment of hybrid models (i.e., LSTM-CNN and GAN-GRU) is expected to eventually improve the forecasting accuracy of COVID-19 future trends. The performance of the investigated deep learning and machine learning models was tested using confirmed and recovered COVID-19 cases time-series data from seven impacted countries: Brazil, France, India, Mexico, Russia, Saudi Arabia, and the US. The results reveal that hybrid deep learning models can efficiently forecast COVID-19 cases. Also, results confirmed the superior performance of deep learning models compared to the two considered baseline machine learning models. Furthermore, results showed that LSTM-CNN achieved improved performances with an averaged mean absolute percentage error of 3.718%, among others.

LSTM-CNN network for human activity recognition using WiFi CSI data

Human Activity Recognition (HAR) has had a diverse range of applications in various fields such as health, security and smart homes. Among different approaches of HAR, WiFi-based solutions are getting popular since it solves the problem of deployment cost, privacy concerns and restriction of the applicable environment. In this paper, we propose a WiFi-based human activity recognition system that can identify different activities via the channel state information from WiFi devices. A special deep learning framework, Long Short-Term Memory-Convolutional Neural Network (LSTM-CNN), is designed for accurate recognition. LSTM-CNN is going to be compared with the LSTM network and the experimental results demonstrate that LSTM-CNN outperforms existing models and has an average accuracy of 94.14% in multi-activity classification.

A Hybrid CNN-LSTM Model for Psychopathic Class Detection from Tweeter Users

In today’s digital era, the use of online social media networks, such as Google, YouTube, Facebook, and Twitter, permits people to generate a massive amount of textual content. The textual content that is produced by people reveals essential information regarding their personality, with psychopathy being among these distinct personality types. This work was aimed at classifying input texts according to the traits of psychopaths and non-psychopaths. Several studies based on traditional techniques, such as the SRPIII technique, using small-sized datasets have been conducted for the detection of psychopathic behavior. However, the purpose of the current study was to build an effective computational model for the detection of psychopaths in the domain of text analytics and computational intelligence. This study was aimed at developing a technique based on a convolutional neural network + long short-term memory (CNN-LSTM) model by using a deep learning approach to detect psychopaths. A convolutional neural network was used to extract local information from a text, while the long short-term memory was used to extract the contextual dependencies of the text. By combining the advantages of convolutional neural network and long short-term memory, the proposed hybrid CNN-LSTM was able to yield a good classification accuracy of 91.67%. Additionally, a large-sized benchmark dataset was acquired for the effective classification of the given input text into psychopath vs. non-psychopath classes, thereby enabling persons with such personality traits to be identified.

Hybrid deep neural model for hourly solar irradiance forecasting

Owing to integrating photovoltaic solar systems into power networks, accurate prediction of solar irradiance plays an increasingly significant role in electric energy planning and management. However, the existing hybrid models ignore the influence of other factors except for the irradiance time series and adopt a single branch independent network structure, which may lead to decrease prediction accuracy. In this paper, a novel multivariate hybrid deep neural model named WPD–CNN–LSTM-MLP for 1-h-ahead solar irradiance forecasting is proposed. The novel WPD–CNN–LSTM-MLP model is based on a sophisticated multi-branch hybrid structure with multi-variable inputs, which the multi-branch hybrid structure combines wavelet packet decomposition (WPD), convolutional neural network (CNN), long short-term memory (LSTM) networks, and multi-layer perceptron network (MLP), and the multi-variable inputs include hourly solar irradiance and three climate variables, namely: temperature, relative humidity, and wind speed and their combination. The new model extracts the inherent characteristics of multi-layer inputs sufficiently, overcomes the shortcomings of traditional models, and achieves more accurate forecasting results. The performance of the model is verified by actual data from Denver, Clark, and Folsom, the United States. Comparative studies of traditional individual back propagation neural network, support vector machine, recurrent neural network, LSTM, the climatology–persistence reference forecasts method and the proposed LSTM-MLP model, CNN-LSTM-MLP model, and WPD–CNN–LSTM model reveal that the proposed WPD–CNN–LSTM-MLP deep learning model has better prediction accuracy in hourly irradiance forecasting.

DeeReCT-APA: Prediction of Alternative Polyadenylation Site Usage Through Deep Learning

Alternative polyadenylation (APA) is a crucial step in post-transcriptional regulation. Previous bioinformatic studies have mainly focused on the recognition of polyadenylation sites (PASs) in a given genomic sequence, which is a binary classification problem. Recently, computational methods for predicting the usage level of alternative PASs in the same gene have been proposed. However, all of them cast the problem as a non-quantitative pairwise comparison task and do not take the competition among multiple PASs into account. To address this, here we propose a deep learning architecture, Deep Regulatory Code and Tools for Alternative Polyadenylation (DeeReCT-APA), to quantitatively predict the usage of all alternative PASs of a given gene. To accommodate different genes with potentially different numbers of PASs, DeeReCT-APA treats the problem as a regression task with a variable-length target. Based on a convolutional neural network-long short-term memory (CNN-LSTM) architecture, DeeReCT-APA extracts sequence features with CNN layers, uses bidirectional LSTM to explicitly model the interactions among competing PASs, and outputs percentage scores representing the usage levels of all PASs of a gene. In addition to the fact that only our method can quantitatively predict the usage of all the PASs within a gene, we show that our method consistently outperforms other existing methods on three different tasks for which they are trained: pairwise comparison task, highest usage prediction task, and ranking task. Finally, we demonstrate that our method can be used to predict the effect of genetic variations on APA patterns and sheds light on future mechanistic understanding in APA regulation. Our code and data are available at https://github.com/lzx325/DeeReCT-APA-repo.

Ultra‐short‐term multi‐step wind power forecasting based on CNN‐LSTM

The fluctuation and intermission of large-scale wind power integration is a serious threat to the stability and security of the power system. Accurate prediction of wind power is of great significance to the safety of wind power grid connection. This study proposes a novel spatio-temporal correlation model (STCM) for ultra-short-term wind power prediction based on convolutional neural networks-long short-term memory (CNN-LSTM). The original meteorological factors at multi-historical time points of different sites throughout the target wind farm can be reconstructed into the input window of the model, and thus a new data reconstruction method is represented. CNN is used to extract the spatial correlation feature vectors of meteorological factors of different sites and the temporal correlation vectors of the meteorological features in ultra-short term, which are reconstructed in time series and used as the input data of LSTM. Then, LSTM extracts the temporal feature relationship between the historical time points for multi-step wind power forecasting. The STCM based on CNN-LSTM proposed in this study is suitable for wind farms that can collect meteorological factors at different locations. Taking the measured meteorological factors and wind power dataset of a wind farm in China as an example, four evaluation metrics of the CNN-LSTM model, CNN and LSTM individually used for multi-step wind power prediction, are obtained. The results show that the proposed STCM based on CNN-LSTM has better spatial and temporal characteristics extraction ability than the traditional structure model and can forecast the power of wind farm more accurately.

Deep learning-based models for temporal satellite data processing: Classification of paddy transplanted fields

Deep learning-based frameworks have not been much explored to incorporate the temporal dimension of the remote sensing data. In this research work, deep learning-based models have been developed which exploit the spectral-temporal domain of bi-sensor remote sensing data obtained from Sentinel-2 and Landsat-8 satellites. Application of proposed deep learning frameworks has been tested for mapping transplanted paddy fields using Class Based Sensor Independent - Normalised Difference Vegetation Index (CBSI-NDVI) spectral reduction approach. Two deep learning-based models viz., one-dimensional Convolutional Neural Network (1-D CNN) and hybrid 1-D Convolutional Neural Network – Long Short Term Memory (CNN-LSTM) consisting of 1-D CNN and LSTM layers respectively were developed. For both of the models the optimized model hyper-parameters were also deduced. Though the performance of both the models were above satisfactory level still the 1-D CNN model performed slightly better than the hybrid 1-D CNN-LSTM based model with average overall accuracies of 93.75% and 91.25% respectively. The average F-measure values were computed as 0.93 and 0.90 respectively for the 1-D CNN based and hybrid CNN-LSTM based models. This study indicates that 1-D CNN based deep learning models provide an effective solution to handle mono/bi-sensor temporal remote sensing data of medium spatial resolution with small size training datasets.

Advertising Click-Through Rate Prediction Based on CNN-LSTM Neural Network.

In the era of big data information, how to effectively predict and analyze the click-through rate of information advertising is the key for enterprises in various fields to seek returns. The point rate prediction of advertising is one of the core contents of advertising calculation. The traditional shallow prediction model cannot meet the nonlinear relationship of data processing, and the manual processing of data information extraction method is very resource consuming. To solve the above problems, this paper proposes a CNN-LSTM (convolutional neural network-long short-term memory) convolution hybrid neural network algorithm to predict the click-through rate of advertisements. According to the neural network algorithm, the prediction model is constructed, and the effective features are extracted in the process of model establishment, and the prediction analysis is carried out according to the simplified LSTM neural network time serialization features. CNN convolution neural network is used to train the prediction model. This paper analyzes the characteristics of traditional prediction methods and the corresponding solutions and carries out feature learning and prediction model construction for advertising click-through rate prediction. Then, the unknown behavior of advertising users is judged and predicted. The results show that, compared with the single structure network of traditional prediction model, the prediction effect based on CNN-LSTM neural network algorithm has higher accuracy.

Abnormal Detection of Electricity Consumption of User Based on Particle Swarm Optimization and Long Short Term Memory With the Attention Mechanism

In the process of power transmission and distribution, non-technical losses are usually caused by users' abnormal power consumption behavior. It will not only affect the dispatch and operation of the distribution network, bring hidden dangers to the security of the power grid, but also damage the operating costs of power companies and disrupt the operation of the power market. Aiming at users' abnormal electricity consumption behavior, this paper proposes a model based on particle swarm optimization and long-short term memory with the attention mechanism (PSO-Attention-LSTM). Firstly, according to the actual electricity theft behavior, six typical electricity theft modes are summarized, and 4 composite modes are obtained by combining them, so as to comprehensively test the detection performance of the model for various electricity theft behaviors. Secondly, a detection model based on PSO-Attention-LSTM is proposed, and the model is built using the TensorFlow framework. The model uses the attention mechanism to give different weights to the hidden state of LSTM, which reduces the loss of historical information, strengthens important information and suppresses useless information. Use PSO to solve the difficult problem of model parameter selection, and optimize the hyperparameters to improve the model performance. Finally, the data set of the University of Massachusetts was used for simulation and compared with convolutional neural network-long short term memory (CNN-LSTM), attention mechanism-based long short term memory (Attention-LSTM), LSTM, gated recurrent unit (GRU), support vector regression (SVR), random forest (RF) and linear regression (LR) to verify the effectiveness and accuracy of the method used in this article. In this paper, Matlab software is used to analyze and visualize the detection result data.