Long Short-term Memory Architecture Research Articles

LSTM network as a screening tool to detect moderate traumatic brain injury from resting-state electroencephalogram

Fraud and abuse in insurance and health care claims have received major attention as they can cause increasing losses of revenue. Processing health care claims requires extensive workloads because the staffs have to investigate the legibility of the report. For the investigation of brain injury claims, the related insurance company may request medical images of the brain from the hospital and subsequently get opinions from the medical staff. Conventionally, computed tomography (CT) or magnetic resonance imaging (MRI) is utilized for this purpose. However, to perform a CT scan or an MRI scan for every patient that requested medical claims is impractical due to the limited resources. Thus, we proposed a screening approach that uses the resting-state electroencephalogram (EEG) recordings as the input to a long short-term memory (LSTM) network. This LSTM architecture can classify the resting state EEG into two classes, which are either as a moderate traumatic brain injury (TBI) patient or a healthy person. Experimental results show that the proposed approach is able to outperform two similar recent works by achieving a classification accuracy of 74.33%.

Evolving Long Short-Term Memory Network-Based Text Classification.

Recently, long short-term memory (LSTM) networks are extensively utilized for text classification. Compared to feed-forward neural networks, it has feedback connections, and thus, it has the ability to learn long-term dependencies. However, the LSTM networks suffer from the parameter tuning problem. Generally, initial and control parameters of LSTM are selected on a trial and error basis. Therefore, in this paper, an evolving LSTM (ELSTM) network is proposed. A multiobjective genetic algorithm (MOGA) is used to optimize the architecture and weights of LSTM. The proposed model is tested on a well-known factory reports dataset. Extensive analyses are performed to evaluate the performance of the proposed ELSTM network. From the comparative analysis, it is found that the LSTM network outperforms the competitive models.

Deep Learning for Vegetation Health Forecasting: A Case Study in Kenya

East Africa has experienced a number of devastating droughts in recent decades, including the 2010/2011 drought. The National Drought Management Authority in Kenya relies on real-time information from MODIS satellites to monitor and respond to emerging drought conditions in the arid and semi-arid lands of Kenya. Providing accurate and timely information on vegetation conditions and health—and its probable near-term future evolution—is essential for minimising the risk of drought conditions evolving into disasters as the country’s herders directly rely on the conditions of grasslands. Methods from the field of machine learning are increasingly being used in hydrology, meteorology, and climatology. One particular method that has shown promise for rainfall-runoff modelling is the Long Short Term Memory (LSTM) network. In this study, we seek to test two LSTM architectures for vegetation health forecasting. We find that these models provide sufficiently accurate forecasts to be useful for drought monitoring and forecasting purposes, showing competitive performances with lower resolution ensemble methods and improved performances over a shallow neural network and a persistence baseline.

Symmetrically Stacked Long Short-Term Memory Networks for Fall Event Recognition Using Compact Convolutional Neural Networks-Based Tracker

In recent years, the advancement of pattern recognition algorithms, specifically the deep learning-related techniques, have propelled a tremendous amount of researches in fall event recognition systems. It is important to detect a fall incident as early as possible, whereby a slight delay in providing immediate assistance can cause severe unrecoverable injuries. One of the main challenges in fall event recognition is the imbalanced training data between fall and no-fall events, where a real-life fall incident is a sporadic event that occurs infrequently. Most of the recent techniques produce a lot of false alarms, as it is hard to train them to cover a wide range of fall situations. Hence, this paper aims to detect the exact fall frame in a video sequence, as such it will not be dependent on the whole clip of the video sequence. Our proposed approach consists of a two-stage module where the first stage employs a compact convolutional neural network tracker to generate the object trajectory information. Features of interest will be sampled from the generated trajectory paths, which will be fed as the input to the second stage. The next stage network then models the temporal dependencies of the trajectory information using symmetrical Long Short-Term Memory (LSTM) architecture. This two-stage module is a novel approach as most of the techniques rely on the detection module rather than the tracking module. The simulation experiments were tested using Fall Detection Dataset (FDD). The proposed approach obtains an expected average overlap of 0.167, which is the best performance compared to Multi-Domain Network (MDNET) and Tree-structured Convolutional Neural Network (TCNN) trackers. Furthermore, the proposed 3-layers of stacked LSTM architecture also performs the best compared to the vanilla recurrent neural network and single-layer LSTM. This approach can be further improved if the tracker model is firstly pre-tuned in offline mode with respect to a specific type of object of interest, rather than a general object.

Predicting Discharges in Sewer Pipes Using an Integrated Long Short-Term Memory and Entropy A-TOPSIS Modeling Framework

Predicting discharges in sewage systems play an essential role in reducing sewer overflows and impacts on the environment and public health. Choosing a suitable model to predict discharges in these systems is essential to realizing these aforementioned goals. Long Short-Term Memory (LSTM) has been proposed as a robust technique for predicting discharges in wastewater networks. This study explored the potential application of an LSTM model to predict discharges using 3-month data set in a sewer network in Ålesund city, Norway. Different sequence-to-sequence LSTMs were investigated using various input and output datasets. The impact of data aggregation (10-min and 30-min intervals) was examined and compared to original sensor data (5-min intervals) to evaluate the performance of the LSTM model. The results show that 50-neuron LSTM architecture performed better (MAPE = 0.09, RMSE = 0.0008, R2 = 0.8) in predicting discharges for the study area. The study indicates that using the same sequence length for the prior and the forecast can improve the effectiveness of the LSTM model. Based on the results, using a 10-min aggregated discharge dataset reduces energy consumption, transmission bandwidth, and storage capacity. Additionally, it improves prediction performance compared to an original 5-min interval data in Ålesund city.

Sentinel-1 Spatiotemporal Simulation Using Convolutional LSTM for Flood Mapping

The synthetic aperture radar (SAR) imagery has been widely applied for flooding mapping based on change detection approaches. However, errors in the mapping result are expected since not all land-cover changes are flood-induced, and those changes are sensitive to SAR data, such as crop growth or harvest over agricultural lands, clearance of forested areas, and/or modifications on the urban landscape. This study, therefore, incorporated historical SAR images to boost the detection of flood-induced changes during extreme weather events, using the Long Short-Term Memory (LSTM) method. Additionally, to incorporate the spatial signatures for the change detection, we applied a deep learning-based spatiotemporal simulation framework, Convolutional Long Short-Term Memory (ConvLSTM), for simulating a synthetic image using Sentinel One intensity time series. This synthetic image will be prepared in advance of flood events, and then it can be used to detect flood areas using change detection when the post-image is available. Practically, significant divergence between the synthetic image and post-image is expected over inundated zones, which can be mapped by applying thresholds to the Delta image (synthetic image minus post-image). We trained and tested our model on three events from Australia, Brazil, and Mozambique. The generated Flood Proxy Maps were compared against reference data derived from Sentinel Two and Planet Labs optical data. To corroborate the effectiveness of the proposed methods, we also generated Delta products for two baseline models (closest post-image minus pre-image and historical mean minus post-image) and two LSTM architectures: normal LSTM and ConvLSTM. Results show that thresholding of ConvLSTM Delta yielded the highest Cohen’s Kappa coefficients in all study cases: 0.92 for Australia, 0.78 for Mozambique, and 0.68 for Brazil. Lower Kappa values obtained in the Mozambique case can be subject to the topographic effect on SAR imagery. These results still confirm the benefits in terms of classification accuracy that convolutional operations provide in time series analysis of satellite data employing spatially correlated information in a deep learning framework.

The Best Techniques to Deal with Unbalanced Sequential Text Data in Deep Learning

Datasets with a balanced distribution of data are often difficult to find in real life. Although various methods have been developed and proven successful using shallow learning algorithms, handling unbalanced classes using a deep learning approach is still limited. Most of these studies only focus on image data using the Convolution Neural Network (CNN) architecture. In this study, we tried to apply several class handling techniques to three datasets of unbalanced text data. Both use a data-level approach with resampling techniques on word vectors and algorithm-level using Weighted Cross-Entropy Loss (WCEL) to handle cases of imbalanced text classification. With Bidirectional Long-Short Term Memory (BiLSTM) architecture. We tested each method using three datasets with different characteristics and levels of imbalance. Based on the experiments that have been carried out, each technique applied has a different performance on each dataset.

Single-Channel Speech Separation Using Soft-Minimum Permutation Invariant Training

The goal of speech separation is to extract multiple speech sources from a single microphone recording. Recently, with the advancement of deep learning and availability of large datasets, speech separation has been formulated as a supervised learning problem. These approaches aim to learn discriminative patterns of speech, speakers, and background noise using a supervised learning algorithm, typically a deep neural network. A long-lasting problem in supervised speech separation is finding the correct label for each separated speech signal, referred to as label permutation ambiguity. Permutation ambiguity refers to the problem of determining the output-label assignment between the separated sources and the available single-speaker speech labels. Finding the best output-label assignment is required for calculation of separation error, which is later used for updating parameters of the model. Recently, Permutation Invariant Training (PIT) has been shown to be a promising solution in handling the label ambiguity problem. However, the overconfident choice of the output-label assignment by PIT results in a sub-optimal trained model. In this work, we propose a probabilistic optimization framework to address the inefficiency of PIT in finding the best output-label assignment. Our proposed method entitled trainable Soft-minimum PIT is then employed on the same Long-Short Term Memory (LSTM) architecture used in Permutation Invariant Training (PIT) speech separation method. The results of our experiments show that the proposed method outperforms conventional PIT speech separation significantly (p-value $ < 0.01$) by +1dB in Signal to Distortion Ratio (SDR) and +1.5dB in Signal to Interference Ratio (SIR).

Prediction of malaria using deep learning models: A case study on city clusters in the state of Amazonas, Brazil, from 2003 to 2018.

ABSTRACTBackground: Malaria is curable. Nonetheless, over 229 million cases of malaria were recorded in 2019, along with 409,000 deaths. Although over 42 million Brazilians are at risk of contracting malaria, 99% percent of all malaria cases in Brazil are located in or around the Amazon rainforest. Despite declining cases and deaths, malaria remains a major public health issue in Brazil. Accurate spatiotemporal prediction of malaria propagation may enable improved resource allocation to support efforts to eradicate the disease. Methods: In response to calls for novel research on malaria elimination strategies that suit local conditions, in this study, we propose machine learning (ML) and deep learning (DL) models to predict the probability of malaria cases in the state of Amazonas. Using a dataset of approximately 6 million records (January 2003 to December 2018), we applied k-means clustering to group cities based on their similarity of malaria incidence. We evaluated random forest, long-short term memory (LSTM) and dated recurrent unit (GRU) models and compared their performance. Results: The LSTM architecture achieved better performance in clusters with less variability in the number of cases, whereas the GRU presents better results in clusters with high variability. Although Diebold-Mariano testing suggested that both the LSTM and GRU performed comparably, GRU can be trained significantly faster, which could prove advantageous in practice. Conclusions: All models showed satisfactory accuracy and strong performance in predicting new cases of malaria, and each could serve as a supplemental tool to support regional policies and strategies.

Algorithm design for wind prediction in Berakit Bay, Bintan Island using Long Short-Term Memory (LSTM) method

Wind speed is a crucial parameter alongside coastal areas, especially Indonesia. Above average wind speed can cause harmful effects on human activities. This study uses wind speed data from Berakit Bay, Bintan Island is a potential location for coastal community settlement, fisheries, and tourist activities. The wind parameter then predicted using the Long Short-Term Memory or LSTM algorithm. This algorithm is able to study long-term dependencies by converting simple nervous system designs into specialized blocks containing cells. It is suitable to be applied to long-term wind predictions where the wind speed at this time is very influential with the wind speed in the future. In preparing the LSTM, the data preprocessing and the architecture used will determine the prediction results. In this study, four different architectures were made in order to determine the most optimal architecture. The results show that the LSTM architecture is able to obtain a relatively good RMSE value of 1.87 and an accuracy of 39.40% with the use of two LSTM layers, 256 units in the first layer and 128 in the second layer. The LSTM algorithm in predicting wind can also be applied to other areas in Indonesia.

Deep Learning for seasonality modelling in Inflation-Indexed Swap pricing

An Inflation-Indexed Swap (IIS) is a derivative in which, at every payment date, the counterparties swap an inflation rate with a fixed rate. For the calculation of the Inflation Leg cash flows it is necessary to build a mathematical model suitable for the Consumer Price Index (CPI) projection. For this purpose, quants typically start by using market quotes for the Zero-Coupon swaps in order to derive the future trend of the inflation index, together with a seasonality model for capturing the typical periodical effects. In this study, we propose a forecasting model for inflation seasonality based on a Long Short Term Memory (LSTM) network: a deep learning methodology particularly useful for forecasting purposes. The CPI predictions are conducted using a FinTech paradigm, but in respect of the traditional quantitative finance theory developed in this research field. The paper is structured according to the following sections: the first two parts illustrate the pricing methodologies for the most popular IIS: the Zero Coupon Inflation-Indexed Swap (ZCIIS) and the Year-on-Year Inflation-Indexed Swap (YYIIS); section 3 deals with the traditional standard method for the forecast of CPI values (trend + seasonality), while section 4 describes the LSTM architecture, and section 5 focuses on CPI projections, also called inflation bootstrap. Then section 6 describes a robust check, implementing a traditional SARIMA model in order to improve the interpretation of the LSTM outputs; finally, section 7 concludes with a real market case, where the two methodologies are used for computing the fair-value for a YYIIS and the model risk is quantified.

Short-Term Forecasting of Wind Energy: A Comparison of Deep Learning Frameworks

Wind energy has been recognized as the most promising and economical renewable energy source, attracting increasing attention in recent years. However, considering the variability and uncertainty of wind energy, accurate forecasting is crucial to propel high levels of wind energy penetration within electricity markets. In this paper, a comparative framework is proposed where a suite of long short-term memory (LSTM) recurrent neural networks (RNN) models, inclusive of standard, bidirectional, stacked, convolutional, and autoencoder architectures, are implemented to address the existing gaps and limitations of reported wind power forecasting methodologies. These integrated networks are implemented through an iterative process of varying hyperparameters to better assess their effect, and the overall performance of each architecture, when tackling one-hour to three-hours ahead wind power forecasting. The corresponding validation is carried out through hourly wind power data from the Spanish electricity market, collected between 2014 and 2020. The proposed comparative error analysis shows that, overall, the models tend to showcase low error variability and better performance when the networks are able to learn in weekly sequences. The model with the best performance in forecasting one-hour ahead wind power is the stacked LSTM, implemented with weekly learning input sequences, with an average MAPE improvement of roughly 6, 7, and 49%, when compared to standard, bidirectional, and convolutional LSTM models, respectively. In the case of two to three-hours ahead forecasting, the model with the best overall performance is the bidirectional LSTM implemented with weekly learning input sequences, showcasing an average improved MAPE performance from 2 to 23% when compared to the other LSTM architectures implemented.

Deep learning models to predict flood events in fast-flowing watersheds

This study aims to explore the reliability of flood warning forecasts based on deep learning models, in particular Long-Short Term Memory (LSTM) architecture. We also wish to verify the applicability of flood event predictions for a river with flood events lasting only a few hours, with the aid of hydrometric control stations. This methodology allows for the creation of a system able to identify flood events with acceptable errors within several hours' notice. In terms of errors, the results obtained in this study can be compared to those obtained by using physics-based models for the same study area. These kinds of models use few types of data, unlike physical models that require the estimation of several parameters. However, the deep learning models are data-driven and for this reason they can influence the results obtained. Therefore, we tested the stability of the models by simulating the missing or wrong input data of the model, and this allowed us to achieve excellent results. Indeed, the models were stable even if several data were missing. This method makes it possible to lay the foundations for the future application of these techniques when there is an absence of geological-hydrogeological information preventing physical modeling of the run-off process or in cases of relatively small basins, where the complex system and the unsatisfactory modeling of the phenomenon do not allow a correct application of physical-based models. The forecast of flood events is fundamental for correct and adequate territory management, in particular when significant climatic changes occur. The study area is that of the Arno River (in Tuscany, Italy), which crosses some of the most important cities of central Italy, in terms of population, cultural heritage, and socio-economic activities.

Benchmarking data-driven rainfall–runoff models in Great Britain: a comparison of long short-term memory (LSTM)-based models with four lumped conceptual models

Abstract. Long short-term memory (LSTM) models are recurrent neural networks from the field of deep learning (DL) which have shown promise for time series modelling, especially in conditions when data are abundant. Previous studies have demonstrated the applicability of LSTM-based models for rainfall–runoff modelling; however, LSTMs have not been tested on catchments in Great Britain (GB). Moreover, opportunities exist to use spatial and seasonal patterns in model performances to improve our understanding of hydrological processes and to examine the advantages and disadvantages of LSTM-based models for hydrological simulation. By training two LSTM architectures across a large sample of 669 catchments in GB, we demonstrate that the LSTM and the Entity Aware LSTM (EA LSTM) models simulate discharge with median Nash–Sutcliffe efficiency (NSE) scores of 0.88 and 0.86 respectively. We find that the LSTM-based models outperform a suite of benchmark conceptual models, suggesting an opportunity to use additional data to refine conceptual models. In summary, the LSTM-based models show the largest performance improvements in the north-east of Scotland and in south-east of England. The south-east of England remained difficult to model, however, in part due to the inability of the LSTMs configured in this study to learn groundwater processes, human abstractions and complex percolation properties from the hydro-meteorological variables typically employed for hydrological modelling.

A deep learning approach for efficient topology optimization based on the element removal strategy

In this paper, a deep learning-based model is proposed which is capable of automatically generating the structural topology configurations with the minimum structural compliance and deformation under various load conditions and volume fraction limitations. The deep-learning model combines the advanced algorithms of Convolutional Neural Network (CNN) with U-net architecture and Recurrent Neural Network (RNN) with Long-Short Term Memory (LSTM) architecture. The established data-driven framework learns the structural evolution process from training data samples, which are randomly generated in finite element simulations employing an element removal strategy. The well-trained model is successfully utilized for two types of cases: two-dimensional and three-dimensional cantilever-beam structural topology designs. The deep-learning model outperforms the traditional methods in terms of lower time cost and broader applicability, demonstrating the potential of such a data-driven approach to accelerate the process of preliminary structural design.

CNN-based bi-directional and directional long-short term memory network for determination of face mask

ContextThe COVID-19 virus, exactly like in numerous other diseases, can be contaminated from person to person by inhalation. In order to prevent the spread of this virus, which led to a pandemic around the world, a series of rules have been set by governments that people must follow. The obligation to use face masks, especially in public spaces, is one of these rules. ObjectiveThe aim of this study is to determine whether people are wearing the face mask correctly by using deep learning methods. MethodsA dataset consisting of 2000 images was created. In the dataset, images of a person from three different angles were collected in four classes, which are “masked”, “non-masked”, “masked but nose open”, and “masked but under the chin”. Using this data, new models are proposed by transferring the learning through AlexNet and VGG16, which are the Convolutional Neural network architectures. Classification layers of these models were removed and, Long-Short Term Memory and Bi-directional Long-Short Term Memory architectures were added instead. Result and conclusionsAlthough there are four different classes to determine whether the face masks are used correctly, in the six models proposed, high success rates have been achieved. Among all models, the TrVGG16 + BiLSTM model has achieved the highest classification accuracy with 95.67%. SignificanceThe study has proven that it can take advantage of the proposed models in conjunction with transfer learning to ensure the proper and effective use of the face mask, considering the benefit of society.

End-to-end multimodal clinical depression recognition using deep neural networks: A comparative analysis

Background and Objective: Major Depressive Disorder is a highly prevalent and disabling mental health condition. Numerous studies explored multimodal fusion systems combining visual, audio, and textual features via deep learning architectures for clinical depression recognition. Yet, no comparative analysis for multimodal depression analysis has been proposed in the literature. Methods: In this paper, an up-to-date literature overview of multimodal depression recognition is presented and an extensive comparative analysis of different deep learning architectures for depression recognition is performed. First, audio features based Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) are studied. Then, early-level and model-level fusion of deep audio features with visual and textual features through LSTM and CNN architectures are investigated. Results: The performance of the proposed architectures using an hold-out strategy on the DAIC-WOZ dataset (80% training, 10% validation, 10% test split) for binary and severity levels of depression recognition is tested. Using this strategy, a set of experiments have been performed and they have demonstrated: (1) LSTM-based audio features perform slightly better than CNN ones with an accuracy of 66.25% versus 65.60% for binary depression classes. (2) the model level fusion of deep audio and visual features using LSTM network performed the best with an accuracy of 77.16%, a precision of 53% for the depressed class, and a precision of 83% for the non-depressed class. The given network obtained a normalized Root Mean Square Error (RMSE) of 0.15 for depression severity level prediction. Using a Leave-One-Subject-Out strategy, this network achieved an accuracy of 95.38% for binary depression detection, and a normalized RMSE of 0.1476 for depression severity level prediction. Our best-performing architecture outperforms all state-of-the-art approaches on DAIC-WOZ dataset. Conclusions: The obtained results show that the proposed LSTM-based surpass the proposed CNN-based architectures allowing to learn temporal dynamics representations of multimodal features. Furthermore, model-level fusion of audio and visual features using an LSTM network leads to the best performance. Our best-performing architecture successfully detects depression using a speech segment of less than 8 seconds, and an average prediction computation time of less than 6ms; making it suitable for real-world clinical applications.

An Auditory Saliency Pooling-Based LSTM Model for Speech Intelligibility Classification

Speech intelligibility is a crucial element in oral communication that can be influenced by multiple elements, such as noise, channel characteristics, or speech disorders. In this paper, we address the task of speech intelligibility classification (SIC) in this last circumstance. Taking our previous works, a SIC system based on an attentional long short-term memory (LSTM) network, as a starting point, we deal with the problem of the inadequate learning of the attention weights due to training data scarcity. For overcoming this issue, the main contribution of this paper is a novel type of weighted pooling (WP) mechanism, called saliency pooling where the WP weights are not automatically learned during the training process of the network, but are obtained from an external source of information, the Kalinli’s auditory saliency model. In this way, it is intended to take advantage of the apparent symmetry between the human auditory attention mechanism and the attentional models integrated into deep learning networks. The developed systems are assessed on the UA-speech dataset that comprises speech uttered by subjects with several dysarthria levels. Results show that all the systems with saliency pooling significantly outperform a reference support vector machine (SVM)-based system and LSTM-based systems with mean pooling and attention pooling, suggesting that Kalinli’s saliency can be successfully incorporated into the LSTM architecture as an external cue for the estimation of the speech intelligibility level.

Intelligent Islanding Detection of Microgrids Using Long Short-Term Memory Networks

This paper presents a new intelligent islanding detection scheme (IIDS) based on empirical wavelet transform (EWT) and long short-term memory (LSTM) network to identify islanding events in microgrids. The concept of EWT is extended to extract features from three-phase signals. First, the three-phase voltage signals sampled at the terminal of targeted distributed energy resource (DER) or point of common coupling (PCC) are decomposed into empirical modes/frequency subbands using EWT. Then, instantaneous amplitudes and instantaneous frequencies of the three-phases at different frequency subbands are combined, and various statistical features are calculated. Finally, the EWT-based features along with the three-phase voltage signals are input to the LSTM network to differentiate between non-islanding and islanding events. To assess the efficacy of the proposed IIDS, extensive simulations are performed on an IEC microgrid and an IEEE 34-node system. The simulation results verify the effectiveness of the proposed IIDS in terms of non-detection zone (NDZ), computational time, detection accuracy, and robustness against noisy measurement. Furthermore, comparisons with existing intelligent methods and different LSTM architectures demonstrate that the proposed IIDS offers higher reliability by significantly reducing the NDZ and stands robust against measurements uncertainty.

Deep Learning Method for Martian Atmosphere Reconstruction

The reconstruction of atmospheric properties encountered during Mars entry trajectories is a crucial element of postflight mission analysis. This paper proposes a deep learning architecture using a long short-term memory (LSTM) network for the reconstruction of Martian density and wind profiles from inertial measurements and guidance commands. The LSTM is trained on a large set of Mars entry trajectories controlled through the fully numerical predictor-corrector entry guidance (FNPEG) algorithm, with density and wind from the Mars Global Reference Atmospheric Model (GRAM) 2010. The training of the network is examined, ensuring that the LSTM generalizes well to samples not present in the training set, and the performance of the network is assessed on a separate training set. The errors of the reconstructed density and wind profiles are, respectively, within 0.54 and 1.9%. Larger wind errors take place at high altitudes due to the decreased sensitivity of the trajectory in regions of low dynamic pressure. The LSTM architecture reliably reproduces the atmospheric density and wind encountered during descent.