Learning Long Short-term Memory Networks Research Articles

Machine learning in electron beam lithography to boost photoresist formulation design for high-resolution patterning.

The reduction of the critical dimension (CD) usually improves the resolution of patterns and performance of chips. In chip manufacturing, electron beam lithography (EBL) is a promising technology for preparing sub-10 nm patterns, and its imaging resolution is primarily determined by the photoresist formulation. However, the smaller CDs are mainly achieved by optimizing process conditions, and little attention has been paid to the photoresist formulation optimization. Screening suitable photoresist formulations remains a significant challenge due to the considerable time and high cost. Herein, we report a formulation optimization technique of a metal oxide nanoparticle photoresist that combines EBL experiments with a machine learning long short-term memory (LSTM) network. Using the LSTM network, a CD photoresist evaluation model is established. Leveraging the CD model, a photoresist formulation optimizer is developed with a line width of 26 nm. The verification results demonstrate that the CDs predicted by the LSTM network are basically consistent with the EBL experimental results, and the photoresist formulations that meet the CD requirements can be screened. This work opens up a novel perspective to boost photoresist formulation design for high-resolution patterning with artificial intelligence and provides guidance for EBL experiments.

Daily flow discharge prediction using integrated methodology based on LSTM models: Case study in Brahmani-Baitarani basin

For flood control, hydropower operation, and agricultural planning, among other applications, flow discharge prediction is a critical first step toward the strong and dependable planning and management of water resources. Floods are destructive natural calamities that destroy human lives and infrastructure across the world. Development of effective flood forecasting and prediction models is critical for minimising deaths and mitigating damages. This study employs hybrid deep learning Long Short Term Memory (LSTM) algorithms like LSTM, Convolution LSTM (Conv-LSTM) and Convolutional Neural Network LSTM (CNN-LSTM) to predict likelihood flood events using daily precipitation, daily temperature and daily relative humidity from two flood-forecasting stations i.e., Champua (Baitarani River, Odisha) and Jarikela (Brahmani River, Odisha) over a 20-year period. The results show that CNN-LSTM performed best followed by Conv-LSTM and LSTM in terms of R2 = 0.98055, 0.96564, and 0.93244, RMSE = 19.137, 35.635, and 49.347, MAE = 18.372, 33.766, and 47.058, NSE = 0.971, 0.9517 and 0.9257 respectively. The findings support the claim that machine learning models and algorithms, in particular CNN-LSTM model, can be applied to flood forecasting with high accuracy, thereby enhancing water and hazard management.

Deep Learning Based Method for Early Warning of Price Risk in Railroad Bulk Freight Transport with Consideration of Time Series of Comprehensive Impact Index

The aim in this study is to enhance the price competitiveness of medium- and long-haul railroad bulk cargoes in the transportation market. Efficiently triggering an early warning for railroad bulk freight price risk is crucial. This paper discusses relevant factors affecting the pricing of railway freight from four aspects, namely the macroeconomy, the transportation market, the freight owner, and the railway enterprise. We construct a comprehensive index system for generating an early warning of railroad bulk freight prices. Utilizing a collected data set of 20 factor indicators that affected prices from 2015 to 2017, we calculate a comprehensive price risk warning index using the integrated entropy weight and the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) method. We propose a risk warning level classification method based on the k-means clustering method to determine five levels of price risk warning. Subsequently, we establish comprehensive index prediction methods for issuing price risk warnings based on the back propagation (BP) neural network and the deep learning long short-term memory (LSTM) network. We verify that the prediction performance of the deep learning LSTM network is significantly better than that of the BP neural network by combining all coal freight data from the Shanghai Railway Bureau for the period 2015 to 2017 as a case study. The proposed method can predict the level of freight price risk more accurately in the next phase, based on the status of the factor indicators, and be used to assist railway departments in developing a more reasonable freight price adjustment strategy to scientifically mitigate freight price risk.

Prediction of the epileptic seizure through deep learning techniques using electroencephalography

<span>Electroencephalography (EEG) is a widely used and significant technique for aiding in epilepsy diagnosis and investigating the electrical patterns of the human brain. Due to the non-stationary nature of EEG signals, seizure patterns will vary across different recording sessions for individual patients. In this study, a new deep learning long short-term memory (LSTM) model is implemented for the detection of brain tumors and seizures. The process consists of four key steps: EEG signal pre-processing, preictal feature extraction, hyper optimization using grey wolf optimization (GWO), and LSTM-based classification. The evaluation makes use of long-term EEG recordings from the EEG and ABIDE fMRI datasets. By experimenting with various modules and layers of memory units, a pre-analysis is first conducted to determine the best LSTM network architecture. The LSTM model makes use of numerous retrieved features, including temporal and frequency domain information between EEG channels that were extracted before classification. The discovery of the implemented methodology revealed significant advantages in accurately predicting seizures while minimizing false alarms. The implemented LSTM method achieves a 99% accuracy rate, 98% precision, 99% recall, and 98% f1-measure which is better when compared with cross sub-pattern correlation-based principal component analysis (SUBXPCA) and gradient-boosting decision tree (GBDT) methods.</span>

Deep Learning Recurrent Neural Network for Concussion Classification in Adolescents Using Raw Electroencephalography Signals: Toward a Minimal Number of Sensors.

Artificial neural networks (ANNs) are showing increasing promise as decision support tools in medicine and particularly in neuroscience and neuroimaging. Recently, there has been increasing work on using neural networks to classify individuals with concussion using electroencephalography (EEG) data. However, to date the need for research grade equipment has limited the applications to clinical environments. We recently developed a deep learning long short-term memory (LSTM) based recurrent neural network to classify concussion using raw, resting state data using 64 EEG channels and achieved high accuracy in classifying concussion. Here, we report on our efforts to develop a clinically practical system using a minimal subset of EEG sensors. EEG data from 23 athletes who had suffered a sport-related concussion and 35 non-concussed, control athletes were used for this study. We tested and ranked each of the original 64 channels based on its contribution toward the concussion classification performed by the original LSTM network. The top scoring channels were used to train and test a network with the same architecture as the previously trained network. We found that with only six of the top scoring channels the classifier identified concussions with an accuracy of 94%. These results show that it is possible to classify concussion using raw, resting state data from a small number of EEG sensors, constituting a first step toward developing portable, easy to use EEG systems that can be used in a clinical setting.

Recurrent neural network-based acute concussion classifier using raw resting state EEG data

Concussion is a global health concern. Despite its high prevalence, a sound understanding of the mechanisms underlying this type of diffuse brain injury remains elusive. It is, however, well established that concussions cause significant functional deficits; that children and youths are disproportionately affected and have longer recovery time than adults; and that individuals suffering from a concussion are more prone to experience additional concussions, with each successive injury increasing the risk of long term neurological and mental health complications. Currently, the most significant challenge in concussion management is the lack of objective, clinically- accepted, brain-based approaches for determining whether an athlete has suffered a concussion. Here, we report on our efforts to address this challenge. Specifically, we introduce a deep learning long short-term memory (LSTM)-based recurrent neural network that is able to distinguish between non-concussed and acute post-concussed adolescent athletes using only short (i.e. 90 s long) samples of resting state EEG data as input. The athletes were neither required to perform a specific task nor expected to respond to a stimulus during data collection. The acquired EEG data were neither filtered, cleaned of artefacts, nor subjected to explicit feature extraction. The LSTM network was trained and validated using data from 27 male, adolescent athletes with sports related concussion, benchmarked against 35 non-concussed adolescent athletes. During rigorous testing, the classifier consistently identified concussions with an accuracy of > 90% and achieved an ensemble median Area Under the Receiver Operating Characteristic Curve (ROC/AUC) equal to 0.971. This is the first instance of a high-performing classifier that relies only on easy-to-acquire resting state, raw EEG data. Our concussion classifier represents a promising first step towards the development of an easy-to-use, objective, brain-based, automatic classification of concussion at an individual level.

Clock bias prediction algorithm for navigation satellites based on a supervised learning long short-term memory neural network

In a satellite navigation system, high-precision prediction of satellite clock bias directly determines the accuracy of navigation, positioning, and time synchronization and is the key to realizing autonomous navigation. To further improve satellite clock bias prediction accuracy, we establish a satellite clock bias prediction model by using long short-term memory (LSTM) that can accurately express the nonlinear characteristics of the navigation satellite clock bias. Outliers in the original clock bias should be preprocessed before using the clock bias for prediction. By analyzing the working principle of the traditional median absolute deviations method, the ambiguity of the mathematical model of that method was improved. Experimental results show that the improved method is better than the traditional method at detecting gross errors. The single difference sequence of the preprocessed satellite clock bias was taken as the research object. First, a quadratic polynomial model was fit to the trend term of the single difference sequence. Second, based on the LSTM neural network model and the basic principles of supervised learning, a supervised learning LSTM network model (SL-LSTM) was proposed that models cyclic and random terms. Finally, the prediction function of the satellite clock bias was realized by extrapolating the model by adding a trend term. We adopt the GPS precision satellite clock bias of International GNSS Service data forecast experiments and apply wavelet neural network (WNN), autoregressive integrated moving average (ARIMA), and quadratic polynomial (QP) models to compare their prediction effects. The average prediction RMSE for 3 h, 6 h, 12 h, 1 d, and 3 d based on the SL-LSTM improved by approximately −21.80, −1.85, 8.57, 2.27, and 40.79%, respectively, compared with the results of the WNN. The average prediction RMSE based on the SL-LSTM improved by approximately 38.23, 65.48, 80.22, 85.18, and 94.51% compared with the ARIMA results. The average prediction RMSE based on the SL-LSTM improved by approximately 82.37, 75.88, 67.24, 45.71, and 58.22% compared with the QP results. Compared with the WNN, the SL-LSTM method has no obvious advantages in the prediction accuracy and stability in short-term prediction but achieves a better long-term prediction accuracy and stability. With an increased prediction duration, the SL-LSTM method is clearly better than the other three methods in terms of the prediction accuracy and stability. The results indicated that the quality of satellite clock bias prediction by the SL-LSTM method is better than that of the above three methods and is more suitable for the middle- and long-term prediction of satellite clock bias.

Unidirectional and Bidirectional LSTM Models for Short-Term Traffic Prediction

This paper presents the development and evaluation of short-term traffic prediction models using unidirectional and bidirectional deep learning long short-term memory (LSTM) neural networks. The unidirectional LSTM (Uni-LSTM) model provides high performance through its ability to recognize longer sequences of traffic time series data. In this work, Uni-LSTM is extended to bidirectional LSTM (BiLSTM) networks which train the input data twice through forward and backward directions. The paper presents a comparative evaluation of the two models for short-term speed and traffic flow prediction using a common dataset of field observations collected from multiple freeways in Australia. The results showed BiLSTM performed better for variable prediction horizons for both speed and flow. Stacked and mixed Uni-LSTM and BiLSTM models were also investigated for 15-minute prediction horizons resulting in improved accuracy when using 4-layer BiLSTM networks. The optimized 4-layer BiLSTM model was then calibrated and validated for multiple prediction horizons using data from three different freeways. The validation results showed a high degree of prediction accuracy exceeding 90% for speeds up to 60-minute prediction horizons. For flow, the model achieved accuracies above 90% for 5- and 10-minute prediction horizons and more than 80% accuracy for 15- and 30-minute prediction horizons. These findings extend the set of AI models available for road operators and provide them with confidence in applying robust models that have been tested and evaluated on different freeways in Australia.

Deep Learning-Based Minute-Scale Digital Prediction Model of Temperature-Induced Deflection of a Cable-Stayed Bridge: Case Study

The evolution rule of temperature-induced deflection in main girders is an important index to evaluate the service performance of long-span cable-stayed bridges, which directly reflects the coupling effect between the vertical stiffness of the main girder and the tension of multiple cables. However, temperature-induced deflection is caused by the complex temperature field of the main girder, cable tower and cable, while monitoring data have documented a time-lag effect between the temperature and temperature-induced deflection. Hence, it is difficult to accurately describe and model the behavior of the temperature-induced deflection in a long-span cable-stayed bridge in service. To this end, by utilizing the advantage of long short-term memory (LSTM) network for time series prediction, a digital model in minute scale based on monitoring data and deep learning can be developed to predict temperature-induced deflection, and resolve the low precision caused by the single-point input and time-lag effect. Compared with traditional machine learning algorithm and linear regression, a deep learning LSTM network has the best performance. For the cable-stayed bridge in this paper, the mean absolute error of the LSTM model was even less than 0.5 mm, and with the combined hypothesis test, the early warning accuracy for the abnormal change of temperature-induced deflection could achieve a minimum of 0.3%.

UWB Indoor Localization Using Deep Learning LSTM Networks

Localization using ultra-wide band (UWB) signals gives accurate position results for indoor localization. The penetrating characteristics of UWB pulses reduce the multipath effects and identify the user position with precise accuracy. In UWB-based localization, the localization accuracy depends on the distance estimation between anchor nodes (ANs) and the UWB tag based on the time of arrival (TOA) of UWB pulses. The TOA errors in the UWB system, reduce the distance estimation accuracy from ANs to the UWB tag and adds the localization error to the system. The position accuracy of a UWB system also depends on the line of sight (LOS) conditions between the UWB anchors and tag, and the computational complexity of localization algorithms used in the UWB system. To overcome these UWB system challenges for indoor localization, we propose a deep learning approach for UWB localization. The proposed deep learning model uses a long short-term memory (LSTM) network for predicting the user position. The proposed LSTM model receives the distance values from TOA-distance model of the UWB system and predicts the current user position. The performance of the proposed LSTM model-based UWB localization system is analyzed in terms of learning rate, optimizer, loss function, batch size, number of hidden nodes, timesteps, and we also compared the mean localization accuracy of the system with different deep learning models and conventional UWB localization approaches. The simulation results show that the proposed UWB localization approach achieved a 7 cm mean localization error as compared to conventional UWB localization approaches.

A deep learning based framework for optimizing cloud consumer QoS-based service composition

The service composition problem in Cloud computing is formulated as a multiple criteria decision-making problem. Due to the extensive search space, Cloud service composition is addressed as an NP-hard problem. In addition, it is a long term based and economically driven. Building an accurate economic model for service composition has great attention to interest and importance for the Cloud consumer. A deep learning based service composition (DLSC) framework has been proposed in this paper. The proposed DLSC framework is considered an amalgamation between the deep learning long short term memory (LSTM) network and particle swarm optimization (PSO) algorithm. The LSTM network is applied to accurately predict the Cloud QoS provisioned values, and the output of LSTM network is fed to PSO algorithm to compose the best Cloud providers to contract with them for composing the needed services to minimize the consumer cost function. The proposed DLSC framework has been implemented using a real QoS dataset. According to the comparative results, it is found that the performance of the proposed framework outperforms the existing models with respect to the predictive accuracy and composition accuracy.

The Memory Advantage of Long Short-Term Memory Networks for Bond Yield Forecasting

The importance of bond markets in the financial industry stems from its dimension, its direct relevance for other asset classes and for the overall economy. In this paper, we conduct the first study of bond yield forecasting using deep learning long short-term memory (LSTM) networks, validating the potential of LSTMs networks for that purpose, and identifying the LSTM's memory advantage over standard feedforward neural networks, in particular, the multilayer perceptron (MLP). Specifically, we model the 10-year Euro government bond yield using univariate LSTMs with different input sequences (6, 21 and 61 time steps), considering five forecasting horizons, from next day to 20 days ahead. We compare those LSTM models with MLPs, both univariate as well as using the most relevant features for each forecasting horizon. The results show that the univariate LSTM model with additional memory is capable of achieving similar results as the multivariate MLP using information from markets and the economy. Moreover, the direct comparison of models in identical conditions, i.e. small input sequence of 5 time steps, leads to results with LSTMs that are similar or better with lower standard deviations. Furthermore, with the LSTMs, shorter forecasting horizons require smaller input sequences and vice-versa. In summary, the results are encouraging for its use incorporating macroeconomic and market information, adjusting the input sequence length to the forecasting horizon considered.

Early Prediction of Sepsis in EMR Records Using Traditional ML Techniques and Deep Learning LSTM Networks.

Sepsis is a life-threatening condition caused by infection and subsequent overreaction by the immune system. Physicians effectively treat sepsis with early administration of antibiotics. However, excessive use of antibiotics on false positive cases cultivates antibiotic resistant bacterial strains and can waste resources while false negative cases result in unacceptable mortality rates. Accurate early prediction ensures correct, early antibiotic treatment; unfortunately, prediction remains daunting due to error-ridden electronic medical records (EMRs) and the inherent complexity of sepsis. We aimed to predict sepsis using only the first 24 and 36 hours of lab results and vital signs for a patient. We used the Medical Information Mart for Intensive Care III (MIMIC3) dataset to test machine learning (ML) techniques including traditional methods (i.e., random forest (RF) and logistic regression (LR)) as well as deep learning techniques (i.e., long short-term memory (LSTM) neural networks). We successfully created a data pipeline to process and clean data, identified important predictive features using RF and LR techniques, and trained LSTM networks. We found that our best performing traditional classifier, RF, had an Area Under the Curve (AUC-ROC) score of 0.696, and our LSTM networks did not outperform RF.

Real-Time Classification of Earthquake using Deep Learning

Existing Earthquake Early Warning Systems (EEWSs) calculates the location and magnitude of an earthquake using real-time waveforms from seismic stations within a few seconds. Typically, three to six stations are necessary to estimate earthquake parameters. Waiting for primary (P-) wave information from closest stations results in a blind-zone area where the arrival of secondary (S-) wave cannot be provided around the epicenter of an earthquake. If an earthquake occurred under a city center, EEWSs would not work even though each building has a seismic sensor in a smart city in future. Here, we present a methodology to classify earthquake vibrations into near-source or far-source within one second after P-wave detection. This will allow warnings to citizens who are the residence of earthquake epicenter in case of an earthquake very close by. We trained a deep learning Long Short-Term Memory (LSTM) network for sequence-to-label classification. 305 three component accelerations recorded between 2000 and 2018 in Japan are used to train the artificial network by extracting thirteen features of one second of P-wave. The accuracy of the methodology is 98.2%. 54 out of 55 near-source waveforms classified correctly and only 2 of 80 waveforms were misclassified. We tested the LSTM network with 2018 Northern Osaka (M 6.1.) earthquakes in Japan where closest stations are correctly identified with 83.3% accuracy. Therefore, smart cities donated with smart automated shut-on/off machines and sensors will be more resilient against earthquake disaster even EEWSs are not available in the blind zone area in future.