Combination Of Long Short-term Memory Research Articles

Energy-Efficient Anomaly Detection and Chaoticity in Electric Vehicle Driving Behavior.

Detection of abnormal situations in mobile systems not only provides predictions about risky situations but also has the potential to increase energy efficiency. In this study, two real-world drives of a battery electric vehicle and unsupervised hybrid anomaly detection approaches were developed. The anomaly detection performances of hybrid models created with the combination of Long Short-Term Memory (LSTM)-Autoencoder, the Local Outlier Factor (LOF), and the Mahalanobis distance were evaluated with the silhouette score, Davies-Bouldin index, and Calinski-Harabasz index, and the potential energy recovery rates were also determined. Two driving datasets were evaluated in terms of chaotic aspects using the Lyapunov exponent, Kolmogorov-Sinai entropy, and fractal dimension metrics. The developed hybrid models are superior to the sub-methods in anomaly detection. Hybrid Model-2 had 2.92% more successful results in anomaly detection compared to Hybrid Model-1. In terms of potential energy saving, Hybrid Model-1 provided 31.26% superiority, while Hybrid Model-2 provided 31.48%. It was also observed that there is a close relationship between anomaly and chaoticity. In the literature where cyber security and visual sources dominate in anomaly detection, a strategy was developed that provides energy efficiency-based anomaly detection and chaotic analysis from data obtained without additional sensor data.

E-LSTM: EfficientNet and Long Short-Term Memory Model for Detection of Glaucoma Diseases

Glaucoma is an eye disease that often has no symptoms until it is advanced. According to the World Health Organization (WHO), after cataracts, glaucoma is the second-leading cause of permanent blindness globally and is expected to affect 111.8 million patients by 2040. Early detection of glaucoma is important to reduce the risk of permanent blindness. Detection is achieved by structural measurement of early thinning of the retinal nerve fiber layer (RNFL). The RNFL is the portion of the retina located outside the optic nerve head (ONH) and can be observed in fundus images of the retina. Analysis of retinal fundus images can be performed with computer assistance using machine learning, especially deep learning. This study proposes a deep learning-based model, a convolutional neural network (CNN) using the EfficientNet architecture combined with long short-term memory (LSTM), for laucoma detection. Using ACRIMA, DRISHTI-GS, and RIM-ONE DL datasets with k-fold cross-validation, the model achieved high performance on the ACRIMA dataset: accuracy 0.9799, loss 0.0596, precision 0.9802, sensitivity 0.9799, specificity 0.9771, and F1score 0.9799. This EfficientNet and LSTM combination (e-LSTM) outperformed previous studies, offering a promising alternative for evaluating retinal fundus images in glaucoma detection.

Fraud Detection of the Electricity Consumption by combining Deep Learning and Statistical Methods

An important issue for the electricity distribution companies is the non-technical loss (NTL), also known as electricity fraud. This issue has a significant impact on the economies of all countries in the world. In this context, we studied the problem of the imbalance between the electrical energy invoiced and the electrical energy supplied within the Algerian economic sector. This article presents an approach to detecting electrical fraud using a combination of Long Short-Term Memory (LSTM) and robust Exponential and Holt-Winters Smoothing (EHWS) methods in order to enhance the accuracy and efficacy of fraud detection mechanisms. The proposed approach investigates the fraudulent behaviour of electricity consumers and unfolds in several key phases. In the first step, monthly consumption forecasts are made and the model with the most accurate results is selected. Then, the phase of detection of anomalies in economic meters and detection of cases of fraud by economic customers begins. This phase relies on the robust exponential and Holt-Winters Smoothing methods for uncovering irregular patterns indicative of potential fraud instances. The proposed model was trained and evaluated, and several experiments were carried out using a large dataset of real users from the economic sector. The dataset comprised approximately 2,000 customers and encompassed 14 years of monthly electricity usage in Bejaia, Algeria. The results of the experiments demonstrate promising performance, underscoring the efficacy of our proposed solution in effectively detecting instances of fraud, leading us to conclude that this proposition is robust and can help improve the accuracy of locating abnormal consumer behaviour and increase the company's profits.

Evaluating the machine learning models based on natural language processing tasks

<span lang="EN-US">In the realm of natural language processing (NLP), a diverse array of language models has emerged, catering to a wide spectrum of tasks, ranging from speaker recognition and auto-correction to sentiment analysis and stock prediction. The significance of language models in enabling the execution of these NLP tasks cannot be overstated. This study proposes an approach to enhance accuracy by leveraging a hybrid language model, combining the strengths of long short-term memory (LSTM) and gated recurrent unit (GRU). LSTM excels in preserving long-term dependencies in data, while GRU's simpler gating mechanism expedites the training process. The research endeavors to evaluate four variations of this hybrid model: LSTM, GRU, <a name="_Hlk158992630"></a>bidirectional long short-term memory (Bi-LSTM), and a combination of LSTM with GRU. These models are subjected to rigorous testing on two distinct datasets: one focused on IBM stock price prediction, and the other on Jigsaw toxic comment classification (sentiment analysis). This work represents a significant stride towards democratizing NLP capabilities, ensuring that even in resource-constrained settings, NLP models can exhibit improved performance. The anticipated implications of these findings span a wide spectrum of real-world applications and hold the potential to stimulate further research in the field of NLP. </span>

Intelligent forecasting temperature measurements of solar PV cells using modified recurrent neural network

For microgrids to operate optimally and minimize the effects of uncertainty, anticipating solar PV measurements is essential. For residential and commercial microgrids that use solar PV, the predicting of solar energy over a short period is crucial for managing grid-connected PVeffectively. Therefore, this work develops a Recurrent Neural Network (RNN) for forecasting temperature measurements as time series records, where a combination of long short-term memory (LSTM) architecture with RNN is used to process input measurements by updating the RNN state and winding over time degrees. Data from the entire prior time steps is stored in the RNN state. A dataset of temperature waveform measurements is used, which includes 2000 unnaturally produced signals of three channels with varying length. An LSTM neural network can be used to expect future values of a time series or sequence utilizing data from earlier time steps as input. Training of a regression LSTM neural network through the output of a sequence is performed, where the goals are the training sequence with records shifting one-time step, for training theLSTM neural architecture with time series forecasting. In other words, the weights of the LSTM neural structure learn to predict the following time step values of the input sequence at every time step. By considering the past forecasts as inputs, the closed-loop prediction forecasts the next time steps of sequences. The model makes the forecast without using the true data. The cross-entropy loss serves as the loss function. It is found that the mean RMSE overall test observations were about 0.5080 which promises to make better predictions from learning the temporal context of input sequences

Various optimized machine learning techniques to predict agricultural commodity prices

AbstractRecent increases in global food demand have made this research and, therefore, the prediction of agricultural commodity prices, almost imperative. The aim of this paper is to build efficient artificial intelligence methods to effectively forecast commodity prices in light of these global events. Using three separate, well-structured models, the commodity prices of eleven major agricultural commodities that have recently caused crises around the world have been predicted. In achieving its objective, this paper proposes a novel forecasting model for agricultural commodity prices using the extreme learning machine technique optimized with the genetic algorithm. In predicting the eleven commodities, the proposed model, the extreme learning machine with the genetic algorithm, outperforms the model formed by the combination of long short-term memory with the genetic algorithm and the autoregressive integrated moving average model. Despite the fluctuations and changes in agricultural commodity prices in 2022, the extreme learning machine with the genetic algorithm model described in this study successfully predicts both qualitative and quantitative behavior in such a large number of commodities and over such a long period of time for the first time. It is expected that these predictions will provide benefits for the effective management, direction and, if necessary, restructuring of agricultural policies by providing food requirements that adapt to the dynamic structure of the countries.

Dynamic constitutive identification of concrete based on improved dung beetle algorithm to optimize long short-term memory model

In order to improve the accuracy of concrete dynamic principal identification, a concrete dynamic principal identification model based on Improved Dung Beetle Algorithm (IDBO) optimized Long Short-Term Memory (LSTM) network is proposed. Firstly, the apparent stress–strain curves of concrete containing damage evolution were measured by Split Hopkinson Pressure Bar (SHPB) test to decouple and separate the damage and rheology, and this system was modeled by using LSTM network. Secondly, for the problem of low convergence accuracy and easy to fall into local optimum of Dung Beetle Algorithm (DBO), the greedy lens imaging reverse learning initialization population strategy, the embedded curve adaptive weighting factor and the PID control optimal solution perturbation strategy are introduced, and the superiority of IDBO algorithm is proved through the comparison of optimization test with DBO, Harris Hawk Optimization Algorithm, Gray Wolf Algorithm, and Fruit Fly Algorithm and the combination of LSTM is built to construct the IDBO-LSTM dynamic homeostasis identification model. The final results show that the IDBO-LSTM model can recognize the concrete material damage without considering the damage; in the case of considering the damage, the IDBO-LSTM prediction curves basically match the SHPB test curves, which proves the feasibility and excellence of the proposed method.

Deep hybrid modeling of a HEK293 process: Combining long short-term memory networks with first principles equations.

The combination of physical equations with deep learning is becoming a promising methodology for bioprocess digitalization. In this paper, we investigate for the first time the combination of long short-term memory (LSTM) networks with first principles equations in a hybrid workflow to describe human embryonic kidney 293 (HEK293) culture dynamics. Experimental data of 27 extracellular state variables in 20 fed-batch HEK293 cultures were collected in a parallel high throughput 250 mL cultivation system in an industrial process development setting. The adaptive moment estimation method with stochastic regularization and cross-validation were employed for deep learning. A total of 784 hybrid models with varying deep neural network architectures, depths, layers sizes and node activation functions were compared. In most scenarios, hybrid LSTM models outperformed classical hybrid Feedforward Neural Network (FFNN) models in terms of training and testing error. Hybrid LSTM models revealed to be less sensitive to data resampling than FFNN hybrid models. As disadvantages, Hybrid LSTM models are in general more complex (higher number of parameters) and have a higher computation cost than FFNN hybrid models. The hybrid model with the highest prediction accuracy consisted in a LSTM network with seven internal states connected in series with dynamic material balance equations. This hybrid model correctly predicted the dynamics of the 27 state variables (R2 = 0.93 in the test data set), including biomass, key substrates, amino acids and metabolic by-products for around 10 cultivation days.

Video-Based Identification and Prediction Techniques for Stable Vessel Trajectories in Bridge Areas.

In recent years, the global upswing in vessel-bridge collisions underscores the vital need for robust vessel track identification in accident prevention. Contemporary vessel trajectory identification strategies often integrate target detection with trajectory tracking algorithms, employing models like YOLO integrated with DeepSORT or Bytetrack algorithms. However, the accuracy of these methods relies on target detection outcomes and the imprecise boundary acquisition method results in erroneous vessel trajectory identification and tracking, leading to both false positives and missed detections. This paper introduces a novel vessel trajectory identification framework. The Co-tracker, a long-term sequence multi-feature-point tracking method, accurately tracks vessel trajectories by statistically calculating the translation and heading angle transformation of feature point clusters, mitigating the impact of inaccurate vessel target detection. Subsequently, vessel trajectories are predicted using a combination of Long Short-Term Memory (LSTM) and a Graph Attention Neural Network (GAT) to facilitate anomaly vessel trajectory warnings, ensuring precise predictions for vessel groups. Compared to prevalent algorithms like YOLO integrated with DeepSORT, our proposed method exhibits superior accuracy and captures crucial heading angle features. Importantly, it effectively mitigates the common issues of false positives and false negatives in detection and tracking tasks. Applied in the Three Rivers area of Ningbo, this research provides real-time vessel group trajectories and trajectory predictions. When the predicted trajectory suggests potential entry into a restricted zone, the system issues timely audiovisual warnings, enhancing real-time alert functionality. This framework markedly improves vessel traffic management efficiency, diminishes collision risks, and ensures secure navigation in multi-target and wide-area vessel scenarios.

Breast cancer recurrence prediction with deep neural network and feature optimization

Breast cancer remains a pervasive global health concern, necessitating continuous efforts to attain effectiveness of recurrence prediction schemes. This work focuses on breast cancer recurrence prediction using two advanced architectures such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), integrated with feature selection techniques utilizing Logistic Regression (LR) and Analysis of Variance (ANOVA). The well-known Wisconsin cancer registry dataset, which contains vital diagnostic data from breast mass fine-needle aspiration biopsies, was employed in this study. The mean values of accuracy, precision, recall and F1-score for the proposed LR-CNN-LSTM model were calculated as 98.24%, 99.14%, 98.30% and 98.14% respectively. The mean values of accuracy, precision, recall and F1-score for the proposed ANOVA-GRU model were calculated as 96.49%, 97.04%, 96.67% and 96.67% respectively. The comparison with traditional methods showcases the superiority of our proposed approach. Moreover, the insights gained from feature selection contribute to a deeper understanding of the critical factors influencing breast cancer recurrence. The combination of LSTM and GRU models with feature selection methods not only enhances prediction accuracy but also provides valuable insights for medical practitioners. This research holds the potential to aid in early diagnosis and personalized treatment strategies.

Multimodal Deep Learning for Oil Price Forecasting Using Economic Indicators

In the evolving landscape of global energy, accurately forecasting oil prices plays a pivotal role in strategizing the transition to green energy. Traditional forecasting methods, though widely used, often fall short in capturing the multifaceted influences on oil price dynamics. This research introduces a multimodal deep learning approach that incorporates both time series oil price data and key economic indicators to enhance forecasting accuracy. By employing a combination of Long Short-Term Memory (LSTM) and Temporal Convolutional Networks (TCN), the model adeptly captures temporal patterns and economic influences. The findings not only demonstrate the model's superior performance over traditional methods but also underscore the profound impact of specific economic indicators on oil prices.

SENTIMENT ANALYSIS WITH LONG-SHORT TERM MEMORY (LSTM) AND GATED RECURRENT UNIT (GRU) ALGORITHMS

Sentiment analysis is a form of machine learning that functions to obtain emotional polarity values or data tendencies from data in the form of text. Sentiment analysis is needed to analyze opinions, sentiments, reviews, and criticisms from someone for a product, service, organization, topic, etc. Recurrent Neural Network (RNN) is one of the Natural Language Processing (NLP) algorithms that is used in sentiment analysis. RNN is a neural network that can use internal memory to process input. RNN itself has a weakness in Long-Term Memory (LTM). Therefore, this article examines the combination of Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) algorithms. GRU is an algorithm that is used to make each recurrent unit able to record adaptively at different time scales. Meanwhile, LSTM is a network architecture with the advantage of learning long-term dependencies on data. LSTM can remember long-term memory information, learn long-sequential data, and form information relation data in LTM. The combination of LSTM and GRU aims to overcome RNN’s weakness in LTM. The LSTM-GRU is combined by adding GRU to the data generated from LSTM. The combination of LSTM and GRU creates a better performance algorithm for addressing the LTM problem.

A novel machine learning and deep learning semi-supervised approach for automatic detection of InSAR-based deformation hotspots

Over the past two decades, Interferometric synthetic aperture radar (InSAR) has been invaluable for studying earth surface deformation and related effects. Deformation maps generated through multi-temporal InSAR processing methods are however difficult to interpret accurately by general individual users, decision-makers, and non-domain experts owing to the volume, variety, and velocity they are produced. This paper proposes a semi-supervised machine learning based information mining approach to simplify these deformation maps and detect hotspots by extracting prominent signals from time series deformation. The approach initially combines two machine learning based clustering methods named time series k-means (TSKM) and Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithms to derive clusters with unique spatiotemporal deformation behavior, using time series deformation output generated from Wavelet-based InSAR (WabInSAR) method. Clustering results generated from this unsupervised machine learning approach are later used as training labels to develop two deep learning models, one using long short term memory (LSTM) networks alone and another using a combination of LSTM and single-layer perceptron for supervised training. The developed LSTM and LSTM + Perceptron models efficiently learn from the cluster labels, reaching an accuracy of 97.3 %. Further, the deep learning models significantly reduce the computational time from orders of days (∼5) to hours (∼2) while training and from hours to minutes during prediction. We evaluate the developed approach over Los Angeles, a highly challenging area affected by umpteen deformation events that are challenging to categorize. The outcome of the proposed approach produces hotspots of deforming areas in Los Angeles, providing a generalized and more precise picture of events, much appreciable to non-domain experts. The approach can augment any of the multi-temporal InSAR processing chains and is applicable to different deformation prone sites, aiding in derivation of deformation hotspots from time series deformation maps.

Traffic Flow Forecast using Time Series Analysis based on Machine Learning Algorithm

Time series forecasting plays a critical role in predicting future trends and making informed decisions across various domains. In this research paper, we present a multi-model approach to improve the accuracy and robustness of time series forecasting. Our methodology involves integrating two powerful models, Long Short-Term Memory (LSTM) and Seasonal Autoregressive Integrated Moving Average (SARIMA), to capture complex temporal patterns and seasonality in the data. We leverage real-world traffic data from different junctions and employ a comprehensive pipeline for data preprocessing, including data selection, resampling, and sequence generation. Our study focuses on four junctions, and for each junction, we individually train LSTM models, followed by SARIMA models for comparison. The LSTM models are designed to capture long-term dependencies in the time series data, while the SARIMA models are tailored to handle seasonality and autocorrelation. We implement a custom sequence generation process and split the dataset into training and testing sets for model evaluation. The LSTM models are trained using Pytorch, optimizing them for accurate short-term predictions. Our results show that the combination of LSTM and SARIMA models yields superior forecasting performance compared to using each model individually. We present a detailed analysis of the forecasted results, including Root Mean Square Error (RMSE) calculations and visualizations to demonstrate the effectiveness of our multi-model approach. This research contributes to the field of time series forecasting by showcasing the benefits of combining deep learning and classical statistical methods. The proposed approach provides a flexible and robust framework applicable to various time series prediction tasks, offering improved accuracy and reliability in forecasting future trends.

A System to Detect Oilwell Anomalies Using Deep Learning and Decision Diagram Dual Approach

Summary Detecting unexpected events is a field of interest in oil and gas companies to improve operational safety and reduce costs associated with nonproductive time (NPT) and failure repair. This work presents a system for real-time monitoring of unwanted events using the production sensor data from oil wells. It uses a combination of long short-term memory (LSTM) autoencoder and a rule-based analytic approach to perform the detection of anomalies from sensor data. Initial studies are conducted to determine the behavior and correlations of pressure and temperature values for the most common combinations of well valve states. The proposed methodology uses pressure and temperature sensor data, from which a decision diagram (DD) classifies the well status, and this response is applied to the training of neural networks devoted to anomaly detection. Data sets related to several operations in wells located at different oil fields are used to train and validate the dual approach presented. The combination of the two techniques enables the deep neural network to evolve constantly through the normal data collected by the analytical method. The developed system exhibits high accuracy, with true positive detection rates exceeding 90% in the early stages of anomalies identified in both simulated and actual well production scenarios. It was implemented in more than 20 floating production, storage, and offloading (FPSO) vessels, monitoring more than 250 production/injection subsea wells, and can be applied both in real-time operation and in testing scenarios.

A surrogate non-intrusive reduced order model of quasi-geostrophic turbulence dynamics based on a combination of LSTM and different approaches of DMD

Mathematical modeling is applied to study phenomena and system behavior.In various engineering fields, many physical phenomena are illustrated using a set of differential equations.In many real-world applications, the mathematical models are very complex, and numerical simulations in high-dimensional systems are challenging.Examples of these problems are large-scale physical problems such as geophysical, which have high temporal and spatial variations.In these problems, model order reduction is a useful method for achieving an appropriate approximation because it can significantly decrease computational costs.Deep learning has recently been used to explore information from data and make predictions.There are several methods for dimensionality reduction.In this paper, we combine the dynamic mode decomposition (DMD) and the long short-term memory (LSTM) network.This is because LSTM can predict nonlinear systems and time series data.We use LSTM and DMD to predict nonlinear systems and reduce dimensions, respectively.Four common DMD schemes have been applied for dimensionality reduction.The common geophysical dataset has been used to evaluate the performance of the proposed method.Finally, we compare the variations of the modal coefficients which are obtained from snapshots projection and the reduced-order model.These results show the high accuracy of our proposed method.One of the things that is important is the time complexity of algorithm implementation.The time complexity of the proposed method is 10 times faster when 15 modes are used for modeling than when all features are used.

Investigation of a Hybrid LSTM + 1DCNN Approach to Predict In-Cylinder Pressure of Internal Combustion Engines

The control of internal combustion engines is becoming increasingly challenging to the customer’s requirements for growing performance and ever-stringent emission regulations. Therefore, significant computational efforts are required to manage the large amount of data coming from the field for engine optimization, leading to increased operating times and costs. Machine-learning techniques are being increasingly used in the automotive field as virtual sensors, fault detection systems, and performance-optimization applications for their real-time and low-cost implementation. Among them, the combination of long short-term memory (LSTM) together with one-dimensional convolutional neural networks (1DCNN), i.e., LSTM + 1DCNN, has proved to be a promising tool for signal analysis. The architecture exploits the CNN characteristic to combine feature classification and extraction, creating a single adaptive learning body with the ability of LSTM to follow the sequential nature of sensor measurements over time. The current research focus is on evaluating the possibility of integrating virtual sensors into the on-board control system. Specifically, the primary objective is to assess and harness the potential of advanced machine-learning technologies to replace physical sensors. In realizing this goal, the present work establishes the first step by evaluating the forecasting performance of a LSTM + 1DCNN architecture. Experimental data coming from a three-cylinder spark-ignition engine under different operating conditions are used to predict the engine’s in-cylinder pressure traces. Since using in-cylinder pressure transducers in road cars is not economically viable, adopting advanced machine-learning technologies becomes crucial to avoid structural modifications while preserving engine integrity. The results show that LSTM + 1DCNN is particularly suited for the prediction of signals characterized by a higher variability. In particular, it consistently outperforms other architectures utilized for comparative purposes, achieving average error percentages below 2%. As cycle-to-cycle variability increases, LSTM + 1DCNN reaches average error percentages below 1.5%, demonstrating the architecture’s potential for replacing physical sensors.

Image Orator - Image to Speech Using CNN, LSTM and GTTS

Abstract: This report presents an image to audio system that utilizes a combination of Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNN) for image captioning and Google Text-to-Speech (GTTS) for generating audio output. The aim of the project is to create an accessible system that converts images into descriptive audio signals for visually impaired individuals. The proposed system has the potential to provide meaningful context and information about the image through descriptive audio output, making it easier for visually impaired individuals to engage with visual content. In conclusion, the proposed image to audio system, which combines LSTM and CNN for image captioning and GTTS for audio generation, is a promising approach to making visual content more accessible to individuals with visual impairments. Future work may involve exploring different neural network architectures, optimising the system for real-time performance, and incorporating additional audio features to enhance the overall user experience.

Sentiment analysis of Indonesian datasets based on a hybrid deep-learning strategy

Various attempts have been conducted to improve the performance of text-based sentiment analysis. These significant attempts have focused on text representation and model classifiers. This paper introduced a hybrid model based on the text representation and the classifier models, to address sentiment classification with various topics. The combination of BERT and a distilled version of BERT (DistilBERT) was selected in the representative vectors of the input sentences, while the combination of long short-term memory and temporal convolutional networks was taken to enhance the proposed model in understanding the semantics and context of each word. The experiment results showed that the proposed model outperformed various counterpart schemes in considered metrics. The reliability of the proposed model was confirmed in a mixed dataset containing nine topics.

A short-term prediction model of global ionospheric VTEC based on the combination of long short-term memory and convolutional long short-term memory

A short-term prediction model of global ionospheric VTEC based on the combination of long short-term memory and convolutional long short-term memory