Combination Of Long Short-term Memory Research Articles

Evaluating the machine learning models based on natural language processing tasks

<span lang="EN-US">In the realm of natural language processing (NLP), a diverse array of language models has emerged, catering to a wide spectrum of tasks, ranging from speaker recognition and auto-correction to sentiment analysis and stock prediction. The significance of language models in enabling the execution of these NLP tasks cannot be overstated. This study proposes an approach to enhance accuracy by leveraging a hybrid language model, combining the strengths of long short-term memory (LSTM) and gated recurrent unit (GRU). LSTM excels in preserving long-term dependencies in data, while GRU's simpler gating mechanism expedites the training process. The research endeavors to evaluate four variations of this hybrid model: LSTM, GRU, <a name="_Hlk158992630"></a>bidirectional long short-term memory (Bi-LSTM), and a combination of LSTM with GRU. These models are subjected to rigorous testing on two distinct datasets: one focused on IBM stock price prediction, and the other on Jigsaw toxic comment classification (sentiment analysis). This work represents a significant stride towards democratizing NLP capabilities, ensuring that even in resource-constrained settings, NLP models can exhibit improved performance. The anticipated implications of these findings span a wide spectrum of real-world applications and hold the potential to stimulate further research in the field of NLP. </span>

Various optimized machine learning techniques to predict agricultural commodity prices

AbstractRecent increases in global food demand have made this research and, therefore, the prediction of agricultural commodity prices, almost imperative. The aim of this paper is to build efficient artificial intelligence methods to effectively forecast commodity prices in light of these global events. Using three separate, well-structured models, the commodity prices of eleven major agricultural commodities that have recently caused crises around the world have been predicted. In achieving its objective, this paper proposes a novel forecasting model for agricultural commodity prices using the extreme learning machine technique optimized with the genetic algorithm. In predicting the eleven commodities, the proposed model, the extreme learning machine with the genetic algorithm, outperforms the model formed by the combination of long short-term memory with the genetic algorithm and the autoregressive integrated moving average model. Despite the fluctuations and changes in agricultural commodity prices in 2022, the extreme learning machine with the genetic algorithm model described in this study successfully predicts both qualitative and quantitative behavior in such a large number of commodities and over such a long period of time for the first time. It is expected that these predictions will provide benefits for the effective management, direction and, if necessary, restructuring of agricultural policies by providing food requirements that adapt to the dynamic structure of the countries.

Dynamic constitutive identification of concrete based on improved dung beetle algorithm to optimize long short-term memory model

In order to improve the accuracy of concrete dynamic principal identification, a concrete dynamic principal identification model based on Improved Dung Beetle Algorithm (IDBO) optimized Long Short-Term Memory (LSTM) network is proposed. Firstly, the apparent stress–strain curves of concrete containing damage evolution were measured by Split Hopkinson Pressure Bar (SHPB) test to decouple and separate the damage and rheology, and this system was modeled by using LSTM network. Secondly, for the problem of low convergence accuracy and easy to fall into local optimum of Dung Beetle Algorithm (DBO), the greedy lens imaging reverse learning initialization population strategy, the embedded curve adaptive weighting factor and the PID control optimal solution perturbation strategy are introduced, and the superiority of IDBO algorithm is proved through the comparison of optimization test with DBO, Harris Hawk Optimization Algorithm, Gray Wolf Algorithm, and Fruit Fly Algorithm and the combination of LSTM is built to construct the IDBO-LSTM dynamic homeostasis identification model. The final results show that the IDBO-LSTM model can recognize the concrete material damage without considering the damage; in the case of considering the damage, the IDBO-LSTM prediction curves basically match the SHPB test curves, which proves the feasibility and excellence of the proposed method.

Deep hybrid modeling of a HEK293 process: Combining long short-term memory networks with first principles equations.

The combination of physical equations with deep learning is becoming a promising methodology for bioprocess digitalization. In this paper, we investigate for the first time the combination of long short-term memory (LSTM) networks with first principles equations in a hybrid workflow to describe human embryonic kidney 293 (HEK293) culture dynamics. Experimental data of 27 extracellular state variables in 20 fed-batch HEK293 cultures were collected in a parallel high throughput 250 mL cultivation system in an industrial process development setting. The adaptive moment estimation method with stochastic regularization and cross-validation were employed for deep learning. A total of 784 hybrid models with varying deep neural network architectures, depths, layers sizes and node activation functions were compared. In most scenarios, hybrid LSTM models outperformed classical hybrid Feedforward Neural Network (FFNN) models in terms of training and testing error. Hybrid LSTM models revealed to be less sensitive to data resampling than FFNN hybrid models. As disadvantages, Hybrid LSTM models are in general more complex (higher number of parameters) and have a higher computation cost than FFNN hybrid models. The hybrid model with the highest prediction accuracy consisted in a LSTM network with seven internal states connected in series with dynamic material balance equations. This hybrid model correctly predicted the dynamics of the 27 state variables (R2 = 0.93 in the test data set), including biomass, key substrates, amino acids and metabolic by-products for around 10 cultivation days.

Video-Based Identification and Prediction Techniques for Stable Vessel Trajectories in Bridge Areas.

In recent years, the global upswing in vessel-bridge collisions underscores the vital need for robust vessel track identification in accident prevention. Contemporary vessel trajectory identification strategies often integrate target detection with trajectory tracking algorithms, employing models like YOLO integrated with DeepSORT or Bytetrack algorithms. However, the accuracy of these methods relies on target detection outcomes and the imprecise boundary acquisition method results in erroneous vessel trajectory identification and tracking, leading to both false positives and missed detections. This paper introduces a novel vessel trajectory identification framework. The Co-tracker, a long-term sequence multi-feature-point tracking method, accurately tracks vessel trajectories by statistically calculating the translation and heading angle transformation of feature point clusters, mitigating the impact of inaccurate vessel target detection. Subsequently, vessel trajectories are predicted using a combination of Long Short-Term Memory (LSTM) and a Graph Attention Neural Network (GAT) to facilitate anomaly vessel trajectory warnings, ensuring precise predictions for vessel groups. Compared to prevalent algorithms like YOLO integrated with DeepSORT, our proposed method exhibits superior accuracy and captures crucial heading angle features. Importantly, it effectively mitigates the common issues of false positives and false negatives in detection and tracking tasks. Applied in the Three Rivers area of Ningbo, this research provides real-time vessel group trajectories and trajectory predictions. When the predicted trajectory suggests potential entry into a restricted zone, the system issues timely audiovisual warnings, enhancing real-time alert functionality. This framework markedly improves vessel traffic management efficiency, diminishes collision risks, and ensures secure navigation in multi-target and wide-area vessel scenarios.

Breast cancer recurrence prediction with deep neural network and feature optimization

Breast cancer remains a pervasive global health concern, necessitating continuous efforts to attain effectiveness of recurrence prediction schemes. This work focuses on breast cancer recurrence prediction using two advanced architectures such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), integrated with feature selection techniques utilizing Logistic Regression (LR) and Analysis of Variance (ANOVA). The well-known Wisconsin cancer registry dataset, which contains vital diagnostic data from breast mass fine-needle aspiration biopsies, was employed in this study. The mean values of accuracy, precision, recall and F1-score for the proposed LR-CNN-LSTM model were calculated as 98.24%, 99.14%, 98.30% and 98.14% respectively. The mean values of accuracy, precision, recall and F1-score for the proposed ANOVA-GRU model were calculated as 96.49%, 97.04%, 96.67% and 96.67% respectively. The comparison with traditional methods showcases the superiority of our proposed approach. Moreover, the insights gained from feature selection contribute to a deeper understanding of the critical factors influencing breast cancer recurrence. The combination of LSTM and GRU models with feature selection methods not only enhances prediction accuracy but also provides valuable insights for medical practitioners. This research holds the potential to aid in early diagnosis and personalized treatment strategies.

SENTIMENT ANALYSIS WITH LONG-SHORT TERM MEMORY (LSTM) AND GATED RECURRENT UNIT (GRU) ALGORITHMS

Sentiment analysis is a form of machine learning that functions to obtain emotional polarity values or data tendencies from data in the form of text. Sentiment analysis is needed to analyze opinions, sentiments, reviews, and criticisms from someone for a product, service, organization, topic, etc. Recurrent Neural Network (RNN) is one of the Natural Language Processing (NLP) algorithms that is used in sentiment analysis. RNN is a neural network that can use internal memory to process input. RNN itself has a weakness in Long-Term Memory (LTM). Therefore, this article examines the combination of Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) algorithms. GRU is an algorithm that is used to make each recurrent unit able to record adaptively at different time scales. Meanwhile, LSTM is a network architecture with the advantage of learning long-term dependencies on data. LSTM can remember long-term memory information, learn long-sequential data, and form information relation data in LTM. The combination of LSTM and GRU aims to overcome RNN’s weakness in LTM. The LSTM-GRU is combined by adding GRU to the data generated from LSTM. The combination of LSTM and GRU creates a better performance algorithm for addressing the LTM problem.

A novel machine learning and deep learning semi-supervised approach for automatic detection of InSAR-based deformation hotspots

Over the past two decades, Interferometric synthetic aperture radar (InSAR) has been invaluable for studying earth surface deformation and related effects. Deformation maps generated through multi-temporal InSAR processing methods are however difficult to interpret accurately by general individual users, decision-makers, and non-domain experts owing to the volume, variety, and velocity they are produced. This paper proposes a semi-supervised machine learning based information mining approach to simplify these deformation maps and detect hotspots by extracting prominent signals from time series deformation. The approach initially combines two machine learning based clustering methods named time series k-means (TSKM) and Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithms to derive clusters with unique spatiotemporal deformation behavior, using time series deformation output generated from Wavelet-based InSAR (WabInSAR) method. Clustering results generated from this unsupervised machine learning approach are later used as training labels to develop two deep learning models, one using long short term memory (LSTM) networks alone and another using a combination of LSTM and single-layer perceptron for supervised training. The developed LSTM and LSTM + Perceptron models efficiently learn from the cluster labels, reaching an accuracy of 97.3 %. Further, the deep learning models significantly reduce the computational time from orders of days (∼5) to hours (∼2) while training and from hours to minutes during prediction. We evaluate the developed approach over Los Angeles, a highly challenging area affected by umpteen deformation events that are challenging to categorize. The outcome of the proposed approach produces hotspots of deforming areas in Los Angeles, providing a generalized and more precise picture of events, much appreciable to non-domain experts. The approach can augment any of the multi-temporal InSAR processing chains and is applicable to different deformation prone sites, aiding in derivation of deformation hotspots from time series deformation maps.

Traffic Flow Forecast using Time Series Analysis based on Machine Learning Algorithm

Time series forecasting plays a critical role in predicting future trends and making informed decisions across various domains. In this research paper, we present a multi-model approach to improve the accuracy and robustness of time series forecasting. Our methodology involves integrating two powerful models, Long Short-Term Memory (LSTM) and Seasonal Autoregressive Integrated Moving Average (SARIMA), to capture complex temporal patterns and seasonality in the data. We leverage real-world traffic data from different junctions and employ a comprehensive pipeline for data preprocessing, including data selection, resampling, and sequence generation. Our study focuses on four junctions, and for each junction, we individually train LSTM models, followed by SARIMA models for comparison. The LSTM models are designed to capture long-term dependencies in the time series data, while the SARIMA models are tailored to handle seasonality and autocorrelation. We implement a custom sequence generation process and split the dataset into training and testing sets for model evaluation. The LSTM models are trained using Pytorch, optimizing them for accurate short-term predictions. Our results show that the combination of LSTM and SARIMA models yields superior forecasting performance compared to using each model individually. We present a detailed analysis of the forecasted results, including Root Mean Square Error (RMSE) calculations and visualizations to demonstrate the effectiveness of our multi-model approach. This research contributes to the field of time series forecasting by showcasing the benefits of combining deep learning and classical statistical methods. The proposed approach provides a flexible and robust framework applicable to various time series prediction tasks, offering improved accuracy and reliability in forecasting future trends.

A System to Detect Oilwell Anomalies Using Deep Learning and Decision Diagram Dual Approach

Summary Detecting unexpected events is a field of interest in oil and gas companies to improve operational safety and reduce costs associated with nonproductive time (NPT) and failure repair. This work presents a system for real-time monitoring of unwanted events using the production sensor data from oil wells. It uses a combination of long short-term memory (LSTM) autoencoder and a rule-based analytic approach to perform the detection of anomalies from sensor data. Initial studies are conducted to determine the behavior and correlations of pressure and temperature values for the most common combinations of well valve states. The proposed methodology uses pressure and temperature sensor data, from which a decision diagram (DD) classifies the well status, and this response is applied to the training of neural networks devoted to anomaly detection. Data sets related to several operations in wells located at different oil fields are used to train and validate the dual approach presented. The combination of the two techniques enables the deep neural network to evolve constantly through the normal data collected by the analytical method. The developed system exhibits high accuracy, with true positive detection rates exceeding 90% in the early stages of anomalies identified in both simulated and actual well production scenarios. It was implemented in more than 20 floating production, storage, and offloading (FPSO) vessels, monitoring more than 250 production/injection subsea wells, and can be applied both in real-time operation and in testing scenarios.

A surrogate non-intrusive reduced order model of quasi-geostrophic turbulence dynamics based on a combination of LSTM and different approaches of DMD

Mathematical modeling is applied to study phenomena and system behavior.In various engineering fields, many physical phenomena are illustrated using a set of differential equations.In many real-world applications, the mathematical models are very complex, and numerical simulations in high-dimensional systems are challenging.Examples of these problems are large-scale physical problems such as geophysical, which have high temporal and spatial variations.In these problems, model order reduction is a useful method for achieving an appropriate approximation because it can significantly decrease computational costs.Deep learning has recently been used to explore information from data and make predictions.There are several methods for dimensionality reduction.In this paper, we combine the dynamic mode decomposition (DMD) and the long short-term memory (LSTM) network.This is because LSTM can predict nonlinear systems and time series data.We use LSTM and DMD to predict nonlinear systems and reduce dimensions, respectively.Four common DMD schemes have been applied for dimensionality reduction.The common geophysical dataset has been used to evaluate the performance of the proposed method.Finally, we compare the variations of the modal coefficients which are obtained from snapshots projection and the reduced-order model.These results show the high accuracy of our proposed method.One of the things that is important is the time complexity of algorithm implementation.The time complexity of the proposed method is 10 times faster when 15 modes are used for modeling than when all features are used.

Image Orator - Image to Speech Using CNN, LSTM and GTTS

Abstract: This report presents an image to audio system that utilizes a combination of Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNN) for image captioning and Google Text-to-Speech (GTTS) for generating audio output. The aim of the project is to create an accessible system that converts images into descriptive audio signals for visually impaired individuals. The proposed system has the potential to provide meaningful context and information about the image through descriptive audio output, making it easier for visually impaired individuals to engage with visual content. In conclusion, the proposed image to audio system, which combines LSTM and CNN for image captioning and GTTS for audio generation, is a promising approach to making visual content more accessible to individuals with visual impairments. Future work may involve exploring different neural network architectures, optimising the system for real-time performance, and incorporating additional audio features to enhance the overall user experience.

Sentiment analysis of Indonesian datasets based on a hybrid deep-learning strategy

Various attempts have been conducted to improve the performance of text-based sentiment analysis. These significant attempts have focused on text representation and model classifiers. This paper introduced a hybrid model based on the text representation and the classifier models, to address sentiment classification with various topics. The combination of BERT and a distilled version of BERT (DistilBERT) was selected in the representative vectors of the input sentences, while the combination of long short-term memory and temporal convolutional networks was taken to enhance the proposed model in understanding the semantics and context of each word. The experiment results showed that the proposed model outperformed various counterpart schemes in considered metrics. The reliability of the proposed model was confirmed in a mixed dataset containing nine topics.

A short-term prediction model of global ionospheric VTEC based on the combination of long short-term memory and convolutional long short-term memory

A short-term prediction model of global ionospheric VTEC based on the combination of long short-term memory and convolutional long short-term memory

Anomaly-based Network Intrusion Detection System using Deep Intelligent Technique

Background and objectives: Computer systems and network infrastructures are still exposed to many security risks and cyber-attack vulnerabilities despite advancements of information security. Traditional signature-based intrusion detection systems and security solutions by matching rule-based mechanism and prior knowledge are insufficient of fully protecting computer networks against novel attacks. For this purpose, Anomaly-based Network Intrusion Detection System (A-NIDS) as cyber security tool is considered for identifying and detecting anomalous behavior in the flow-based network traffic alongside with firewalls and other security measures. The main objective of the research is to improve the detection rate and reduce false-positive rates of the classifier using anomaly-based technique. Methods: an intelligent technique using deep learning algorithm and mutual information feature selection (MIFS) method to select optimal features on the benchmark datasets. Proposed method accurately capable of classifying normal and anomalous states of the data packets in a comprehensive way by combination of Long-Short term memory (LSTM) algorithm and MIFS method. Results: The model achieved encouraging results in terms of accuracy 99.79%, 0.002 false-positive rate with minimum time compared to other models recorded only 81.75s on CSE-CIC-IDS2018 dataset. At the end of the study, comparative studies are conducted to verify the effectiveness of proposed method on three realistic and latest intrusion detection datasets, named CSE_CIC-IDS2018, CIC-IDS2017, and NF-CSE-CIC-IDS2018 dataset. Conclusions: Proposed model in a combination of LSTM NN and Feature selection method (MIFS) increased detection rate and reduced false-positive alarms, also the model able to detect low frequent attacks while other existing models are suffering from.

Deep Learning Methods for Arabic Autoencoder Speech Recognition System for Electro-Larynx Device

Recent advances in speech recognition have achieved remarkable performance comparable with human transcribers’ abilities. But this significant performance is not the same for all the spoken languages. The Arabic language is one of them. Arabic speech recognition is bounded to the lack of suitable datasets. Artificial intelligence algorithms have shown promising capabilities for Arabic speech recognition. Arabic is the official language of 22 countries, and it has been estimated that 400 million people speak the Arabic language worldwide. Speech disabilities have been one of the expanding problems in the last decades, even in kids. Some devices can be used to generate speech for those people. One of these devices is the Servox Digital Electro-Larynx (EL). In this research, we developed an autoencoder with a combination of long short-term memory (LSTM) and gated recurrent units (GRU) models to recognize recorded signals from Servox Digital EL Electro-Larynx. The proposed framework consisted of three steps: denoising, feature extraction, and Arabic speech recognition. The experimental results show 95.31% accuracy for Arabic speech recognition with the proposed model. In this research, we evaluated different combinations of LSTM and GRU for constructing the best autoencoder. A rigorous evaluation process indicates better performance with the use of GRU in both encoder and decoder structures. The proposed model achieved a 4.69% word error rate (WER). Experimental results confirm that the proposed model can be used for developing a real-time app to recognize common Arabic spoken words.

Real-Time Economic Dispatch of CHP Systems with Battery Energy Storage for Behind-the-Meter Applications

The use of combined heat and power (CHP) systems has recently increased due to their high combined efficiency and low emissions. Using CHP systems in behind-the-meter applications, however, can introduce some challenges. Firstly, the CHP system must operate in load-following mode to prevent power export to the grid. Secondly, if the load drops below a predefined threshold, the engine will operate at a lower temperature and hence lower efficiency, as the fuel is only half-burnt, creating significant emissions. The aforementioned issues may be solved by combining CHP with a battery energy storage system (BESS); however, the dispatch of CHP and BESS must be optimised. Offline optimisation methods based on load prediction will not prevent power export to the grid due to prediction errors. Therefore, this paper proposes a real-time Energy Management System (EMS) using a combination of Long Short-Term Memory (LSTM) neural networks, Mixed Integer Linear Programming (MILP), and Receding Horizon (RH) control strategy. The RH control strategy is suggested to reduce the impact of prediction errors and enable real-time implementation of the EMS exploiting actual generation and demand data on the day. Simulation results show that the proposed method can prevent power export to the grid and reduce the operational cost by 8.75% compared to the offline method.

Forecasting for Chaotic Time Series Based on GRP-lstmGAN Model: Application to Temperature Series of Rotary Kiln

Rotary kiln temperature forecasting plays a significant part of the automatic control of the sintering process. However, accurate forecasts are difficult owing to the complex nonlinear characteristics of rotary kiln temperature time series. With the development of chaos theory, the prediction accuracy is improved by analyzing the essential characteristics of time series. However, the existing prediction methods of chaotic time series cannot fully consider the local and global characteristics of time series at the same time. Therefore, in this study, the global recurrence plot (GRP)-based generative adversarial network (GAN) and the long short-term memory (LSTM) combination method, named GRP-lstmGAN, are proposed, which can effectively display important information about time scales. First, the data is subjected to a series of pre-processing operations, including data smoothing. Then, transforming one-dimensional time series into two-dimensional images by GRP makes full use of the global and local information of time series. Finally, the combination of LSTM and improves GAN models for temperature time series prediction. The experimental results show that our model is better than comparison models.

A Garlic-Price-Prediction Approach Based on Combined LSTM and GARCH-Family Model

The frequent and sharp fluctuations in garlic prices seriously affect the sustainable development of the garlic industry. Accurate prediction of garlic prices can facilitate correct evaluation and scientific decision making by garlic practitioners, thereby avoiding market risks and promoting the healthy development of the garlic industry. To improve the prediction accuracy of garlic prices, this paper proposes a garlic-price-prediction method based on a combination of long short-term memory (LSTM) and multiple generalized autoregressive conditional heteroskedasticity (GARCH)-family models for the nonstationary and nonlinear characteristics of garlic-price series. Firstly, we obtain volatility characteristic information such as the volatility aggregation of garlic-price series by constructing GARCH-family models. Then, we leverage the LSTM model to learn the complex nonlinear relationships between the garlic-price series and the volatility characteristic information of the series, and predict the garlic price. We applied the proposed model to a real-world garlic dataset. The experimental results show that the prediction performance of the combined LSTM and GARCH-family model containing volatility characteristic information of garlic price is generally better than those of the separate models. The combined LSTM model incorporating GARCH and PGARCH models (LSTM-GP) had the best performance in predicting garlic price in terms of evaluation indexes, such as mean absolute error, root mean-square error, and mean absolute percentage error. The combined model of LSTM-GARCH provides the best results in garlic price prediction and can provide support for garlic price prediction.

Research on the Filtering and Classification Method of Interactive Music Education Resources Based on Neural Network.

This work intends to classify and integrate music genres and emotions to improve the quality of music education. This work proposes a web image education resource retrieval method based on semantic network and interactive image filtering for a music education environment. It makes a judgment on these music source data and then uses these extracted feature sequences as the emotions expressed in the model of the combination of Long Short-Term Memory (LSTM) and Attention Mechanism (AM), thus judging the emotion category of music. The emotion recognition accuracy has increased after improving LSTM-AM into the BiGR-AM model. The greater the difference between emotion genres is, the easier it is to analyze the feature sequence containing emotion features, and the higher the recognition accuracy is. The classification accuracy of the excited, relieved, relaxed, and sad emotions can reach 76.5%, 71.3%, 80.8%, and 73.4%, respectively. The proposed interactive filtering method based on a Convolutional Recurrent Neural Network can effectively classify and integrate music resources to improve the quality of music education.