Attention-based LSTM Model Research Articles

Automated Medical Image Captioning with Soft Attention-Based LSTM Model Utilizing YOLOv4 Algorithm

Automated Medical Image Captioning with Soft Attention-Based LSTM Model Utilizing YOLOv4 Algorithm

Utilizing an Attention-Based LSTM Model for Detecting Sarcasm and Irony in Social Media

Sarcasm and irony represent intricate linguistic forms in social media communication, demanding nuanced comprehension of context and tone. In this study, we propose an advanced natural language processing methodology utilizing long short-term memory with an attention mechanism (LSTM-AM) to achieve an impressive accuracy of 99.86% in detecting and interpreting sarcasm and irony within social media text. Our approach involves innovating novel deep learning models adept at capturing subtle cues, contextual dependencies, and sentiment shifts inherent in sarcastic or ironic statements. Furthermore, we explore the potential of transfer learning from extensive language models and integrating multimodal information, such as emojis and images, to heighten the precision of sarcasm and irony detection. Rigorous evaluation against benchmark datasets and real-world social media content showcases the efficacy of our proposed models. The outcomes of this research hold paramount significance, offering a substantial advancement in comprehending intricate language nuances in digital communication. These findings carry profound implications for sentiment analysis, opinion mining, and an enhanced understanding of social media dynamics.

Aspect-Based Sentiment Analysis

Aspect-Based Sentiment Analysis (ABSA) is an advanced NLP application that aims to identify aspect terms present in the given review and predict the sentiment associated with those aspect terms. ABSA is better than sentence-based sentiment classification because it considers the aspect terms present in the reviews to determine the sentiment rather than considering the individual sentence. Entrepreneurs could make use of ABSA to understand the customers' opinions about different aspects of their products or services. The task of Aspect-based sentiment analysis can be divided into two subtasks: Aspect Term Extraction (ATE) and Aspect Term Sentiment Classification (ATSC). In this paper, an SVM model is proposed for the task of ATE, and an Attention based LSTM model is proposed for the task of ATSC. The proposed models will be trained and tested on SemEval-2014 dataset.

LSTM-based throughput prediction for LTE networks

Throughput prediction is crucial for reducing latency in time-critical services. We study the attention-based LSTM model for predicting future throughput. First, we collected the TCP logs and throughputs in LTE networks and transformed them using CUBIC and BBR trace log data. Then, we use the sliding window method to create input data for the prediction model. Finally, we trained the LSTM model with an attention mechanism. In the experiment, the proposed method shows lower normalized RMSEs than the other method.

Spatially Offset Raman Spectroscopy Combined with Attention-Based LSTM for Freshness Evaluation of Shrimp

Optical detection of the freshness of intact in-shell shrimps is a well-known difficult task due to shell occlusion and its signal interference. The spatially offset Raman spectroscopy (SORS) is a workable technical solution for identifying and extracting subsurface shrimp meat information by collecting Raman scattering images at different distances from the offset laser incidence point. However, the SORS technology still suffers from physical information loss, difficulties in determining the optimum offset distance, and human operational errors. Thus, this paper presents a shrimp freshness detection method using spatially offset Raman spectroscopy combined with a targeted attention-based long short-term memory network (attention-based LSTM). The proposed attention-based LSTM model uses the LSTM module to extract physical and chemical composition information of tissue, weight the output of each module by an attention mechanism, and come together as a fully connected (FC) module for feature fusion and storage dates prediction. Modeling predictions by collecting Raman scattering images of 100 shrimps within 7 days. The R2, RMSE, and RPD of the attention-based LSTM model achieved 0.93, 0.48, and 4.06, respectively, which is superior to the conventional machine learning algorithm with manual selection of the optimal spatially offset distance. This method of automatically extracting information from SORS data by Attention-based LSTM eliminates human error and enables fast and non-destructive quality inspection of in-shell shrimp.

Attention-Based Lane Change and Crash Risk Prediction Model in Highways

Lane change and crash risk prediction are critical technologies for autonomous driving. An attention-based LSTM model is proposed in this paper for lane change behavior prediction in highways, considering both the surrounding vehicles’ information around the target vehicle at the current time and the historical trajectory data of the vehicle, and it shows higher accuracy and better interpretability than other models. There are two sections in this prediction model, including a pre-judgment model based on the C4.5 decision tree and bagging ensemble learning, and a multi-step lane change prediction model of LSTM with attention mechanism. In addition, the model is applied in the NGSIM datasets to test practicability and accuracy, and the precision of left lane change and right lane change is 98% and 94% respectively 1 second before the vehicle changes lane. Finally, to judge the safety of vehicles in the driving process, a crash risk prediction model based on Time-To-Collision is proposed, which further verifies the effectiveness of the method.

Multi-scale local cues and hierarchical attention-based LSTM for stock price trend prediction

Stock price trend prediction is to seek profit maximum of stock investment by estimating future stock price tendency. Nevertheless, it is still a tough task due to noisy and non-stationary properties of stock market. Thus, it is important how to relieve such negative effects and to improve prediction accuracy. In this paper, we leverage four diverse local descriptors in short durations to alleviate noisy fluctuations of stock price. In detail, piecewise aggregate approximation (PAA) collects relatively stable average values; the derivatives of short-time series reflect the change ratio of stock price; the slope implies the short-time price trend; hog-1D aggregates different oriented gradients into histograms in a statistical fashion. They provide diverse and comprehensive cues about the stock price series across different aspects. Building upon such local descriptors, we propose a multi-scale local cues and hierarchical attention-based LSTM model (MLCA-LSTM) to capture the underlying price trend patterns. It has two advantages: 1) multi-scale information is further enriched by performing different scale sliding windows over stock price series to induce diverse local descriptors, 2) temporal dependency and multi-scale interactions are jointly attended and aggregated through the hierarchical attention mechanism and multi-branch LSTM structure. Experiments on the real stock price data confirm the efficacy of the proposed model as compared to the state-of-the-art counterparts.

Asian stock markets closing index forecast based on secondary decomposition, multi-factor analysis and attention-based LSTM model

The analysis and prediction of stock markets in Asian is an important issue which can help to promote the integration and globalization of financial cooperation. However, owning to the non-stationary and complexity of the stock market fluctuation, it is challenging to predict the stock price accurately. Especially after the decomposition of the original series, how to solve the problem of pseudo information and filter the exogenous variables is often certain challenging. This paper presents a hybrid model based on secondary decomposition (SD), multi-factor analysis (MFA) and attention-based long short-term memory (ALSTM) to predict the stock market price trends of four major Asian countries. The original stock price series is preprocessed by two decomposition algorithms so as to capture further non-linear feature and better filter the noise. Multi-factor analysis is introduced as a supplement to the original data information. In the prediction stage, attention layer is added in long short-term memory model to increase the weights of effective information. Finally, four datasets about Asian stock markets and nine compared models were used to verify the performance of the proposed model. The empirical analysis results show that compared to the general long short-term memory, our proposed model can obtain higher 30% accuracy at least. The mean average percentage errors of the system were also the lowest among all models mentioned in this paper (0.612%, 0.903%, 0.606% and 0.402% respectively), which proves the effectiveness of the hybrid model.

Frame-wise detection of surgeon stress levels during laparoscopic training using kinematic data.

Excessive stress experienced by the surgeon can have a negative effect on the surgeon's technical skills. The goal of this study is to evaluate and validate a deep learning framework for real-time detection of stressed surgical movements using kinematic data. 30 medical students were recruited as the subjects to perform a modified peg transfer task and were randomized into two groups, a control group (n=15) and a stressed group (n=15) that completed the task under deteriorating, simulated stressful conditions. To classify stressed movements, we first developed an attention-based Long-Short-Term-Memory recurrent neural network (LSTM) trained to classify normal/stressed trials and obtain the contribution of each data frame to the stress level classification. Next, we extracted the important frames from each trial and used another LSTM network to implement the frame-wise classification of normal and stressed movements. The classification between normal and stressed trials using attention-based LSTM model reached an overall accuracy of 75.86% under Leave-One-User-Out (LOUO) cross-validation. The second LSTM classifier was able to distinguish between the typical normal and stressed movement with an accuracy of 74.96% with an 8-second observation under LOUO. Finally, the normal and stressed movements in stressed trials could be classified with the accuracy of 68.18% with a 16-second observation under LOUO. In this study, we extracted the movements which are more likely to be affected by stress and validated the feasibility of using LSTM and kinematic data for frame-wise detection of stress level during laparoscopic training. The proposed classifier could be potentially be integrated with robot-assisted surgery platforms for stress management purposes.

Deep machine learning-based power usage effectiveness prediction for sustainable cloud infrastructures

The expansion of online services, the advent of big data, and the development of Internet of Things (IoT) technology have led to an exponential growth in the number of data centers (DC). Within this new digital systems, DCs are the keystone. This poses major environmental problems through high energy consumption. The energy consumption growth of the DC’s systems are expected to account for 3.2% of global carbon emissions by 2025. This remarkable growth in the energy consumption of DCs represents today a major challenge for energy efficiency. To meet these challenges, Power Usage Effectiveness (PUE) can be used to measure and optimize DC energy efficiency. Thus, accurate PUE prediction is challenging. Moreover, the non-linearity of the different DCs subsystems interactions and their inter-dependencies make the prediction of PUE more difficult since it is affected by complex factors such as workload variation, in addition to weather conditions and humidity, etc.In this paper, we present three machine learning models, MLP, Resilient Backpropagation-based DNN and Attention-based LSTM, to predict PUE values. The performance of these models has been evaluated using two datasets gathered from two case studies of real DCs. For both DC cases, experimental results showed that the attention-based LSTM model outperforms resilient backpropagation-based DNN and MLP models in terms of prediction errors and PUE.

Question Text Classification Method of Tourism Based on Deep Learning Model

The Internet of Things applications are diverse in nature, and a key aspect of it is multimedia sensors and devices. These IoT multimedia devices form the Internet of Multimedia Things (IoMT). Compared with the Internet of Things, it generates a large amount of text data with different characteristics and requirements. Aiming at the problems that machine learning and single structure deep learning model cannot effectively grasp the text emotional information in text processing, resulting in poor classification effect, this paper proposes a text classification method of tourism questions based on deep learning model. First, the corpus is trained with word2vec tool based on continuous word bag model to obtain the text word vector representation. Then, the attention mechanism is introduced into the long-short term network (LSTM), and the attention-based LSTM model is constructed for text feature extraction, which highlights the impact of different words in the input text on the text emotion category. Finally, the text features are input into the Softmax classifier to obtain the probability distribution of text categories, and the model is trained combined with the cross entropy loss function. The experimental results show that the average accuracy, recall, and F value are 0.943, 0.867, and 0.903, respectively, which has better classification effect than other methods.

Potential Analysis of the Attention-Based LSTM Model in Ultra-Short-Term Forecasting of Building HVAC Energy Consumption

Predicting system energy consumption accurately and adjusting dynamic operating parameters of the HVAC system in advance is the basis of realizing the model predictive control (MPC). In recent years, the LSTM network had made remarkable achievements in the field of load forecasting. This paper aimed to evaluate the potential of using an attentional-based LSTM network (A-LSTM) to predict HVAC energy consumption in practical applications. To evaluate the application potential of the A-LSTM model in real cases, the training set and test set used in experiments are the real energy consumption data collected by Kitakyushu Science Research Park in Japan. Pearce analysis was first carried out on the source data set and built the target database. Then five baseline models (A-LSTM, LSTM, RNN, DNN, and SVR) were built. Besides, to optimize the super parameters of the model, the Tree-structured of Parzen Estimators (TPE) algorithm was introduced. Finally, the applications are performed on the target database, and the results are analyzed from multiple perspectives, including model comparisons on different sizes of the training set, model comparisons on different system operation modes, graphical examination, etc. The results showed that the performance of the A-LSTM model was better than other baseline models, it could provide accurate and reliable hourly forecasting for HVAC energy consumption.

Predicting the Cases of Hepatitis B with the A-LSTM Model

Hepatitis B is a disease caused by hepatitis B virus. It’s of great value to predict the cases of hepatitis B because of its strong infectivity and carcinogenicity. To predict the monthly new patients of hepatitis B in China accurately, a neural network with an attention-based LSTM model is proposed. Driven by the historical data provided by the Data-center of China Public Health Science, the model’s evaluation indexes of RMSE, MAPE, MAE and R-squared are 1780.495, 1.789%, 1469.208 and 0.867 respectively, while the evaluation indexes of BPNN are 3532.959, 3.311%, 2677.009 and 0.478 respectively. The result shows that A-LSTM model in this work has an excellent prediction on the monthly new patients of hepatitis B and performs much better than BPNN and other traditional time series models.

Hindi Conversational Agents for Mental Health Assistance

In current times, many people are suffering from anxiety-related problems. The problem is manifest in a developing country like India. To deal with this issue, we present an empathetic chatbot that can converse in the native language of Hindi based on the seq2seq encoder-decoder model with an attention mechanism to capture the context. Our model is trained on the benchmark Facebook Empathetic Dialogue dataset comprising of English dialogues. The resultant conversational agent can generate empathetic responses and conducts a friendly conversation with people suffering from mental health disorders. Since a vast population of Indians have Hindi as their mother tongue, this chatbot is adapted to converse in the Hindilanguage. The Google Translate API is used to perform the translation. To the best of our knowledge, no other paper has proposed such an idea of dealing with anxiety and mental stress by building an empathetic chatbot that can converse in the native Hindi language. Results obtained from our attention-based LSTM model is quite impressive and acts in favor of the argument of implementing such a chatbot at a practical level and facilitating extension to other languages. The BLEU score is used for comparing results and our attention-based conversational agent with GloVe word embedding gives a BLEU score of 0.143.

An Attention-Based LSTM Model for Stock Price Trend Prediction Using Limit Order Books

Stock price trend prediction has been a hot issue in the financial field, which has been paid much attention by both academia and industry. It is a challenging task due to the non-stationary and high volatility of the stock prices. Traditional methods for predicting stock price trends are mostly based on the historical OHLC (i.e., open, high, low, and close prices) data. However, it eliminates most of the trading information. To address this problem, in this paper, another type of stock price data, i.e., limit order books (LOBs), is used. For better exploring the relationship of the LOBs and stock price trend, inspired by the successful application of deep learning-based methods, an attention-based LSTM model is applied. The trend of stock price can be predicted by using the LOBs data of the previous day. By using the real stock price data of the China stock market, the effectiveness of the proposed model is validated by experimental results.

Constructing domain-dependent sentiment dictionary for sentiment analysis

Sentiment dictionary is of great value to sentiment analysis, which is used widely in sentiment analysis compositionality. However, the sentiment polarity and intensity of the word may vary from one domain to another. In this paper, we introduce a novel approach to build domain-dependent sentiment dictionary, SentiDomain. We propose a weak supervised neural model that aims at learning a set of sentiment clusters embedding from the sentence global representation of the target domain. The model is trained on unlabeled data with weak supervision by reconstructing the input sentence representation from the resulting representation. Furthermore, we also propose an attention-based LSTM model to address aspect-level sentiment analysis task based on the sentiment score retrieved from the proposed dictionary. The key idea is to weight-down the non-sentiment parts among aspect-related information in a given sentence. Our extensive experiments on both English and Chinese benchmark datasets have shown that compared to the state-of-the-art alternatives, our proposals can effectively improve polarity detection.

Breeze

Slow-paced biofeedback-guided breathing training has been shown to improve cardiac functioning and psychological well-being. Current training options, however, attract only a fraction of individuals and are limited in their scalability as they require dedicated biofeedback hardware. In this work, we present Breeze, a mobile application that uses a smartphone's microphone to continuously detect breathing phases, which then trigger a gamified biofeedback-guided breathing training. Circa 2.76 million breathing sounds from 43 subjects and control sounds were collected and labeled to train and test our breathing detection algorithm. We model breathing as inhalation-pause-exhalation-pause sequences and implement a phase-detection system with an attention-based LSTM model in conjunction with a CNN-based breath extraction module. A biofeedback-guided breathing training with Breeze takes place in real-time and achieves 75.5% accuracy in breathing phases detection. Breeze was also evaluated in a pilot study with 16 new subjects, which demonstrated that the majority of subjects prefer Breeze over a validated active control condition in its usefulness, enjoyment, control, and usage intentions. Breeze is also effective for strengthening users' cardiac functioning by increasing high-frequency heart rate variability. The results of our study suggest that Breeze could potentially be utilized in clinical and self-care activities.

Graph neural news recommendation with long-term and short-term interest modeling

With the information explosion of news articles, personalized news recommendation has become important for users to quickly find news that they are interested in. Existing methods on news recommendation mainly include collaborative filtering methods which rely on direct user-item interactions and content based methods which characterize the content of user reading history. Although these methods have achieved good performances, they still suffer from data sparse problem, since most of them fail to extensively exploit high-order structure information (similar users tend to read similar news articles) in news recommendation systems. In this paper, we propose to build a heterogeneous graph to explicitly model the interactions among users, news and latent topics. The incorporated topic information would help indicate a user’s interest and alleviate the sparsity of user-item interactions. Then we take advantage of graph neural networks to learn user and news representations that encode high-order structure information by propagating embeddings over the graph. The learned user embeddings with complete historic user clicks capture the users’ long-term interests. We also consider a user’s short-term interest using the recent reading history with an attention based LSTM model. Experimental results on real-world datasets show that our proposed model significantly outperforms state-of-the-art methods on news recommendation.

Two-Dimensional Attention-Based LSTM Model for Stock Index Prediction

Two-Dimensional Attention-Based LSTM Model for Stock Index Prediction

AT-LSTM: An Attention-based LSTM Model for Financial Time Series Prediction

This paper proposes an attention-based LSTM (AT-LSTM) model for financial time series prediction. We divide the prediction process into two stages. For the first stage, we apply an attention model to assign different weights to the input features of the financial time series at each time step. In the second stage, the attention feature is utilized to effectively select the relevant feature sequences as input to the LSTM neural network for the prediction in the next time frame. Our proposed framework not only solves the long-term dependence problem of time series prediction effectively, but also improves the interpretability of the time series prediction methods based on the neural network. In the end of this paper, we conducted experiments on financial time series prediction task with three real-world data sets. The experimental results show that our framework for time series pre-diction is state-of-the-art against the baselines.