Bidirectional Long-Short Term Research Articles

Innovations in Time Related Expression Recognition Using LSTM Networks

The proposed architecture leverages the strengths of both Convolutional Neural Network (CNN) and Bidirectional Long Short-Term (BLSTM) to create a robust model for temporal expression recognition in clinical texts. The CNN component effectively captures morphological and orthographic features at the character level, which enriches the semantic understanding of complex medical terminologies that are often abbreviated or have unique suffixes and prefixes. The BLSTM component excels in capturing long-range dependencies in text, which is crucial for understanding the context in which temporal expressions occur. By integrating these models with a CRF layer, the system not only predicts discrete labels but also ensures that the sequence of predicted labels is coherent and contextually appropriate, addressing the limitations of models that predict labels independently. The integration of pre-trained biomedical word vectors provides significant contextual grounding tailored to the medical domain, enhancing the model's ability to discern and interpret the nuances of medical language. This is crucial in clinical contexts where accurate interpretation of temporal phrases can be critical for patient management and treatment timelines. Further, experiments conducted on the dataset validate the effectiveness of the proposed model, demonstrating a notable improvement over traditional methods that rely heavily on hand-crafted features and rule-based approaches. Future work could explore the adaptability of this model to other subdomains of the medical field and its efficacy in processing multilingual texts, potentially increasing its applicability in global healthcare settings, with further refinement of the neural architecture and optimization of training strategies potentially yielding even better performance and faster processing times essential for real-time clinical decision support systems.

Assessment and prediction of significant wave height using hybrid CNN-BiLSTM deep learning model for sustainable wave energy in Australia

Wave energy is regarded as one of the powerful renewable energy sources and depends on the assessment of significant wave height (Hs) for feasibility. Hence, this study explores the potential of wave energy by assessing and predicting Hs for two study sites in Queensland (Emu Park and Townsville), Australia. Assessment and prediction of Hs is extremely important for reliable planning, cost management and implementation of wave energy projects. The study utilized oceanic datasets based on wave measurements obtained from buoys along coastal regions of Queensland that are transmitted to nearby receiver stations. The parameters of the datasets include maximum wave height, zero up crossing wave period, peak energy wave period and sea surface temperature to accurately predict Hs. A new hybrid Convolutional Neural Network (CNN) and Bidirectional Long Short Term (BiLSTM) deep learning model with Multivariate Variational Mode Decomposition (MVMD) is developed which is benchmarked by Multi-Layer Perceptron (MLP), Random Forest (RF) and Categorical Boosting (CatBoost) to compare the performance. All models attain relatively high-performance results. The MVMD-CNN-BiLSTM attains slightly better performance values for both study sites among all developed models with highest correlation values of 0.9957 and 0.9986 for Emu Park and Townsville, respectively. Other performance evaluation metrics were also higher for MVMD-CNN-BiLSTM with lowest error values in comparison to the benchmark models. The annual mean of Hs is also computed to compare and obtain an insight with a linear projection. There is a greater ocean wave energy potential for Emu Park for a 10-year period with a projected mean Hs of 0.865 m in comparison to Townsville where the projected mean was of 0.665 m.

Limit and screen sequences with high degree of secondary structures in DNA storage by deep learning method

BackgroundIn single-stranded DNAs/RNAs, secondary structures are very common especially in long sequences. It has been recognized that the high degree of secondary structures in DNA sequences could interfere with the correct writing and reading of information in DNA storage. However, how to circumvent its side-effect is seldom studied. MethodAs the degree of secondary structures of DNA sequences is closely related to the magnitude of the free energy released in the complicated folding process, we first investigate the free-energy distribution at different encoding lengths based on randomly generated DNA sequences. Then, we construct a bidirectional long short-term (BiLSTM)-attention deep learning model to predict the free energy of sequences. ResultsOur simulation results indicate that the free energy of DNA sequences at a specific length follows a right skewed distribution and the mean increases as the length increases. Given a tolerable free energy threshold of 20 kcal/mol, we could control the ratio of serious secondary structures in the encoding sequences to within 1% of the significant level through selecting a feasible encoding length of 100 nt. Compared with traditional deep learning models, the proposed model could achieve a better prediction performance both in the mean relative error (MRE) and the coefficient of determination (R2). It achieved MRE = 0.109 and R2 = 0.918 respectively in the simulation experiment. The combination of the BiLSTM and attention module can handle the long-term dependencies and capture the feature of base pairing. Further, the prediction has a linear time complexity which is suitable for detecting sequences with severe secondary structures in future large-scale applications. Finally, 70 of 94 predicted free energy can be screened out on a real dataset. It demonstrates that the proposed model could screen out some highly suspicious sequences which are prone to produce more errors and low sequencing copies.

Coswara: A respiratory sounds and symptoms dataset for remote screening of SARS-CoV-2 infection

This paper presents the Coswara dataset, a dataset containing diverse set of respiratory sounds and rich meta-data, recorded between April-2020 and February-2022 from 2635 individuals (1819 SARS-CoV-2 negative, 674 positive, and 142 recovered subjects). The respiratory sounds contained nine sound categories associated with variants of breathing, cough and speech. The rich metadata contained demographic information associated with age, gender and geographic location, as well as the health information relating to the symptoms, pre-existing respiratory ailments, comorbidity and SARS-CoV-2 test status. Our study is the first of its kind to manually annotate the audio quality of the entire dataset (amounting to 65 hours) through manual listening. The paper summarizes the data collection procedure, demographic, symptoms and audio data information. A COVID-19 classifier based on bi-directional long short-term (BLSTM) architecture, is trained and evaluated on the different population sub-groups contained in the dataset to understand the bias/fairness of the model. This enabled the analysis of the impact of gender, geographic location, date of recording, and language proficiency on the COVID-19 detection performance.

Prediction of Mean Sea Level with GNSS-VLM Correction Using a Hybrid Deep Learning Model in Australia

The prediction of sea level rise is extremely important for improved future climate change mitigation and adaptation strategies. This study uses a hybrid convolutional neural Network (CNN) and a bidirectional long short-term (BiLSTM) model with successive variational mode decomposition (SVMD) to predict the absolute sea level for two study sites in Australia (Port Kembla and Milner Bay). More importantly, the sea level measurements using a tide gauge were corrected using Global Navigation Satellite System (GNSS) measurements of the vertical land movement (VLM). The SVMD-CNN-BiLSTM model was benchmarked by a multi-layer perceptron (MLP), support vector regression (SVR) and gradient boosting (GB). The SVMD-CNN-BiLSTM model outperformed all the comparative models with high correlation values of more than 0.95 for Port Kembla and Milner Bay. Similarly, the SVMD-CNN-BiLSTM model achieved the highest values for the Willmott index, the Nash–Sutcliffe index and the Legates and McCabe index for both study sites. The projected linear trend showed the expected annual mean sea rise for 2030. Using the current trend, Port Kembla was projected to have an MSL value of 1.03 m with a rate rise of approx. 4.5 mm/year. The rate of the MSL for Milner Bay was comparatively lower with a value of approx. 2.75 mm/year and an expected MSL value of 1.27 m for the year 2030.

Interaction-feedback network for multi-task daily activity forecasting

The category and occurrence time of daily activity are two basic tasks of daily activity forecasting in a smart home. Multi-task learning has been employed for daily activity forecasting. However, the category and occurrence time of daily activity are weakly correlated to each other, and the loss functions of the two tasks are difficult to achieve at the same time in multi-task learning. In this paper, a daily activity forecasting approach based on an interaction-feedback network is presented. The proposed method combines the multi-head attention mechanism, convolutional neural networks (CNNs), and bidirectional long short-term (Bi-LSTM) memory units to generate a parallel multi-task forecasting model. In addition, a feedback mechanism is introduced to supervise the learning of the two involved tasks. Public datasets are utilized to evaluate the proposed method. Experimental findings show its effectiveness for multi-task daily activity forecasting in terms of precision, recall, F-score, MAE, RMSE, and R2.

Sentence opinion mining model for fusing target entities in official government documents

<abstract> <p>When drafting official government documents, it is necessary to firmly grasp the main idea and ensure that any positions stated within the text are consistent with those in previous documents. In combination with the field's demands, By taking advantage of suitable text-mining techniques to harvest opinions from sentences in official government documents, the efficiency of official government document writers can be significantly increased. Most existing opinion mining approaches employ text classification methods to directly mine the sentential text of official government documents while disregarding the influence of the objects described within the documents (i.e., the target entities) on the sentence opinion categories. To address these issues, this study proposes a sentence opinion mining model that fuses the target entities within documents. Based on the Bi-directional long short-term (BiLSTM) and attention mechanisms, the model fully considers the attention given by a official government document's target entity to different words within the corresponding sentence text, as well as the dependency between words of the sentence. The model subsequently fuses two by using feature vector fusion to obtain the final semantic representation of the text, which is then classified using a fully connected network and softmax function. Experimental results based on a dataset of official government documents show that the model significantly outperforms baseline models such as Text-convolutional neural network (TextCNN), recurrent neural network (RNN), and BiLSTM.</p> </abstract>

Forecasting the Risk Factor of Frontier Markets: A Novel Stacking Ensemble of Neural Network Approach

Forecasting the risk factor of the financial frontier markets has always been a very challenging task. Unlike an emerging market, a frontier market has a missing parameter named “volatility”, which indicates the market’s risk and as a result of the absence of this missing parameter and the lack of proper prediction, it has almost become difficult for direct customers to invest money in frontier markets. However, the noises, seasonality, random spikes and trends of the time-series datasets make it even more complicated to predict stock prices with high accuracy. In this work, we have developed a novel stacking ensemble of the neural network model that performs best on multiple data patterns. We have compared our model’s performance with the performance results obtained by using some traditional machine learning ensemble models such as Random Forest, AdaBoost, Gradient Boosting Machine and Stacking Ensemble, along with some traditional deep learning models such as Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM) and Bidirectional Long Short-Term (BiLSTM). We have calculated the missing parameter named “volatility” using stock price (Close price) for 20 different companies of the frontier market and then made predictions using the aforementioned machine learning ensemble models, deep learning models and our proposed stacking ensemble of the neural network model. The statistical evaluation metrics RMSE and MAE have been used to evaluate the performance of the models. It has been found that our proposed stacking ensemble neural network model outperforms all other traditional machine learning and deep learning models which have been used for comparison in this paper. The lowest RMSE and MAE values we have received using our proposed model are 0.3626 and 0.3682 percent, respectively, and the highest RMSE and MAE values are 2.5696 and 2.444 percent, respectively. The traditional ensemble learning models give the highest RMSE and MAE error rate of 20.4852 and 20.4260 percent, while the deep learning models give 15.2332 and 15.1668 percent, respectively, which clearly states that our proposed model provides a very low error value compared with the traditional models.

Sentiment Analysis of COVID data extracted via Twitter

Different types of social media sites exist, wherein some of them are LinkedIn, Twitter, Facebook, Instagram, WhatsApp, etc. As the number of social media users increases, the opportunity for the user to express their feelings also increases. Twitter is a choice of many users as it not only allows the users to express their thoughts but to interact with official accounts (PMO, Defense Ministry) which can be seen with a verified tick on the website. In this thesis titled ‘Sentiment Analysis of COVID data extracted via Twitter’, multiple machine learning and deep learning techniques have been researched and implemented to perform sentiment analysis. Moreover, a novel approach using deep learning architecture has been proposed. It is based on a combination of Bidirectional Long Short Term (BiLSTM) neural networks and Convolution Neural Networks (CNN). Prior to implementing the algorithms, the data is acquired by using web-scraping techniques and/or public APIs pertaining to Twitter. A comparative analysis of the efficiency and performance of the proposed technique along with other existing approaches discovered during the literature review phase is also presented. KEYWORDS: Sentiment analysis, machine learning, deep learning, Natural Language Processing

Wearables and Deep Learning Classify Fall Risk From Gait in Multiple Sclerosis.

Falls are a significant problem for persons with multiple sclerosis (PwMS). Yet fall prevention interventions are not often prescribed until after a fall has been reported to a healthcare provider. While still nascent, objective fall risk assessments could help in prescribing preventative interventions. To this end, retrospective fall status classification commonly serves as an intermediate step in developing prospective fall risk assessments. Previous research has identified measures of gait biomechanics that differ between PwMS who have fallen and those who have not, but these biomechanical indices have not yet been leveraged to detect PwMS who have fallen. Moreover, they require the use of laboratory-based measurement technologies, which prevent clinical deployment. Here we demonstrate that a bidirectional long short-term (BiLSTM) memory deep neural network was able to identify PwMS who have recently fallen with good performance (AUC of 0.88) based on accelerometer data recorded from two wearable sensors during a one-minute walking task. These results provide substantial improvements over machine learning models trained on spatiotemporal gait parameters (21% improvement in AUC), statistical features from the wearable sensor data (16%), and patient-reported (19%) and neurologist-administered (24%) measures in this sample. The success and simplicity (two wearable sensors, only one-minute of walking) of this approach indicates the promise of inexpensive wearable sensors for capturing fall risk in PwMS.

A Multi-Layer LSTM-Time-Density-Softmax (LDS) Approach for Protein Structure Prediction Using Deep Learning

Introduction: The primary structure of the protein is a polypeptide chain made up of a sequence of amino acids. What happens due to interaction between the atoms of the backbone is that it forms within a polypeptide folded structure, which is very much within the secondary structure. These alignments can be made more accurate by the inclusion of secondary structure information. Objective: It is difficult to identify the sequence information embedded in the secondary structure of the protein. However, Deep learning methods can be used for solving the identification of the sequence information in the protein structures. Methods: The scope of the proposed work is to increase the accuracy of identifying the sequence information in the primary structure and the tertiary structure, thereby increasing the accuracy of the predicted Protein Secondary Structure (PSS). In this proposed work, homology is eliminated by a Recurrent Neural Network (RNN) based network that consists of three layers, namely bi-directional Long Short Term Memory (LSTM), time distributed layer and Softmax layer. Results: The proposed LDS model achieves an accuracy of approximately 86% for the prediction of the three-state secondary structure of the protein. Conclusion: The gap between the number of protein primary structures and secondary structures is huge and increasing. Machine learning is trying to reduce this gap. In most of the other pre attempts in predicting the secondary structure of proteins, the data is divided according to the homology of the proteins. This limits the efficiency of the predicting model and the inputs given to such models. Hence, in our model, homology has not been considered while collecting the data for training or testing out model. This has led to our model to not be affected by the homology of the protein fed to it and hence remove that restriction, so any protein can be fed to it.

Renewable electricity generation proposed pathways for the US and China

In this study, we investigate the direction of the causal relationship between renewable electricity generation (REG) and economic growth for the US and China. The results show a unidirectional causality running from REG to economic growth for the US and China. Furthermore, a bi-directional long short-term (Bi-LSTM) algorithm is formulated to propose REG pathways based on four scenarios. First, we allow the Bi-LSTM network to make business-as-usual (BAU) predictions. The results show that the US and China’s REG as a percentage of total electricity output will hit ∼17.769 and ∼34.688 by 2030, respectively. Second, we set a 30% incremental target and estimate that the US and China REG will hit ∼17.198 and ∼31.105 by 2030, respectively. Third, the results from setting an incremental target of 50% depicts that the US and China REG will hit ∼19.844 and ∼35.891 by 2030, respectively. Finally, the 100% incremental target from the 2015 generation shows that the US and China REG are expected to hit ∼26.458 and ∼47.854 by 2030, respectively. As a policy implication, if investments in renewable electricity generation are not intensified, the US and China stand a risk of not meeting the pathway targets proposed.

Deep Learning based Intelligent E-mail Autoresponder

Handling huge volume of emails is a very challenging task in the customer support applications and an automated email responding system will be of great help. In this paper, an intelligent email autoresponder system is developed which either attempts to respond to the incoming emails from various category of customers or generate token for service request to address the issue manually by an expert member. First, based on the content the system has to predict whether the mail belong to the category of auto responding or to invoke a service request. This classification of email is carried out using long short term (LSTM) and bi-directional LSTM networks (Bi-LSTM) networks and the classification performance is analyzed. The results presented in this work show that the Bi-LSTM classifier outperforms LSTM network.

Implementation of generative adversarial network-CLS combined with bidirectional long short-term memory for lithium-ion battery state prediction

• Combination of GAN-CLS and bidirectional-long short term (LSTM). • Battery data generation from images using GAN-CLS. • Bidirectional LSTM model training for predicting the battery internal state. • Provision of the reduction time, cost and a more accurate state estimation. This study newly introduces a complementary cooperative algorithm considering generative adversarial network (GAN)-Conditional Latent Space (CLS) combined with bidirectional long short-term memory (BLSTM) for improved and efficient lithium-ion rechargeable battery state prediction. The GAN-CLS algorithm, which is an advanced method of GAN, can generate corresponding images from an input label description. Long short-term memory (LSTM) is a specific recurrent neural network (RNN) architecture that can predict sequences more accurately than conventional RNNs. In terms of battery state prediction, the combination of two methods (GAN-CLS and LSTM) surely provides more improved and efficient rechargeable battery state prediction in contrast to conventional state predictors. The procedure of this study is as follows. First, we propose methods to enhance the data from battery charge/discharge by converting prepared data to images; then, the GAN-CLS method is used to generate corresponding battery data from previous images. Subsequently, the generated data is used to train the BLSTM model. Finally, the trained model is used to predict the battery state. By various experiments and verification, it is concluded that the proposed study can be a good solution for rechargeable battery state prediction (reduction of the time cost 50 times in modeling and 20 times in train/test, provision of a more accurate prediction mean square error (MSE) smaller than 0.0025 and the average MSE less than 0.0013).

Electric Load Forecasting with Deep Machine Learning

The most commonly used form of energy in houses, factories, buildings and agriculture is the electrical energy, however, in recent years, there has been an increase in electrical energy demand due to technology advancements and rise in population, therefore an appropriated forecasting system must be developed to predict these demands as accurately as possible. For this purpose, five models were selected, they are Bidirectional-Long Short Term Memory (Bi-LSTM), Feed Forward Neural Network (FFNN), Long Short Term Memory (LSTM), Nonlinear Auto Regressive network with eXogenous inputs (NARX) and Multiple Linear Regression (MLR). This paper will demonstrate the development of these selected models using MATLAB and an android mobile application, which is used to visualize and interact with the data. The performance of the selected models was evaluated by performing the Mean Absolute Percent Error (MAPE), the selected historical data used to perform the MAPE was obtained from Toronto, Canada and Tasmania, Australia, where the year 2006 until 2016 was used as training data and the year 2017 was used to test the MAPE of the historical data with the models’ data. It is observed that the NARX model had the least MAPE for both the regions resulting in 1.9% for Toronto, Canada and 2.9% for Tasmania, Australia. Google cloud is used as the IoT (Internet of Things) platform for NARX data model, the 2017 datasets is converted to JavaScript Object Notation (JSON) file using JavaScript programming language, for data visualization and analysis for the android mobile application.

Two Stream Deep Neural Network for Sequence-Based Urdu Ligature Recognition

Urdu text is a complex cursive script and poses a challenge for recognition by OCR systems due to its large number of ligatures and cursive style. In literature, several techniques have been proposed to recognize Urdu ligatures. However, we have investigated that, suitable challenging datasets and the consequently higher recognition rate is needed for ligature recognition. In this paper, a hybrid model based on the holistic approach is adopted for the recognition of Urdu ligatures (compound characters). More than 3800 unique ligatures were used to generate 46K (38K training, 7K testing) synthetic ligatures with 9 different kinds of transformations along with the normal ligatures. Each ligature is processed through two streams of Deep Neural Networks, namely Alexnet and Vgg16 to obtain a unique set of features corresponding to each net. These features are fused and then used as an input to double layer Bidirectional Long Short Term (BLSTM) network for learning a model. The learned model maps ligature images to their corresponding sequence of individual Urdu characters. In the proposed methodology output is in the editable Urdu-script format. The proposed model was evaluated and have shown an accuracy of 97% on the training dataset and 80% on more than 7K parametrically different query ligatures (test-set).