BiLSTM Layer Research Articles

Text Command Intelligent Understanding for Cybersecurity Testing

Research on named entity recognition (NER) and command-line generation for network security evaluation tools is relatively scarce, and no mature models for recognition or generation have been developed thus far. Therefore, in this study, the aim is to build a specialized corpus for network security evaluation tools by combining knowledge graphs and information entropy for automatic entity annotation. Additionally, a novel NER approach based on the KG-BERT-BiLSTM-CRF model is proposed. Compared to the traditional BERT-BiLSTM model, the KG-BERT-BiLSTM-CRF model demonstrates superior performance when applied to the specialized corpus of network security evaluation tools. The graph attention network (GAT) component effectively extracts relevant sequential content from datasets in the network security evaluation domain. The fusion layer then concatenates the feature sequences from the GAT and BiLSTM layers, enhancing the training process. Upon successful NER execution, in this study, the identified entities are mapped to pre-established command-line data for network security evaluation tools, achieving automatic conversion from textual content to evaluation commands. This process not only improves the efficiency and accuracy of command generation but also provides practical value for the development and optimization of network security evaluation tools. This approach enables the more precise automatic generation of evaluation commands tailored to specific security threats, thereby enhancing the timeliness and effectiveness of cybersecurity defenses.

An Intelligent Framework for Sleep Stage Scoring by 1D-Convolutional Neural Network with Bi-LSTM Layer Using EEG signal

An Intelligent Framework for Sleep Stage Scoring by 1D-Convolutional Neural Network with Bi-LSTM Layer Using EEG signal

MFB: A Generalized Multimodal Fusion Approach for Bitcoin Price Prediction Using Time-Lagged Sentiment and Indicator Features

Bitcoin’s volatile nature has made its price prediction a sought-after mathematical model in the FinTech industry. Existing studies, however, need to look into the critical aspect of time-lagged sentiment in Bitcoin price forecasting. This omission is significant because time-lagged sentiment captures delayed market reactions that are not immediately apparent in price movements. Moreover, the correlation between time-lagged sentiment and technical indicators and the limitations of individual machine learning and deep learning models necessitates a comprehensive approach for accurate and reliable Bitcoin price predictions. This paper introduces the multimodal fusion Bitcoin (MFB), an innovative generalized multimodal fusion approach that effectively integrates BiLSTM and BiGRU layers for complex feature extraction. The model employs the BorutaShap algorithm for feature selection and utilizes attention mechanisms and spatial dropout for optimization and generalization. MFB’s training and validation use news and tweet data combined with Bitcoin technical indicators to explore the impact of time-lagged sentiment on price movements, leading to more accurate and timely market predictions. The MFB performs superior Bitcoin prediction performance, achieving 97.63% accuracy and an MAE of 0.0065. Experiments highlight MFB’s capability to outperform existing models, offering significant insights for investors in making informed decisions. MFB’s innovative methodology, particularly in next-hour Bitcoin price forecasting, marks an advancement in financial forecasting. By capturing the nuanced dynamics of market sentiment and its delayed effects, MFB is a pioneering multimodal fusion approach in the FinTech domain, revolutionizing Bitcoin price prediction.

An attention-based hybrid model for spatial and temporal sentiment analysis of COVID-19 related tweets in the contiguous United States

ABSTRACT Understanding the sentiments of social media posts can help health authorities respond to disease outbreaks, through a proxy measure of fear, confidence, and community compliance. Sentiment analysis identifies a pattern of emotion through the written word and assigns a positive, neutral, or negative value to it. As of February 2023, there were 677.7 million confirmed cases of coronavirus disease (COVID-19) and more than 6.7 million confirmed deaths. In this paper, around 170,000 COVID-19-related tweets were collected between September 2020 and January 2021 in the contiguous United States. Data preprocessing and exploratory investigation were completed for analysis of the collected dataset. Further, a novel and unified architecture called attention-based one-dimensional convolution with bidirectional long short-term memory layers (CNN-BiLSTM-ATT) is proposed to classify people’s sentiments as positive, neutral, and negative based on COVID-19-related tweets. In the CNN-BiLSTM-ATT model, the CNN layer can extract the low-level semantic features from textual data, and the BiLSTM layer can extract both the previous and future contextual representations. The attention module can improve the information focus from the outputted layer of the BiLSTM. The proposed method can extract both the local phrase representations and the global feature of sentences. Numerical experiments were conducted on COVID-19-related tweets using the proposed method and other baseline models to compare their performances. Our experimental results demonstrate that the CNN-BiLSTM-ATT model achieves an average accuracy of 95.16% and a macro-average F1-score of 95.12%, which outperforms the baseline models.

Inventory Prediction Using a Modified Multi-Dimensional Collaborative Wrapped Bi-Directional Long Short-Term Memory Model

Inventory prediction is concerned with the forecasting of future demand for products in order to optimize inventory levels and supply chain management. The challenges include demand volatility, data quality, multi-dimensional interactions, lead time variability, seasonal trends, and dynamic pricing. Nevertheless, these models suffer from numerous shortcomings, and in this research, we propose a new model, MMCW-BiLSTM (modified multi-dimensional collaboratively wrapped BiLSTM), for inventory prediction. The MMCW-BiLSTM model reflects a considerable leap in inventory forecasting by combining a number of components in order to consider intricate temporal dependencies and incorporate feature interactions. The MMCW-BiLSTM makes use of BiLSTM layers, collaborative attention mechanisms, and a multi-dimensional attention approach to learn from augmented datasets consisting of the original features and the extracted time series data. Moreover, adding a Taylor series transformation allows for a more precise description of the features in the model, thus improving the prediction precision. The results show that the models make the least mistakes when they use the AV demand forecasting dataset, with MAE values of 1.75, MAPE values of 2.89, MSE values of 6.76, and RMSE values of 2.6. Similarly, when utilizing the product demand dataset, the model also achieves the lowest error values for these metrics at 1.97, 3.91, 8.76, and 2.96. Likewise, when utilizing the dairy goods sales dataset, the model also achieves the lowest error values for these metrics at 2.54, 3.69, 10.39, and 3.22.

Improving biomedical Named Entity Recognition with additional external contexts

Objective:Biomedical Named Entity Recognition (bio NER) is the task of recognizing named entities in biomedical texts. This paper introduces a new model that addresses bio NER by considering additional external contexts. Different from prior methods that mainly use original input sequences for sequence labeling, the model takes into account additional contexts to enhance the representation of entities in the original sequences, since additional contexts can provide enhanced information for the concept explanation of biomedical entities. Methods:To exploit an additional context, given an original input sequence, the model first retrieves the relevant sentences from PubMed and then ranks the retrieved sentences to form the contexts. It next combines the context with the original input sequence to form a new enhanced sequence. The original and new enhanced sequences are fed into PubMedBERT for learning feature representation. To obtain more fine-grained features, the model stacks a BiLSTM layer on top of PubMedBERT. The final named entity label prediction is done by using a CRF layer. The model is jointly trained in an end-to-end manner to take advantage of the additional context for NER of the original sequence. Results:Experimental results on six biomedical datasets show that the proposed model achieves promising performance compared to strong baselines and confirms the contribution of additional contexts for bio NER. Conclusion:The promising results confirm three important points. First, the additional context from PubMed helps to improve the quality of the recognition of biomedical entities. Second, PubMed is more appropriate than the Google search engine for providing relevant information of bio NER. Finally, more relevant sentences from the context are more beneficial than irrelevant ones to provide enhanced information for the original input sequences. The model is flexible to integrate any additional context types for the NER task.

A Computational Predictor for Accurate Identification of Tumor Homing Peptides by Integrating Sequential and Deep BiLSTM Features.

Cancer remains a severe illness, and current research indicates that tumor homing peptides (THPs) play an important part in cancer therapy. The identification of THPs can provide crucial insights for drug-discovery and pharmaceutical industries as they allow for tailored medication delivery towards cancer cells. These peptides have a high affinity enabling particular receptors present upon tumor surfaces, allowing for the creation of precision medications that reduce off-target consequences and enhance cancer patient treatment results. Wet-lab techniques are considered essential tools for studying THPs; however, they're labor-extensive and time-consuming, therefore making prediction of THPs a challenging task for the researchers. Computational-techniques, on the other hand, are considered significant tools in identifying THPs according to the sequence data. Despite many strategies have been presented to predict new THP, there is still a need to develop a robust method with higher rates of success. In this paper, we developed a novel framework, THP-DF, for accurately identifying THPs on a large-scale. Firstly, the peptide sequences are encoded through various sequential features. Secondly, each feature is passed to BiLSTM and attention layers to extract simplified deep features. Finally, an ensemble-framework is formed via integrating sequential- and deep features which are fed to a support vector machine which with 10-fold cross-validation to carry to validate the efficiency. The experimental results showed that THP-DF worked better on both [Formula: see text] and [Formula: see text] datasets by achieving accuracy of > 95% which are higher than existing predictors both datasets. This indicates that the proposed predictor could be a beneficial tool to precisely and rapidly identify THPs and will contribute to the cutting-edge cancer treatment strategies and pharmaceuticals.

Prediction of Airport Taxiway Settlement based on CNNBiLSTM-Attention

The accurate real-time prediction of complex time series in airport runway settlement is crucial. In order to achieve precise and reliable predictions of runway surface settlement, considering the nonlinear and time-dependent characteristics of settlement data under the coupling effects of dynamic loads, water levels, and other factors, a CNN-BiLSTM-Attention approach is proposed. This method first employs Convolutional Neural Networks (CNN) to extensively extract local temporal features from the input sequences. Subsequently, the extracted local temporal features are input to BiLSTM layers to learn the features of the sequences both in the forward and reverse directions. Finally, the output of the BiLSTM layers is fed into an attention mechanism layer. By computing attention weights, crucial information in the input sequences is weighted and added to enhance the model’s focus on key information. The final settlement prediction is then generated through a fully connected layer. To validate the proposed method, field settlement monitoring data from Runway L at Hangzhou Xiaoshan Airport are utilized as an engineering case study. The results indicate that the combined model captures local features of settlement sequences more effectively with a smaller dataset, achieving higher accuracy compared to a standalone BiLSTM model.

Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data

In the modern era, remote sensing data has become increasingly useful for determining land use and coverage requirements. Remote sensing data can be used for a variety of purposes, including the classification of crops. It is possible to aggregate remote sensing data for a specific area over time in order to obtain a more complete picture based on the time series of this data. One example of these types of data is the Breizhcrop dataset, which was collected using satellite images acquired by Sentinel 2 over a period of time. This study aims to investigate a neural network based on attention mechanisms using the BI-LSTM layer in conjunction with Temporal-CNN for the classification of crops. The aim of the research is to find a model for corps classification in image-based time series. In line with this goal, in addition to finding features over time, the presented model also needs to produce high-accuracy features at each time step to increase classification. Utilizing the designed neural network, we seek to find local features with the attention mechanism and general features with a second layer. This neural network was validated on the BreizhCrop dataset and we conclude that it performs better than alternative approaches. The proposed method has been compared with Temporal CNN, Star RNN, and Vanilla LSTM networks and it has obtained better results than the mentioned neural networks. Taking advantage of these local and global features that extract with developed model obtained a high accuracy rate of 82%.

Research on Communication Signal Modulation Recognition Based on a CCLDNN

In this paper, a new automatic modulation recognition (AMR) method named CCLDNN (complex-valued convolution long short-term memory deep neural network) is proposed. It is designed to significantly improve the recognition accuracy of modulation modes in low signal-to-noise ratio (SNR) environments. The model integrates the advantages of existing mainstream neural networks. The phase and amplitude information of complex signals is effectively captured through a complex module in the input layer. The Squeeze-and-Excitation (SE) attention mechanism, Bi-LSTM layer, and deep convolutional layer are introduced in the feature extraction layer to gradually enhance feature expression. Among these, the introduction of LSTM enables the model to capture the sequence dependence of signals, and the application of the SE attention mechanism further improves the model’s ability to focus on key features. Tests using the RadioML2016.10a dataset show that the model performs well at multiple SNR levels, achieving an average recognition accuracy of more than 80% over an SNR range of 0 dB to 18 dB. However, under the condition of a low SNR from −20 dB to −2 dB, the model still maintains a high recognition ability. The advanced CCLDNN method shows great deep learning potential in solving practical communication problems.

Fake news detection in Dravidian languages using multiscale residual CNN_BiLSTM hybrid model

Fake news detection is the process of identifying news that contain purposeful misinformation disseminated through traditional news sources or social media platforms. Detecting fake news in Dravidian languages poses unique challenges due to the linguistic diversity and limited resources available for these languages. In this study, we propose a novel approach for fake news detection in Dravidian languages, leveraging contextualized word embeddings from MuRIL. MuRIL provides contextual embeddings that capture nuanced linguistic nuances, making it well-suited for the complexities of Dravidian languages. The proposed hybrid model combines multiscale residual CNN and BiLSTM layers to capture local and global dependencies in textual data effectively. The multiscale architecture allows the model to extract features at various levels of granularity while the BiLSTM layer captures long-range dependencies and contextual information. We performed extensive experiments on diverse datasets in Dravidian languages to evaluate the proposed approach. The results demonstrate the effectiveness of our hybrid model in discerning between genuine and fake news articles.

Acupuncture and tuina knowledge graph with prompt learning.

Acupuncture and tuina, acknowledged as ancient and highly efficacious therapeutic modalities within the domain of Traditional Chinese Medicine (TCM), have provided pragmatic treatment pathways for numerous patients. To address the problems of ambiguity in the concept of Traditional Chinese Medicine (TCM) acupuncture and tuina treatment protocols, the lack of accurate quantitative assessment of treatment protocols, and the diversity of TCM systems, we have established a map-filling technique for modern literature to achieve personalized medical recommendations. (1) Extensive acupuncture and tuina data were collected, analyzed, and processed to establish a concise TCM domain knowledge base. (2)A template-free Chinese text NER joint training method (TemplateFC) was proposed, which enhances the EntLM model with BiLSTM and CRF layers. Appropriate rules were set for ERE. (3) A comprehensive knowledge graph comprising 10,346 entities and 40,919 relationships was constructed based on modern literature. A robust TCM KG with a wide range of entities and relationships was created. The template-free joint training approach significantly improved NER accuracy, especially in Chinese text, addressing issues related to entity identification and tokenization differences. The KG provided valuable insights into acupuncture and tuina, facilitating efficient information retrieval and personalized treatment recommendations. The integration of KGs in TCM research is essential for advancing diagnostics and interventions. Challenges in NER and ERE were effectively tackled using hybrid approaches and innovative techniques. The comprehensive TCM KG our built contributes to bridging the gap in TCM knowledge and serves as a valuable resource for specialists and non-specialists alike.

BioBBC: a multi-feature model that enhances the detection of biomedical entities

The rapid increase in biomedical publications necessitates efficient systems to automatically handle Biomedical Named Entity Recognition (BioNER) tasks in unstructured text. However, accurately detecting biomedical entities is quite challenging due to the complexity of their names and the frequent use of abbreviations. In this paper, we propose BioBBC, a deep learning (DL) model that utilizes multi-feature embeddings and is constructed based on the BERT-BiLSTM-CRF to address the BioNER task. BioBBC consists of three main layers; an embedding layer, a Long Short-Term Memory (Bi-LSTM) layer, and a Conditional Random Fields (CRF) layer. BioBBC takes sentences from the biomedical domain as input and identifies the biomedical entities mentioned within the text. The embedding layer generates enriched contextual representation vectors of the input by learning the text through four types of embeddings: part-of-speech tags (POS tags) embedding, char-level embedding, BERT embedding, and data-specific embedding. The BiLSTM layer produces additional syntactic and semantic feature representations. Finally, the CRF layer identifies the best possible tag sequence for the input sentence. Our model is well-constructed and well-optimized for detecting different types of biomedical entities. Based on experimental results, our model outperformed state-of-the-art (SOTA) models with significant improvements based on six benchmark BioNER datasets.

A relational extraction approach based on multiple embedding representations and multi-head self-attention

Currently, word segmentation errors and polysemy problems are common in the field of Chinese relationship extraction. Although character-based model input can avoid word segmentation errors, in order to obtain the word information of a sentence, it is often necessary to introduce a dictionary or an external knowledge base to expand the word information, which requires a lot of manpower and time. In response to the above existing problems, this article uses characters as input, uses multiple embedding models to jointly form a character vector sequence, and obtains features containing character information through BiLSTM and attention layers; considering that convolutional neural networks are good at extracting local features, obtain features containing word information through multi-kernel convolutional layers and multi-head self-attention layers, and finally use a gating mechanism to fuse the features. The model was tested on the public SanWen data set and our own cultural-travel data set, and obtained F1 values of 61.22% and 60.26% respectively. Experimental results show that our method can achieve better relationship extraction effects without using word segmentation tools and without building a dictionary or external knowledge base, and the effect is better than most commonly used models currently.

SatCoBiLSTM: Self-attention based hybrid deep learning framework for crisis event detection in social media

Social media platforms have emerged as vital sources of sharing real-time information during crises, enabling users to share critical details about disasters and ensuring timely awareness. This study addresses the daunting challenge of effectively detecting crisis events from a vast, noisy stream of short text data. To address the above-mentioned issue, we present a self-attention-based hybrid deep learning framework, SatCoBiLSTM, designed to extract hierarchical textual features and crucial information from textual data. The proposed model integrates a multi-scale convolution and BiLSTM layer to extract local and contextual relations, alongside a self-attention layer to select and focus on the most relevant crisis features. Various experiments have been conducted on three benchmark real-world crisis datasets to examine the proposed model’s performance. SatCoBiLSTM achieved an impressive F1-score of 96%, 94%, and 95% on the three public datasets. It also shows a promising improvement in the F1-score, compared to the state-of-the-art (SOTA) and baseline methods, by 1%, 1%, and 6%, respectively, showing its effectiveness in crisis event detection. An ablation analysis has been carried out to investigate the validity of each integrated layer in our model. The SatCoBiLSTM model’s capability to identify crisis-related information highlights its potential to enhance real-time awareness during disasters. Eventually, the study advances crisis event detection and lays the foundation for future research to process and handle short-textual data in noisy environments.

Sentiment Analysis on Roman Urdu Students’ Feedback Using Enhanced Word Embedding Technique

Students’ feedback is crucial for educational institutions to assess the performance of their teachers, most opinions are expressed in their native language, especially for people in south Asian regions. In Pakistan, people use Roman Urdu to express their reviews, and this applied in the education domain where students used Roman Urdu to express their feedback. It is very time-consuming and labor-intensive process to handle qualitative opinions manually. Additionally, it can be difficult to determine sentence semantics in a text that is written in a colloquial style like Roman Urdu. This study proposes an enhanced word embedding technique and investigates the neural word Embedding (Word2Vec and Glove) to determine which performs better for Roman Urdu Sentiment analysis. Our suggested model employs the BiLSTM network to maintain the context in both directions and eventually, results for ternary classification are obtained by using the final softmax output layer. A manually labeled data set was used to evaluate the model, data is collected from the HEIs of Pakistan. Model was empirically evaluated on two datasets of Roman Urdu, the newly developed student’s feedback dataset and RUSA-19 publically available data set of Roman Urdu. Our model performs effectively using the word embedding and BiLSTM layer. The proposed model is compared with the baseline models of CNN, RNN, GRU and classic LSTM. The experimental findings demonstrate the proposed model's efficacy with an F1score of 90%.

Prediction of treatment response in major depressive disorder using a hybrid of convolutional recurrent deep neural networks and effective connectivity based on EEG signal.

In this study, we have developed a novel method based on deep learning and brain effective connectivity to classify responders and non-responders to selective serotonin reuptake inhibitors (SSRIs) antidepressants in major depressive disorder (MDD) patients prior to the treatment using EEG signal. The effective connectivity of 30 MDD patients was determined by analyzing their pretreatment EEG signals, which were then concatenated into delta, theta, alpha, and beta bands and transformed into images. Using these images, we then fine tuned a hybrid Convolutional Neural Network that is enhanced with bidirectional Long Short-Term Memory cells based on transfer learning. The Inception-v3, ResNet18, DenseNet121, and EfficientNet-B0 models are implemented as base models. Finally, the models are followed by BiLSTM and dense layers in order to classify responders and non-responders to SSRI treatment. Results showed that the EfficiencyNet-B0 has the highest accuracy of 98.33, followed by DensNet121, ResNet18 and Inception-v3. Therefore, a new method was proposed in this study that uses deep learning models to extract both spatial and temporal features automatically, which will improve classification results. The proposed method provides accurate identification of MDD patients who are responding, thereby reducing the cost of medical facilities and patient care.

Detecting Arabic Misinformation Using an Attention Mechanism-Based Model

The proliferation of fake news or misinformation, commonly referred to as fake news, has a significant effect on a global scale, as it is aimed at influencing public opinion as well as crowd decision-making. It is therefore crucial to verify the truthfulness of news before it is released to the public. Today, most studies on early detection of Arabic misinformation rely on machine learning methods and transformer-based models. Therefore, in the current study, we used deep learning techniques to propose a model for detecting Arabic misinformation by leveraging the contextual features of news article content. The proposed model was built based on BiLSTM and the attention mechanism. To extract features from Arabic text, we utilized a pre-trained AraBERT model, which generates contextual embeddings from text, then are fed to the BiLSTM layer as input features. Moreover, we investigated the effectiveness of the attention mechanism in improving the overall performance of the model by configuring model architecture to exclude the attention mechanism and comparing the results. Two datasets were utilized to train and evaluate the proposed model, namely, the AraNews and ArCovid19-Rumors datasets. Experimental results showed that the proposed model outperformed other existing models, achieving an accuracy of 0.96 on the ArCovid19-Rumors dataset and 0.90 on the AraNews dataset. This remarkable performance was due to the capability of the attention mechanism to enhance the overall performance, allowing the model to capture the relationship between textual features.

A boiler oxygen content and furnace temperature prediction model based on honey badger algorithm optimized neural network

Aiming at the problem that it is difficult to accurately predict the oxygen content and furnace temperature during the combustion process inside the circulating fluidized bed boiler, this paper proposes a prediction model with a mixture of the convolutional neural network(CNN), bi-directional long-short term memory network(biLSTM), and honey badger algorithm(HBA). First, process variables related to furnace temperature and oxygen content were selected as input variables using actual test data from circulating fluidized bed boilers. Secondly, the parsed input variables are analyzed by CNN through an analytical mechanism to extract the data into smaller details and features, which are used as inputs to the biLSTM layer. The BiLSTM layer selectively memorizes the information of the input temporal data and learns the information dependence of the temporal data sequences, which then solves the temporal problem of the training of the irregular trend of the oxygen content and the furnace temperature. Finally, the HBA is utilized to continuously traverse and search the CNN-biLSTM model to find the optimal parameters of the model. The experimental results show: the CNN-biLSTM neural network model mixed with the HBA is able to accurately predict the oxygen content and furnace temperature. The experimental results show that the CNN-biLSTM neural network model with hybrid HBA is able to accurately predict the oxygen content and furnace temperature, and the average MAPE errors for the oxygen content are HBA-CNN-biLSTM (2.92E-03), CNN (7.96E-02), LSTM (5.13E-02), GRU (4.83E-02), RF (4.96E-02), RBF (8.41E-02), SVM (5.71E-02), RNN (5.53E-02), CNN-LSTM (4.79E-02).

Exploring the Subjectivity of English Academic Discourse in the Context of Big Data

Abstract This study develops a sentiment analysis model for English academic discourse based on word information to effectively understand and analyze the sentiment tendencies in English literary texts. The structure of the model includes word embedding layer, character-level feature extraction, word-level feature extraction and feature fusion and classification layer. The word embedding layer realizes the mapping between word vectors and word vectors by microblogging pre-trained word vectors. The character-level feature extraction session uses a multi-window convolutional layer to capture N-Gram information. In contrast, the word-level feature extraction obtains deeper semantic information through a Bi-LSTM layer and fuses it with character-level information to enhance robustness. The feature fusion and classification layer further combines these features and determines the fusion weights through a linear layer to achieve sentiment classification. In performance tests, the model achieves 92.5% sentiment classification accuracy on the standard dataset, an improvement of about 6% compared to traditional methods. In particular, the accuracy is improved by 5% when dealing with text with sentiment polarity transition, showing good adaptability. In addition, using 657 positive and 679 negative sentiment words as seed words effectively expands the sentiment lexicon and enhances the comprehensiveness and accuracy of sentiment analysis.