Bidirectional Context Research Articles

Introducing bidirectional attention for autoregressive models in abstractive summarization

Abstractive summarization methods typically follow the autoregressive paradigm using the causal masks in the decoder for training and inference efficiency. However, this approach leads to a constant context throughout the generation process, which conflicting with the bidirectional characteristics of natural language. Although previous attempts have been made to incorporate bidirectional attention in the decoding process through non-autoregressive approach, the evaluation results are not comparable to the autoregressive methods. To bring bidirectional attention to the autoregressive process while maintaining superior performance, we propose the global autoregressive paradigm, which takes the outputs of the autoregressive process as additional inputs in the subsequent global iteration. Specifically, we build a bidirectional decoder alongside the original encoder and decoder to capture the bidirectional context of the outputs. This context is updated after each autoregressive decoding iteration. The decoder then integrates the updated context into subsequent autoregressive decoding steps, enhancing the generative process with a more comprehensive and authentic context. Additionally, we use contrastive learning to train the model to extract reliable features from the bidirectional context and apply reinforcement learning to improve the model's utilization of this context. We evaluate our method on CNN/DM, XSum, and NYT datasets, and the results highlight the significance of the bidirectional context. Our method achieves the best performance in terms of ROUGE-2 on CNN/DM (23.96), and performs comparably on XSum (25.45) and NYT (27.91). It also outperforms all the baselines in terms of BERTScore, with a score of 89.96 on CNN/DM, 92.70 on XSum, and 90.04 on NYT. Furthermore, our method can perform better with a larger beam size.

Attention-Based Load Forecasting with Bidirectional Finetuning

Accurate load forecasting is essential for the efficient and reliable operation of power systems. Traditional models primarily utilize unidirectional data reading, capturing dependencies from past to future. This paper proposes a novel approach that enhances load forecasting accuracy by fine tuning an attention-based model with a bidirectional reading of time-series data. By incorporating both forward and backward temporal dependencies, the model gains a more comprehensive understanding of consumption patterns, leading to improved performance. We present a mathematical framework supporting this approach, demonstrating its potential to reduce forecasting errors and improve robustness. Experimental results on real-world load datasets indicate that our bidirectional model outperforms state-of-the-art conventional unidirectional models, providing a more reliable tool for short and medium-term load forecasting. This research highlights the importance of bidirectional context in time-series forecasting and its practical implications for grid stability, economic efficiency, and resource planning.

BERT-based log exception detection algorithm LADB

With the development of information systems, they have become increasingly large and complex and have generated a large amount of log information. These log information records the system's health, but the number of log information is huge, and the traditional exception detection algorithm in the case of very large amounts of data is difficult to efficiently and accurately detect anomalies due to poor generalization performance. A BERT-based log anomaly detection algorithm, LADB, is proposed, essentially a semi-supervised, multi-classification algorithm. (1) LADB uses the Transformer encoder as the base component for problems such as feature degradation and gradient explosion. (2) In order to make better use of the bidirectional context, and in view of BERT's excellence in the NLP field, the Masking Log Key Prediction (MLKP) self-monitoring task was designed, drawing on the idea of BERT's Masking Language Model. (3) In order to solve the problem of difficult and slow processing of high-dimensional data, the Deep SVDD algorithm is used for minimum superspheres capacity (VHM) self-supervision training task. Experiments have shown that LADB's combined performance is superior to the four representative log anomaly detection algorithms.

Improving the Accuracy and Effectiveness of Text Classification Based on the Integration of the Bert Model and a Recurrent Neural Network (RNN_Bert_Based)

This paper proposes a new robust model for text classification on the Stanford Sentiment Treebank v2 (SST-2) dataset in terms of model accuracy. We developed a Recurrent Neural Network Bert based (RNN_Bert_based) model designed to improve classification accuracy on the SST-2 dataset. This dataset consists of movie review sentences, each labeled with either positive or negative sentiment, making it a binary classification task. Recurrent Neural Networks (RNNs) are effective for text classification because they capture the sequential nature of language, which is crucial for understanding context and meaning. Bert excels in text classification by providing bidirectional context, generating contextual embeddings, and leveraging pre-training on large corpora. This allows Bert to capture nuanced meanings and relationships within the text effectively. Combining Bert with RNNs can be highly effective for text classification. Bert’s bidirectional context and rich embeddings provide a deep understanding of the text, while RNNs capture sequential patterns and long-range dependencies. Together, they leverage the strengths of both architectures, leading to improved performance on complex classification tasks. Next, we also developed an integration of the Bert model and a K-Nearest Neighbor based (KNN_Bert_based) method as a comparative scheme for our proposed work. Based on the results of experimentation, our proposed model outperforms traditional text classification models as well as existing models in terms of accuracy.

Sentiment analysis of movie review classifications using deep learning approaches

Movie reviews reflect how the public feels about a movie they have watched. However, because many reviews are posted on various websites, it is practically impossible to read each one. Summarizing all movie reviews can help people make informed decisions without reading through all of them. Previous studies have used different machine learning and deep learning techniques for sentiment analysis (SA), but few have combined comprehensive hyperparameter tuning and novel datasets for better performance. This paper presents an SA approach using deep learning models with optimized hyperparameters and a novel Rotten Tomatoes (RT) dataset to help viewers make better movie choices. SA, or opinion mining, is a computational technique to extract and analyze opinions and emotions expressed in text. We explore deep learning models such as Long Short-Term Memory (LSTM), XLNet, Convolutional Neural Networks-LSTM (CNN-LSTM), and Bidirectional Encoder Representations from Transformers (BERT). These models are known for capturing complex language patterns and context from raw text data. XLNet, a pre-trained model, effectively understands context by considering all possible permutations of the input sequence, BERT excels at using bidirectional context to understand text, LSTM retains information about long-term patterns in sequential data, and CNN-LSTM combines local and global context for reliable feature extraction. The RT dataset was pre-processed with data cleaning, spelling correction, lemmatization, and handling of informal words to improve the results. Our experiments show that XLNet performed better than other models on the Rotten Tomatoes dataset. The study demonstrates that SA of movie reviews provides insights into emotions and attitudes, allowing us to estimate a movie’s performance based on its overall sentiment.

Bi-LSTM predictive control-based efficient energy management system for a fuel cell hybrid electric vehicle

Energy management systems (EMS) in hybrid electric vehicles play a vital role in achieving optimal energy utilization, improved fuel efficiency, lower emissions, and improved vehicle performance. Thus, this paper proposes a Bidirectional Long-Term Memory (Bi-LSTM) model based on an efficient EMS for a hybrid electric vehicle that manages the powertrain elements of the IC engine, electric motor(s), fuel cell, and energy storage system. The proposed EMS also continuously monitors various parameters, including vehicle speed, driver inputs, battery state of charge, engine load, and environmental conditions, to make real-time decisions on power distribution and operation modes. Further, the proposed system leverages the capabilities of the Bi-LSTM model to capture intricate temporal dependencies and bidirectional context in driving patterns using the real-time dataset. Empirical studies are conducted to substantiate the efficacy of the proposed energy management strategy vis-à-vis diverse controllers, encompassing Model Predictive Control and Sliding Mode Control. Real-world driving data is employed as the basis for these investigations, aiming to provide a robust assessment of the proposed energy management approach in practical scenarios. Results demonstrate the efficacy of the suggested methodology compared to traditional EMS approaches, showcasing improvements in energy utilization and reduced environmental impact. Finally, this research emphasizes a comparative analysis of the proposed topology with the existing approaches in a real-world fuel cell hybrid electric vehicle system. The findings provide insights into the feasibility of deploying intelligent EMS strategies for improved sustainability and efficiency in a modern energy-efficient environment.

A hybrid neural network model based on transfer learning for Arabic sentiment analysis of customer satisfaction

AbstractSentiment analysis, a method used to classify textual content into positive, negative, or neutral sentiments, is commonly applied to data from social media platforms. Arabic, an official language of the United Nations, presents unique challenges for sentiment analysis due to its complex morphology and dialectal diversity. Compared to English, research on Arabic sentiment analysis is relatively scarce. Transfer learning, which applies the knowledge learned from one domain to another, can address the limitations of training time and computational resources. However, the development of transfer learning for Arabic sentiment analysis is still underdeveloped. In this study, we develop a new hybrid model, RNN‐BiLSTM, which merges recurrent neural networks (RNN) and bidirectional long short‐term memory (BiLSTM) networks. We used Arabic bidirectional encoder representations from transformers (AraBERT), a state‐of‐the‐art Arabic language pre‐trained transformer‐based model, to generate word‐embedding vectors. The RNN‐BiLSTM model integrates the strengths of RNN and BiLSTM, including the ability to learn sequential dependencies and bidirectional context. We trained the RNN‐BiLSTM model on the source domain, specifically the Arabic reviews dataset (ARD). The RNN‐BiLSTM model outperforms the RNN and BiLSTM models with default parameters, achieving an accuracy of 95.75%. We further applied transfer learning to the RNN‐BiLSTM model by fine‐tuning its parameters using random search. We compared the performance of the fine‐tuned RNN‐BiLSTM model with the RNN and BiLSTM models on two target domain datasets: ASTD and Aracust. The results showed that the fine‐tuned RNN‐BiLSTM model is more effective for transfer learning, achieving an accuracy of 95.44% and 96.19% on the ASTD and Aracust datasets, respectively.

Research on a Fault Diagnosis Method for the Braking Control System of an Electric Multiple Unit Based on Deep Learning Integration

A fault diagnosis method based on deep learning integration is proposed focusing on fault text data to effectively improve the efficiency of fault repair and the accuracy of fault localization in the braking control system of an electric multiple unit (EMU). First, the Borderline-SMOTE algorithm is employed to synthesize minority class samples at the boundary, addressing the data imbalance and optimizing the distribution of data within the fault text. Then, a multi-dimensional word representation is generated using the multi-layer bidirectional transformer architecture from the pre-training model, BERT. Next, BiLSTM captures bidirectional context semantics and, in combination with the attention mechanism, highlights key fault information. Finally, the LightGBM classifier is employed to reduce model complexity, enhance analysis efficiency, and increase the practicality of the method in engineering applications. An experimental analysis of fault data from the braking control system of the EMU indicates that the deep learning integration method can further improve diagnostic performance.

Bilateral Context Modeling for Residual Coding in Lossless 3D Medical Image Compression.

Residual coding has gained prevalence in lossless compression, where a lossy layer is initially employed and the reconstruction errors (i.e., residues) are then losslessly compressed. The underlying principle of the residual coding revolves around the exploration of priors based on context modeling. Herein, we propose a residual coding framework for 3D medical images, involving the off-the-shelf video codec as the lossy layer and a Bilateral Context Modeling based Network (BCM-Net) as the residual layer. The BCM-Net is proposed to achieve efficient lossless compression of residues through exploring intra-slice and inter-slice bilateral contexts. In particular, a symmetry-based intra-slice context extraction (SICE) module is proposed to mine bilateral intra-slice correlations rooted in the inherent anatomical symmetry of 3D medical images. Moreover, a bi-directional inter-slice context extraction (BICE) module is designed to explore bilateral inter-slice correlations from bi-directional references, thereby yielding representative inter-slice context. Experiments on popular 3D medical image datasets demonstrate that the proposed method can outperform existing state-of-the-art methods owing to efficient redundancy reduction. Our code will be available on GitHub for future research.

Cyberbullying Detection in Social Networks Using Deep Learning

Cyberbullying causes significant harm, especially among adolescents and young adults. With the growth of social media, online harassment through platforms like Facebook and Twitter has also proliferated rapidly. Though social networks have reporting mechanisms, the volume of user-generated content makes manual moderation infeasible. This necessitates automated detection systems that can accurately identify cyberbullying at scale. Recent advances in deep learning provide promising techniques for text classification tasks. This paper explores Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM), and transformer models like Bidirectional Encoder Representations from Transformers (BERT) for cyberbullying detection in social networks. The models are evaluated on a benchmark dataset containing 11,000 Facebook comments labeled as clean or cyberbullying. Extensive experiments demonstrate that BERT achieves the highest accuracy of 87.3% followed by a hierarchical CNN-LSTM architecture with 86.5% accuracy. The former benefits from bidirectional context modeling using self-attention while the latter combines the strengths of convolutional layers and LSTMs. The results verify the effectiveness of deep learning methodologies for this problem. However, enhancements in multilingual, multimodal support and adversarial robustness are required. Testing on diverse platforms and content along with user privacy considerations remain as future research directions. This empirical study provides useful insights to build robust cyberbullying detection systems.

Negative emotion detection on social media during the peak time of COVID-19 through deep learning with an auto-regressive transformer

Negative emotion detection is challenging during the peak time of the COVID-19 period. Most earlier studies contain individuals’ physical health recognition rather than mental health detection, which is a significant concern in facing the COVID-19 situation. Identifying mental health in advance is essential to understand individuals’ psychological condition. This paper considers the texts from social media during the pandemic of COVID-19. We propose a novel context-based auto-regressive transformer with bidirectional long short-term memory and a convolutional neural network (Context-ABT-BiLSTM-CNN) model to detect emotions such as abuse, anger, anxiety, depression, disgust, fear, guilt, sadness, and shame on social media. The existing works do not suggest relevant terms to detect suitable context; as a result, there is no scope for detecting emotions. We introduce a novel topic-based text (TBT) with a rule-based permutation (RBP) procedure to extract the relevant text from social media to identify emotions. Random search is suggested to store each input’s correlated information and the order of each sequence. We recommend various transformer components to maintain the text sequence, avoid discrepancies during model training, capture the long-distance semantics in bidirectional contexts, and adopt both the permutation and factorization processes to build the model. Moreover, a comparative study is introduced to detect the most dominant emotions on social media during the pandemic and non-pandemic periods. The proposed model with XLNet embeddings surpasses state-of-the-art models for detecting text emotions. The ablation study is conducted to understand the essential components needed for the proposed model.

IgLM: Infilling language modeling for antibody sequence design

Discovery and optimization of monoclonal antibodies for therapeutic applications relies on large sequence libraries but is hindered by developability issues such as low solubility, high aggregation, and high immunogenicity. Generative language models, trained on millions of protein sequences, are a powerful tool for the on-demand generation of realistic, diverse sequences. We present the Immunoglobulin Language Model (IgLM), a deep generative language model for creating synthetic antibody libraries. Compared with prior methods that leverage unidirectional context for sequence generation, IgLM formulates antibody design based on text-infilling in natural language, allowing it to re-design variable-length spans within antibody sequences using bidirectional context. We trained IgLM on 558 million (M) antibody heavy- and light-chain variable sequences, conditioning on each sequence's chain type and species of origin. We demonstrate that IgLM can generate full-length antibody sequences from a variety of species and its infilling formulation allows it to generate infilled complementarity-determining region (CDR) loop libraries with improved in silico developability profiles. A record of this paper's transparent peer review process is included in the supplemental information.

BCRNet: Bidirectional contrastive representation network for deep multimodal learning of exercise representations in online education systems

In online education systems, learning exercise representation is a fundamental task in many applications, such as exercise retrieval and recommendation. To capture the heterogeneous data information of exercises (i.e., texts and images), deep multimodal approaches show the promising performance. However, these methods have two limitations: (1) they only care about context on one side, which fails to utilize the future context chunks in the exercises; (2) they cannot ensure the presentation ability due to the scarcity of labelled data. In this paper, we propose a bidirectional contrastive representation network (BCRNet) to tackle these issues. First, we construct a representation module with a masking constraint loss to take into account the bidirectional context contents of exercises. Second, we design a contrastive learning approach which uses a multimodal contrastive loss to reshape the multimodal representation space and improve model presentation ability without labelled data. Moreover, a text-image matching strategy is designed to provide semantic links between texts and images. On the real-world dataset, experiments demonstrate BCRNet performs significantly better than many strong baselines.

Multi-view aggregation transformer for no-reference point cloud quality assessment

With the increasing maturity of 3D point cloud acquisition, storage, and transmission technologies, a large number of distorted point clouds without original reference exist in practical applications. Hence, it is necessary to design a no-reference point cloud quality assessment (PCQA) for point cloud systems. However, the existing no-reference PCQA metrics ignore the content differences and positional context among the projected images. For this, we propose a Multi-View Aggregation Transformer (MVAT) with two different fusion modules to extract the comprehensive feature representation of PCQA. Specifically, considering the content differences of different projected images, we first design a Content Fusion Module (CFM) to fuse multiple projected image features by adaptive weighting. Then, we design a Bidirectional Context Fusion Module (BCFM) to extract context features for reflecting the contextual relationship among projected images. Finally, we joint the above two fusion modules via Content-Position Fusion Module (CPFM) to fully mine the feature representation of point clouds. Experimental results show that our MVAT can achieve comparable or better performance than state-of-the-art metrics on three open point cloud datasets.

Exploring Bi-Directional Context for Improved Chatbot Response Generation Using Deep Reinforcement Learning

The development of conversational agents that can generate relevant and meaningful replies is a challenging task in the field of natural language processing. Context and predictive capabilities are crucial factors that humans rely on for effective communication. Prior studies have had a significant limitation in that they do not adequately consider the relationship between utterances in conversations when generating responses. This study aims to address this limitation by proposing a novel method that comprehensively models the contextual information of the current utterance for response generation. A commonly used approach is to rely on the information of the current utterance to generate the corresponding response, and as such it does not take advantage of the context of a multi-turn conversation. In our proposal, different from other studies, we will use a bi-directional context in which the historical direction helps the model remember information from the past in the conversation, while the future direction enables the model to anticipate its impact afterward. We combine a Transformer-based sequence-to-sequence model and the reinforcement learning algorithm to achieve our goal. Experimental results demonstrate the effectiveness of the proposed model through qualitative evaluation of some generated samples, in which the proposed model increases 24% average BLEU score and 29% average ROUGE score compared to the baseline model. This result also shows that our proposed model improves from 5% to 151% for the average BLEU score compared with previous related studies.

Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context for Continuous Speech Recognition

Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context for Continuous Speech Recognition

Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context for Continuous Speech Recognition

While speech recognition Word Error Rate (WER) has reached human parity for English, continuous speech recognition scenarios such as voice typing and meeting transcriptions still suffer from segmentation and punctuation problems, resulting from irregular pausing patterns or slow speakers. Transformer sequence tagging models are effective at capturing long bi-directional context, which is crucial for automatic punctuation. Automatic Speech Recognition (ASR) production systems, however, are constrained by real-time requirements, making it hard to incorporate the right context when making punctuation decisions. Context within the segments produced by ASR decoders can be helpful but limiting in overall punctuation performance for a continuous speech session. In this paper, we propose a streaming approach for punctuation or re-punctuation of ASR output using dynamic decoding windows and measure its impact on punctuation and segmentation accuracy across scenarios. The new system tackles over-segmentation issues, improving segmentation F0.5-score by 13.9%. Streaming punctuation achieves an average BLEUscore improvement of 0.66 for the downstream task of Machine Translation (MT).

Context-aware Pseudo-true Video Interpolation at 6G Edge

In the 6G network, lots of edge devices facilitate the low-latency transmission of video. However, with limited processing and storage capabilities, the edge devices cannot afford to reconstruct the vast amount of video data. On the condition of edge computing in the 6G network, this article fuses a self-similarity-based context feature into Frame Rate Up-Conversion (FRUC) to generate the pseudo-true video sequences at high frame rate, and its core is the extraction of the context layer for each video frame. First, we extract the patch centered at each pixel and use the self-similarity descriptor to generate the correlation surface. Then, the expectation or skewness of the correlation surface in statistics is computed to represent its context feature. By attaching an expectation or a skewness to each pixel, the context layer is constructed and added to the video frame as a new channel. According to the context layer, we predict the motion vector field of the absent frame by using the bidirectional context match and finally produce the interpolated frame. From the experimental results, it can be seen that by deploying the proposed FRUC algorithm on edge devices, the output pseudo-true video sequences have satisfying objective and subjective qualities.

Temporal Moment Localization via Natural Language by Utilizing Video Question Answers as a Special Variant and Bypassing NLP for Corpora

Temporal moment localization using natural language (TMLNL) is an emerging issue in computer vision for localizing a specific moment inside a long, untrimmed video. The goal of TMLNL is to obtain the video’s output moment, which is related to the input query in a substantial way. Previous research focused on the visual portion of TMLNL, such as objects, backdrops, and other visual attributes, but natural language processing (NLP) techniques were largely used for the textual portion. A long query requires sufficient context to properly localize moments within a long untrimmed video. Thus, as a consequence of not completely understanding how to handle queries, performances deteriorated, especially when the query was longer. In this paper, we treat the TMLNL challenge as a unique variation of VQA, which equally considers the visual elements by using our proposed VQA joint visual-textual framework (JVTF). However, we also manage complex and long input queries without employing natural language processing (NLP) by improving poorly graded to finely graded distinct granularity representations. Our suggested BCPN searches for insufficient context for long input queries using an approach called query handler (QH) and helps the JVTF find the most relevant moment. Previously, a recurrence of words was caused by increasing the number of encoding layers in transformers, LSTMs, and other NLP techniques; however, our QH ensured that repetition of word locations was reduced. The output of BCPN is combined with JVTF’s guided attention to further improve the end outcome. Therefore, we propose a novel bidirectional context predictor network (BCPN), in addition to a VQA joint visual-textual framework (JVTF), to address the equal importance of videos and queries. Through extensive experiments on three benchmark datasets, we show that the proposed BCPN outperforms the state-of-the-art methods by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$IoU = 0.3 (2.65 \%) $ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$IoU = 0.5 (2.49 \%)$ </tex-math></inline-formula> , and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$IoU = 0.7 (2.06 \%) $ </tex-math></inline-formula> .

Research on sentiment classification for netizens based on the BERT-BiLSTM-TextCNN model.

Sentiment analysis of netizens’ comments can accurately grasp the psychology of netizens and reduce the risks brought by online public opinion. However, there is currently no effective method to solve the problems of short text, open word range, and sometimes reversed word order in comments. To better solve the above problems, this article proposes a hybrid model of sentiment classification, which is based on bidirectional encoder representations from transformers (BERT), bidirectional long short-term memory (BiLSTM) and a text convolution neural network (TextCNN) (BERT-BiLSTM-TextCNN). The experimental results show that (1) the hybrid model proposed in this article can better combine the advantages of BiLSTM and TextCNN; it not only captures local correlation while retaining context information but also has high accuracy and stability. (2) The BERT-BiLSTM-TextCNN model can extract important emotional information more flexibly in text and achieve multiclass classification tasks of emotions more accurately. The innovations of this study are as follows: (1) the use of BERT to generate word vectors has the advantages of more prior information and a full combination of contextual semantics; (2) the BiLSTM model, as a bidirectional context mechanism model, can obtain contextual information well; and (3) the TextCNN model can obtain important features well in the problem of text classification, and the combined effect of the three modules can significantly improve the accuracy of emotional multilabel classification.