Seq2seq Architecture Research Articles

Boosting multi-document summarization with hierarchical graph convolutional networks

The input of the multi-document summarization task is usually long and has high redundancy. Encoding multiple documents is a challenge for the Seq2Seq architecture. The way of concatenating multiple documents into a sequence ignores the relation between documents. Attention-based Seq2Seq architectures have slightly improved the cross-document relation modeling for multi-document summarization. However, these methods ignore the relation between sentences, and there is little improvement that can be achieved through the attention mechanism alone. This paper proposes a hierarchical approach to leveraging the relation between words, sentences, and documents for abstractive multi-document summarization. Our model employs the Graph Convolutional Networks (GCN) for capturing the cross-document and cross-sentence relations. The GCN module can enrich semantic representations by generating high-level hidden features. Our model achieves significant improvement over the attention-based baseline, beating the Hierarchical Transformer by 3.4/1.64, 1.92/1.44 ROUGE-1/2 F1 points on the Multi-News and WikiSum datasets, respectively. Experimental results demonstrate that our delivered method brings substantial improvements over several strong baselines.

Clinical outcome-guided deep temporal clustering for disease progression subtyping

ObjectiveComplex diseases exhibit heterogeneous progression patterns, necessitating effective capture and clustering of longitudinal changes to identify disease subtypes for personalized treatments. However, existing studies often fail to design clustering-specific representations or neglect clinical outcomes, thereby limiting the interpretability and clinical utility. MethodWe design a unified framework for subtyping longitudinal progressive diseases. We focus on effectively integrating all data from disease progressions and improving patient representation for downstream clustering. Specifically, we propose a clinical Outcome-Guided Deep Temporal Clustering (OG-DTC) that generates representations informed by clustering and clinical outcomes. A GRU-based seq2seq architecture captures the temporal dynamics, and the model integrates k-means clustering and outcome regression to facilitate the formation of clustering structures and the integration of clinical outcomes. The learned representations are clustered using a Gaussian mixture model to identify distinct subtypes. The clustering results are extensively validated through reproducibility, stability, and significance tests. ResultsWe demonstrated the efficacy of our framework by applying it to three Alzheimer’s Disease (AD) clinical trials. Through the AD case study, we identified three distinct subtypes with unique patterns associated with differentiated clinical declines across multiple measures. The ablation study revealed the contributions of each component in the model and showed that jointly optimizing the full model improved patient representations for clustering. Extensive validations showed that the derived clustering is reproducible, stable, and significant. ConclusionOur temporal clustering framework can derive robust clustering applicable for subtyping longitudinal progressive diseases and has the potential to account for subtype variability in clinical outcomes.

From text to multimodal: a survey of adversarial example generation in question answering systems

AbstractIntegrating adversarial machine learning with question answering (QA) systems has emerged as a critical area for understanding the vulnerabilities and robustness of these systems. This article aims to review adversarial example-generation techniques in the QA field, including textual and multimodal contexts. We examine the techniques employed through systematic categorization, providing a structured review. Beginning with an overview of traditional QA models, we traverse the adversarial example generation by exploring rule-based perturbations and advanced generative models. We then extend our research to include multimodal QA systems, analyze them across various methods, and examine generative models, seq2seq architectures, and hybrid methodologies. Our research grows to different defense strategies, adversarial datasets, and evaluation metrics and illustrates the literature on adversarial QA. Finally, the paper considers the future landscape of adversarial question generation, highlighting potential research directions that can advance textual and multimodal QA systems in the context of adversarial challenges.

Chatbot Architecture based on NLP Techniques

Chatbots have emerged as a prominent technological innovation with a profound impact on various aspects of human life, including customer service, communication, and automation. This abstract provides an overview of the evolution, functionalities, and implications of chatbots in contemporary The potential of chatbots goes beyond commercial applications. They are being used in education to provide personalized learning experiences and in mental health support, offering immediate responses to those in need. In the workplace, chatbots can streamline administrative tasks, manage appointments, and assist with onboarding processes. Their adaptability and scalability have made them a valuable asset in various industries. The continued development of chatbot technology and a thoughtful approach to their use are essential for maximizing their benefits while mitigating potential drawbacks in the rapidly changing landscape of artificial intelligence and human-computer interaction. Key Words: Chatbot, NLP ,AI, Seq2Seq architecture, RNN-based models

Text Summarization and Document summarization using NLP

Background: Automatic text summarization has become increasingly crucial in navigating the ever- growing ocean of textual information. This research delves into exploring the potential of Natural Language Processing (NLP) techniques for creating efficient and informative summaries. We implemented and evaluated models based on Long Short-Term Memory (LSTM) networks, Sequence-to-Sequence (Seq2Seq) architectures, and transformer-based approaches. By leveraging these powerful algorithms, we aimed to generate concise summaries that capture the essence of the original text. The evaluation highlighted the strengths and limitations of each approach, showcasing the potential of NLP for text summarization while acknowledging the remaining challenges. This research not only contributes to the ongoing discussion on textsummarization techniques but also opens doorsfor further exploration, including integrating domain-specific knowledge, personalizing summaries based on user preferences, and applying these techniques to real-world information overload situations. Ultimately, this work underscores the promise of NLP-driven text summarization in facilitating efficient information access, comprehension, and utilization across various domains. Keywords: Natural Language Processing(NLP), Automatic Text Summarization, Real-world applications.

U-SeqNet: learning spatiotemporal mapping relationships for multimodal multitemporal cloud removal

ABSTRACT Optical remotely sensed time series data have various key applications in Earth surface dynamics. However, cloud cover significantly hampers data analysis and interpretation. Despite synthetic aperture radar (SAR)-to-optical image translation techniques emerging as a promising solution, their effectiveness is diminished by their inability to adequately account for the intertwined nature of temporal and spatial dimensions. This study introduces U-SeqNet, an innovative model that integrates U-Net and Sequence-to-Sequence (Seq2Seq) architectures. Leveraging a pioneering spatiotemporal teacher forcing strategy, U-SeqNet excels in adapting and reconstructing data, capitalizing on available cloud-free observations to improve accuracy. Rigorous assessments through No Reference and Full Reference Image Quality Assessments (NR – IQA and FR – IQA) affirm U-SeqNet’s exceptional performance, marked by a Natural Image Quality Evaluator (NIQE) score of 5.85 and Mean Absolute Error (MAE) of 0.039. These results underline U-SeqNet’s exceptional capabilities in image reconstruction and its potential to improve remote sensing analysis by enabling more accurate and efficient multimodal and multitemporal cloud removal techniques.

A multi-step wind power group forecasting seq2seq architecture with spatial–temporal feature fusion and numerical weather prediction correction

Accurate total wind power forecasting is essential to wind farms. Conventional methods predict each wind turbine separately, which fails to exploit the spatial correlation between turbines and causes error accumulation through summation. Additionally, numerical weather prediction (NWP) with limited spatial–temporal resolutions is difficult to assist in power forecasting effectively, resulting in decreased performance as the forecasting steps increase. To address these drawbacks, a novel “N encoder-1 decoder” multi-channel fusion group forecasting sequence-to-sequence (seq2seq) architecture with wind resource quality assisted spatial attention (WRQASA) and NWP correction (NWPC) is proposed in this paper. The encoders in the architecture extract the temporal features of each turbine, followed by WRQASA adaptively adjusting their impact on the decoder. NWPC constructs micro-scale meteorological data based on its large-scale counterpart. The decoder takes the output from WRQASA and NWPC, and predicts the total power of the turbine group. The model is evaluated on a real-world wind farm. Ablation studies revealed the effectiveness of the novel architecture, WRQASA and NWPC. In 12, 24, 36 and 72-h multi-step forecasting scenarios, the new method outperformed conventional methods, such as seq2seq, light gradient boosting machine and support vector regression, achieving a more effective prediction of total power.

BlazePose-Seq2Seq: Leveraging Regular RGB Cameras for Robust Gait Assessment.

Evaluation of human gait through smartphone-based pose estimation algorithms provides an attractive alternative to costly lab-bound instrumented assessment and offers a paradigm shift with real time gait capture for clinical assessment. Systems based on smart phones, such as OpenPose and BlazePose have demonstrated potential for virtual motion assessment but still lack the accuracy and repeatability standards required for clinical viability. Seq2seq architecture offers an alternative solution to conventional deep learning techniques for predicting joint kinematics during gait. This study introduces a novel enhancement to the low-powered BlazePose algorithm by incorporating a Seq2seq autoencoder deep learning model. To ensure data accuracy and reliability, synchronized motion capture involving an RGB camera and ten Vicon cameras were employed across three distinct self-selected walking speeds. This investigation presents a groundbreaking avenue for remote gait assessment, harnessing the potential of Seq2seq architectures inspired by natural language processing (NLP) to enhance pose estimation accuracy. When comparing BlazePose alone to the combination of BlazePose and 1D convolution Long Short-term Memory Network (1D-LSTM), Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM), the average mean absolute errors decreased from 13.4° to 5.3° for fast gait, from 16.3° to 7.5° for normal gait, and from 15.5° to 7.5° for slow gait at the left ankle joint angle respectively. The strategic utilization of synchronized data and rigorous testing methodologies further bolsters the robustness and credibility of these findings.

Research on Automatic Annotation Method of Korean Language under Data Driving and Fusion

Abstract In the quest to streamline Korean text and speech annotation, this research introduces innovative automatic annotation methods that promise to revolutionize efficiency and technical prowess in constructing Korean annotation datasets. By leveraging the sophisticated Seq2Seq architecture with BERT and bidirectional GRU models, we significantly enhance the model’s ability to grasp contextual nuances, ensuring precise text annotations. The speech annotation frontier benefits from a novel amalgamation of the Hidden Markov Model’s forced alignment and semi-supervised learning, perfected with Seneff auditory features for meticulous phonological consonant boundary detection. Empirical validation across diverse datasets showcases our methodology’s superiority, achieving a remarkable 96.01% accuracy in text annotation and setting a new benchmark for phonological boundary detection at a 14.5ms minimum distance threshold. Our approach outperforms traditional algorithms, marking a pivotal step forward in Korean automatic annotation.

A Physics-Informed Neural Network Approach for Surrogating a Numerical Simulation of Fractured Horizontal Well Production Prediction

With the popularity of deep learning (DL), more and more studies are focusing on replacing time-consuming numerical simulations with efficient surrogate models to predict the production of multi-stage fractured horizontal wells. Previous studies on constructing surrogate models for the prediction of the production of fractured horizontal wells often relied on directly applying existing deep learning architectures without incorporating physical constraints into the model. When dealing with the large number of variables necessary for characterizing the properties of fractures, the input variables of proxy models are often oversimplified; meanwhile, lots of physical information is lost. Consequently, predictions are sometimes physically inconsistent with the underlying principles of the domain. In this study, by modifying the traditional Seq2Seq (LSTM–LSTM) deep learning architecture, a physics-informed encoder–decoder (PIED) architecture was developed to surrogate the numerical simulation codes for predicting the production of horizontal wells with unequal-length intersecting hydraulic fractures on a 2D plane. The encoder is a LSTM network, and the decoder consists of LSTM and fully connected layers. The attention algorithm is also applied in the Seq2Seq architecture. The PIED model’s encoder is capable of extracting the physical information related to fractures. And the attention module effectively passes on the most relevant physical information related to production to the decoder during the training process. By modifying Seq2Seq architecture, the decoder of the PIED incorporates the intermediate input, which is the constant production time, along with the extracted physical information to predict production values. The PIED model excels in extracting sufficient physical information from high-dimensional inputs while ensuring the integrity of the production time information. By considering the physical constraints, the model predicts production values with improved accuracy and generalization capabilities. In addition, a multi-layer perceptron (MLP) which is broadly used as a proxy model; a regular Seq2Seq model (LSTM–Attention–LSTM); and the PIED were compared via a case study, and their MAE values were shown to be 241.76, 184.07, 168.81, respectively. Therefore, the proposed model has higher accuracy and better generalization ability. In the case study, a comparative experiment was conducted by comparing LSTM–MLP (with an MAE of 221.50) and LSTM–LSTM to demonstrate that using LSTM as the decoder structure is better for predicting production series. Moreover, in the task of predicting production sequences, LSTM outperforms MLP. The Seq2Seq architecture demonstrated excellent performance in this problem, and it achieved a 48.4% reduction in MSE compared to MLP. Meanwhile, the time cost for build datasets was considered, and the proposed model was found to be capable of training in a small dataset (e.g., in the case study, 3 days were used to generate 450 samples for training.); thus, the proposed model has a certain degree of practicality.

A Convolutional Sequence-to-Sequence Attention Fusion Framework for Commonsense Causal Reasoning

Commonsense causal reasoning is the process of understanding the causal dependency between common events or actions. Traditionally, it was framed as a selection problem. However, we cannot obtain enough candidates and need more flexible causes (or effects) in many scenarios, such as causal-based QA problems. Thus, the ability to generate causes (or effects) is an important problem. In this paper, we propose a causal attention mechanism that leverages external knowledge from CausalNet, followed by a novel fusion mechanism that combines global causal dependency guidance from the causal attention with local causal dependency obtained through multi-layer soft attention within the CNN seq2seq architecture. Experimental results consistently demonstrate the superiority of the proposed framework, achieving BLEU-1 scores of 20.06 and 36.94, BLEU-2 scores of 9.98 and 27.78, and human-evaluated accuracy rates of 35% and 52% for two evaluation datasets, outperforming all other baselines across all metrics on both evaluation datasets.

An intelligent chatbot design and implementation model using long short-term memory with recurrent neural networks and attention mechanism

Assistant Conversational Agents (chatbots) are becoming increasingly complex and sophisticated since they are enabled to understand and respond to human language. Chatbots have varying organizational functions, from customer support to personal assistance, and have dominated the digital market as they are efficient and effective. Chatbots can collect valuable information from customers through interaction which can result in a better customer experience, but an inappropriate or inadequate response can also lead to the loss of customers. Therefore, the chatbots must be intelligent enough to provide the most efficient and effective response to positively impact the customers and add value to the business and organization. Over the years, several streams of research have been conducted, and multiple chatbots have been created, but they have their limitation, which has been discussed in this article. We suggest a method to upgrade the performance of the chatbots so that they can respond to the user query with better accuracy. The proposed model shows the implementation of a chatbot using Long Short-Term Memory (LSTM), attention mechanism, Bag of Words (BOW), and beam search decoding. The sequence-to-sequence (Seq2Seq) architecture with an LSTM encoder and decoder has been used. The dialog dataset is preferred to train and test the model, and the BleuN algorithm is used to determine the chatbot’s accuracy.

Short Updates- Machine Learning Based News Summarizer

Automated Text Summarization is becoming important due to the vast amount of data being generated. Manual processing of documents is tedious, mostly due to the absence of standards. Therefore, there is a need for a mechanism to reduce text size, structure it, and make it readable for users. Natural Language Processing (NLP) is critical for analyzing large amounts of unstructured, text-heavy data. This project aims to address concerns with extractive and abstractive text summarization by introducing a new neural network model that deals with repetitive and incoherent phrases in longer documents. The model incorporates a novel Seq2Seq architecture that enhances the standard attentional model with an intra-attention mechanism. Additionally, a new training method that combines supervised word prediction and reinforcement learning is employed. The model utilizes a hybrid pointer-generator network, which distinguishes it from the standard encoder-decoder model. This approach produces higher quality summaries than existing models.

Answering Student Queries with a Supervised Memory Conversational Agent

This paper describes a discussion-bot that provides answers to students’ questions about the Data Science master program at the University of Lyon 1. Based on a seq2seq architecture combined with a supervised memory module, the bot identifies the questioner’s interest and encodes relevant information from the past conversation to provide personalized answers. A dialogue generator based on hand-crafted dialogues was built to train our model on these synthetic dialogues. The agent and its memory are adaptable to another context by modifying the intention database of the generator. The model was deployed and the results show that the discussion-bot meets most students’ learning requests. We discuss further directions that might be taken to increase the model's effectiveness.

Indoor Climate Prediction Using Attention-Based Sequence-to-Sequence Neural Network

The Solar Dryer Dome (SDD), a solar-powered agronomic facility for drying, retaining, and processing comestible commodities, needs smart systems for optimizing its energy consumption. Therefore, indoor condition variables such as temperature and relative humidity need to be forecasted so that actuators can be scheduled, as the largest energy usage originates from actuator activities such as heaters for increasing indoor temperature and dehumidifiers for maintaining optimal indoor humidity. To build such forecasting systems, prediction models based on deep learning for sequence-to-sequence cases were developed in this research, which may bring future benefits for assisting the SDDs and greenhouses in reducing energy consumption. This research experimented with the complex publicly available indoor climate dataset, the Room Climate dataset, which can be represented as environmental conditions inside an SDD. The main contribution of this research was the implementation of the Luong attention mechanism, which is commonly applied in Natural Language Processing (NLP) research, in time series prediction research by proposing two models with the Luong attention-based sequence-to-sequence (seq2seq) architecture with GRU and LSTM as encoder and decoder layers. The proposed models outperformed the adapted LSTM and GRU baseline models. The implementation of Luong attention had been proven capable of increasing the accuracy of the seq2seq LSTM model by reducing its test MAE by 0.00847 and RMSE by 0.00962 on average for predicting indoor temperature, as well as decreasing 0.068046 MAE and 0.095535 RMSE for predicting indoor humidity. The application of Luong's attention also improved the accuracy of the seq2seq GRU model by reducing the error by 0.01163 in MAE and 0.021996 in RMSE for indoor humidity. However, the implementation of Luong attention in seq2seq GRU for predicting indoor temperature showed inconsistent results by reducing approximately 0.003193 MAE and increasing roughly 0.01049 RMSE. Doi: 10.28991/CEJ-2023-09-05-06 Full Text: PDF

A Power System Timing Data Recovery Method Based on Improved VMD and Attention Mechanism Bi-Directional CNN-GRU

The temporal data of the power system are expanding with the growth of the power system and the proliferation of automated equipment. However, data loss may arise during the acquisition, measurement, transmission, and storage of temporal data. To address the insufficiency of temporal data in the power system, this study proposes a sequence-to-sequence (Seq2Seq) architecture to restore power system temporal data. This architecture comprises a radial convolutional neural unit (CNN) network and a gated recurrent unit (GRU) network. Specifically, to account for the periodicity and volatility of temporal data, VMD is employed to decompose the time series data output into components of different frequencies. CNN is utilized to extract the spatial characteristics of temporal data. At the same time, Seq2Seq is employed to reconstruct each component based on introducing a feature timing and multi-model combination triple attention mechanism. The feature attention mechanism calculates the contribution rate of each feature quantity and independently mines the correlation between the time series data output and each feature value. The temporal attention mechanism autonomously extracts historical–critical moment information. A multi-model combination attention mechanism is introduced, and the missing data repair value is obtained after modeling the combination of data on both sides of the missing data. Recovery experiments are conducted based on actual data, and the method’s effectiveness is verified by comparison with other methods.

Methodology for multi-temporal prediction of crop rotations using recurrent neural networks

In a context of growing demand for food and the scarcity of natural resources, the development of more sustainable agriculture is imperative. This means it is necessary to limit the environmental impact of agricultural activities on soil and water and to be mindful of the carbon footprint, while maintaining crop yields and economic benefits for producers. Crop rotation is a valuable tool in sustainable agriculture, but this technique has to be appropriately coupled with sustainable fertilization plans to optimize crops. The proposed methodology uses recurrent neural networks (RNN); more precisely, LSTMs, in a Seq2Seq architecture, to predict the most probable scenarios of crop rotations to be exploited in a field in subsequent growing seasons, according to cropping habits. The output can be used in crop models to build a decision support system for greater sustainability in agricultural production by allowing producers to choose the strategy that offers the best compromise between profitability and environmental impact.

Multi-Graph Convolutional-Recurrent Neural Network (MGC-RNN) for Short-Term Forecasting of Transit Passenger Flow

Short-term forecasting of passenger flow is critical for transit management and crowd regulation. Spatial dependencies, temporal dependencies, inter-station correlations driven by other latent factors, and exogenous factors bring challenges to the short-term forecasts of passenger flow of urban rail transit networks. An innovative deep learning approach, Multi-Graph Convolutional-Recurrent Neural Network (MGC-RNN) is proposed to forecast passenger flow in urban rail transit systems to incorporate these complex factors. We propose to use multiple graphs to encode the spatial and other heterogenous inter-station correlations. The temporal dynamics of the inter-station correlations are also modeled via the proposed multi-graph convolutional-recurrent neural network structure. Inflow and outflow of all stations can be collectively predicted with multiple time steps ahead via a sequence to sequence(seq2seq) architecture. The proposed method is applied to the short-term forecasts of passenger flow in Shenzhen Metro, China. The experimental results show that MGC-RNN outperforms the benchmark algorithms in terms of forecasting accuracy. Besides, it is found that the inter-station driven by network distance, network structure, and recent flow patterns are significant factors for passenger flow forecasting. Moreover, the architecture of LSTM-encoder-decoder can capture the temporal dependencies well. In general, the proposed framework could provide multiple views of passenger flow dynamics for fine prediction and exhibit a possibility for multi-source heterogeneous data fusion in the spatiotemporal forecast tasks.

A new seq2seq architecture for hourly runoff prediction using historical rainfall and runoff as input

• A new sequence to sequence (seq2seq) architecture is developed for hourly runoff prediction. • The new model uses a linear layer in the decoder to process historical runoff instead of using RNN cells. • The new model predicts more accurate hourly runoff than other benchmark seq2seq models. • Prediction accuracy is further improved by combining the new model with the attention mechanism. Machine learning especially deep learning methods have been widely used for runoff prediction in recent years. Models based on the sequence-to-sequence (seq2seq) framework were recently introduced into the field of rainfall-runoff modeling. In this paper, we demonstrate a new design of a seq2seq model which shows higher prediction accuracy than previous seq2seq models. Our new design takes advantage of the high linear correlation between historical runoff and future runoff, and processes the runoff input through a linear layer. Our new seq2seq model, named seq2seq-lrp (linear runoff processing), is combined with the attention mechanism (seq2seq-lrp-a), and compared with three seq2seq benchmark models (seq2seq-1, seq2seq-2, and seq2seq-3). The five models (seq2seq-lrp, seq2seq-lrp-a, seq2seq-1, seq2seq-2, seq2seq-3) were used to simulate runoff in the Yutan sub-catchment, Fujian Province, Southeast China, using hourly runoff and rainfall measurements from 2000 to 2014. The results show that the novel seq2seq-lrp and seq2seq-lrp-a models are better suited in hourly flood prediction than the seq2seq benchmark models and the seq2seq-lrp-a model performs best among the five models.

PGN-LM Model and Forcing-Seq2Seq Model: Multiple Automatic Models of Title Generation for Natural Text using Deep Learning

In the current era, the amount of information from the Internet in general and the electronic press in particular has increased rapidly and has extremely useful information value in all aspects of life, many popular users have posted several high-quality writings as casual blogs, notes or reviews. Some of them are even selected by editors to be published in professional venues. However, the original posts often come without titles, which are needed to be manually added by the editing teams. This task would be done automatically, with the recent advancement of AI techniques, especially deep learning. Even though auto-title can be considered as a specific case of text summarization, this job poses some major different requirements. Basically, a title is generally short but it needs to capture major content while still maintaining the writing style of the original document. To fulfill those constraints, we introduce PGN-LM Model, an architecture evolved from the Pointer Generator Network, with the ability to solve Out-of-Vocabulary problems that traditional Seq2Seq models cannot handle, and at the same time combined with language modeling techniques. In addition, we also introduce a model called Forcing-Seq2Seq Model, an enhanced Seq2Seq architecture, in which the classical TF-IDF scores are incorporated with Named Entity Recognition method to identify the major keywords of the original texts. To enforce the appearance of those keywords in the generated titles, the specific Teacher Forcing mechanism combined with the language model technique are employed. We have tested our approaches with real datasets and obtained promising initial results, on both metrics of machine and human perspectives.