Short Text Conversation Research Articles

An Emotional Dialogue System Using Conditional Generative Adversarial Networks with a Sequence-to-Sequence Transformer Encoder

Understanding the expression of emotion and generating appropriate responses are key steps toward constructing emotional, conversational agents. In this paper, we propose a framework for single-turn emotional conversation generation, and there are three main components in our model, namely, a sequence-to-sequence model with stacked encoders, a conditional variational autoencoder, and conditional generative adversarial networks. For the sequence-to-sequence model with stacked encoders, we designed a two-layer encoder by combining Transformer with gated recurrent units-based neural networks. Because of the flexibility of the sequence-to-sequence model, we adopted a conditional variational autoencoder in our framework, which uses latent variables to learn a distribution over potential responses and generates diverse responses. Furthermore, we regard a conditional variational autoencoder-based, sequence-to-sequence model as the generative model, and the training of the generative model is assisted by both a content discriminator and an emotion classifier, which assists our model in promoting content information and emotion expression. We use automated evaluation and human evaluation to evaluate our model and baselines on the NTCIR short text conversation task (STC-3) Chinese emotional conversation generation (CECG) Subtask dataset [44], and the experimental results demonstrate that our proposed framework can generate semantically reasonable and emotionally appropriate responses.

Condition-Transforming Variational Autoencoder for Generating Diverse Short Text Conversations

In this article, conditional-transforming variational autoencoders (CTVAEs) are proposed for generating diverse short text conversations. In conditional variational autoencoders (CVAEs), the prior distribution of latent variable z follows a multivariate Gaussian distribution with mean and variance modulated by the input conditions. Previous work found that this distribution tended to become condition-independent in practical applications. Thus, this article designs CTVAEs to enhance the influence of conditions in CVAEs. In a CTVAE model, the latent variable z is sampled by performing a non-linear transformation on the combination of the input conditions and the samples from a condition-independent prior distribution N (0, I). In our experiments using a Chinese Sina Weibo dataset, the CTVAE model derives z samples for decoding with better condition-dependency than that of the CVAE model. The earth mover’s distance (EMD) between the distributions of the latent variable z at the training stage, and the testing stage is also reduced by using the CTVAE model. In subjective preference tests, our proposed CTVAE model performs significantly better than CVAE and sequence-to-sequence (Seq2Seq) models on generating diverse, informative, and topic-relevant responses.

Relevance-Promoting Language Model for Short-Text Conversation

Despite the effectiveness of sequence-to-sequence framework on the task of Short-Text Conversation (STC), the issue of under-exploitation of training data (i.e., the supervision signals from query text is ignored) still remains unresolved. Also, the adopted maximization-based decoding strategies, inclined to generating the generic responses or responses with repetition, are unsuited to the STC task. In this paper, we propose to formulate the STC task as a language modeling problem and tailor-make a training strategy to adapt a language model for response generation. To enhance generation performance, we design a relevance-promoting transformer language model, which performs additional supervised source attention after the self-attention to increase the importance of informative query tokens in calculating the token-level representation. The model further refines the query representation with relevance clues inferred from its multiple references during training. In testing, we adopt a randomization-over-maximization strategy to reduce the generation of generic responses. Experimental results on a large Chinese STC dataset demonstrate the superiority of the proposed model on relevance metrics and diversity metrics.1

Seq2Seq models for recommending short text conversations

The massive amounts of data on social media networks can be overwhelming for users; for this reason, recommending relevant content becomes an essential task to avoid information overload. In this paper, we propose a new task for recommending users that might be interested in join conversations on specific domains. To that end, we introduce a new corpus that contains conversations threads from popular users on Twitter on domains such as politics, sports, or humanitarian activism. Modeling short-text conversations on microblogs can be difficult because user-generated data is unstructured and noisy. Previous works focused on recommending content to users based on latent factors models and collaborative filtering methods. We propose a state-of-the-art recommendation model based on a sequence-to-sequence neural architecture that encodes the text of users’ profiles and the conversations’ context using several variants of recurrent neural networks. The experimental results show that our method provides as much as 20% higher recall compared to baseline methods. Moreover, we use an end-to-end learning framework that allows downstream applications to use recommender systems (RSs) that generalize better to new content by using pre-trained embeddings, thus being useful across domains or events.

A short text conversation generation model combining BERT and context attention mechanism

The standard Seq2Seq neural network model tends to generate general and safe responses (e.g., I don't know) regardless of the input in the field of short text conversation generation. To address this problem, we propose a novel model that combines the standard Seq2Seq model with the BERT module (a pre-trained model) to improve the quality of responses. Specifically, the encoder of the model is divided into two parts: one is the standard Seq2Seq which generates a context attention vector; the other is the improved BERT module which encodes the input sentence into a semantic vector. Then through a fusion unit, the vectors generated by the two parts are fused to generate a new attention vector. Finally, the new attention vector is transmitted to the decoder. In particular, we describe two ways to acquire a new attention vector in the fusion unit. Empirical results from automatic and human evaluations demonstrate that our model improves the quality and diversity of the responses significantly.

A short text conversation generation model combining BERT and context attention mechanism

The standard Seq2Seq neural network model tends to generate general and safe responses (e.g., I don't know) regardless of the input in the field of short text conversation generation. To address this problem, we propose a novel model that combines the standard Seq2Seq model with the BERT module (a pre-trained model) to improve the quality of responses. Specifically, the encoder of the model is divided into two parts: one is the standard Seq2Seq which generates a context attention vector; the other is the improved BERT module which encodes the input sentence into a semantic vector. Then through a fusion unit, the vectors generated by the two parts are fused to generate a new attention vector. Finally, the new attention vector is transmitted to the decoder. In particular, we describe two ways to acquire a new attention vector in the fusion unit. Empirical results from automatic and human evaluations demonstrate that our model improves the quality and diversity of the responses significantly.

Expanding Comments Using Previously Responded Posts for Short Text Conversation in Open Social Media

Expanding Comments Using Previously Responded Posts for Short Text Conversation in Open Social Media

Generating Multiple Diverse Responses for Short-Text Conversation

Neural generative models have become popular and achieved promising performance on short-text conversation tasks. They are generally trained to build a 1-to-1 mapping from the input post to its output response. However, a given post is often associated with multiple replies simultaneously in real applications. Previous research on this task mainly focuses on improving the relevance and informativeness of the top one generated response for each post. Very few works study generating multiple accurate and diverse responses for the same post. In this paper, we propose a novel response generation model, which considers a set of responses jointly and generates multiple diverse responses simultaneously. A reinforcement learning algorithm is designed to solve our model. Experiments on two short-text conversation tasks validate that the multiple responses generated by our model obtain higher quality and larger diversity compared with various state-ofthe-art generative models.

Improving explicit term matching with implicit topic matching for short text conversation

ABSTRACTShort Text Conversation (STC) is the task of retrieving comments made to a post and reuse them as appropriate comments to a later post. Large vocabulary variations and very limited length of posts and comments pose great challenges to traditional retrieval techniques of matching terms. To address these problems, we propose a new technique of re‐ranking comments initially retrieved through explicit term matching by using evidence of implicit topic matching gained from topic models of large collections of posts and comments. The key of our technique lies in (1) combining each post with the comments responding to it into a longer pseudo‐document for topic modeling purposes, and (2) carrying out implicit topic matching by comparing the most probable topics contained in the topic composition of a comment and that of the query post. Experiments using the NTCIR‐13 STC Chinese WeiBo collection showed that with properly tuned parameters, our technique can lead to improved retrieval effectiveness. The paper analyzes and discusses the results and outlines future work.

Rich Short Text Conversation Using Semantic-Key-Controlled Sequence Generation

With the recent advances of the sequence-to-sequence framework, generation approaches for the short text conversation STC become attractive. The traditional sequence-to-sequence approaches for the STC often suffer from poor diversity and general reply without substantiality. It is also hard to control the topic or semantics of the selected reply from multiple generated candidates. In this paper, a novel external-memory-driven sequence-to-sequence learning approach is proposed to address these problems. A tensor of the external memory is constructed to represent interpretable topics or semantics. During generation, a controllable memory trigger is extracted given the input sequence, and a reply is then generated using the memory trigger as well as the sequence-to-sequence model. Experiments show that the proposed approach can generate much richer diversity than the traditional sequence-to-sequence training with attention. Meanwhile, it achieves better quality score in human evaluation. It is also observed that by manually manipulating the memory trigger, it is possible to interpretably guide the topics or semantics of the reply.

Who is answering whom? Finding “Reply-To” relations in group chats with deep bidirectional LSTM networks

Social networks facilitate communication among Internet users while generating large volumes of online short-text conversations every day. This leads to a huge number of free-style asynchronous conversations where multiple users are involved and multiple topics are discussed at the same time in the same place, e.g., an instant group chat in WeChat. Here emerges an interesting problem: as a result of a large number of users and topics, the conversation structure may get into a mess, which often interferes with the acquisition of messages users are interested in. For example, when a user enters a conversation, (s)he usually does not want to read all the historical messages, but just hope to get the messages that are the most relevant to some messages (s)he cares about. Therefore, it is an essential task to understand the logical correlations among messages, which benefits text mining, natural language processing, and web intelligence techniques. In this paper, we focus on “reply-to” relations, such as QA The other two versions are end-to-end models, which are based on the word level and the sentence level, respectively. Finally, experimental results conducted on two real-world group chat data sets demonstrate the effectiveness of the proposed model.