Retrieval-based Dialogue Research Articles

FinKENet: A Novel Financial Knowledge Enhanced Network for Financial Question Matching.

Question matching is the fundamental task in retrieval-based dialogue systems which assesses the similarity between Query and Question. Unfortunately, existing methods focus on improving the accuracy of text similarity in the general domain, without adaptation to the financial domain. Financial question matching has two critical issues: (1) How to accurately model the contextual representation of a financial sentence? (2) How to accurately represent financial key phrases in an utterance? To address these issues, this paper proposes a novel Financial Knowledge Enhanced Network (FinKENet) that significantly injects financial knowledge into contextual text. Specifically, we propose a multi-level encoder to extract both sentence-level features and financial phrase-level features, which can more accurately represent sentences and financial phrases. Furthermore, we propose a financial co-attention adapter to combine sentence features and financial keyword features. Finally, we design a multi-level similarity decoder to calculate the similarity between queries and questions. In addition, a cross-entropy-based loss function is presented for model optimization. Experimental results demonstrate the effectiveness of the proposed method on the Ant Financial question matching dataset. In particular, the Recall score improves from 73.21% to 74.90% (1.69% absolute).

NIR-Prompt: A Multi-task Generalized Neural Information Retrieval Training Framework

Information retrieval aims to find information that meets users’ needs from the corpus. Different needs correspond to different IR tasks such as document retrieval, open-domain question answering, retrieval-based dialogue, and so on, while they share the same schema to estimate the relationship between texts. It indicates that a good IR model can generalize to different tasks and domains. However, previous studies indicate that state-of-the-art neural information retrieval (NIR) models, e.g., pre-trained language models (PLMs) are hard to generalize. It is mainly because the end-to-end fine-tuning paradigm makes the model overemphasize task-specific signals and domain biases but loses the ability to capture generalized essential signals. To address this problem, we propose a novel NIR training framework named NIR-Prompt for retrieval and reranking stages based on the idea of decoupling signal capturing and combination. NIR-Prompt exploits Essential Matching Module (EMM) to capture the essential matching signals and gets the description of tasks by Matching Description Module (MDM). The description is used as task-adaptation information to combine the essential matching signals to adapt to different tasks. Experiments under in-domain multi-task, out-of-domain multi-task, and new task adaptation settings show that NIR-Prompt can improve the generalization of PLMs in NIR for both retrieval and reranking stages compared with baselines.

Improving BERT with local context comprehension for multi-turn response selection in retrieval-based dialogue systems

Multi-turn response selection is an important branch in the field of natural language processing, which aims to select the most appropriate response based on multi-turn dialogue. Most state-of-the-art models adopt pre-trained language models (PrLMs) and multiple auxiliary tasks to enhance their ability to understand the semantics in multi-turn dialogue. However, some critical challenges still remain to be addressed. Optimizing multiple auxiliary tasks simultaneously may significantly increase the training cost. Meanwhile, the semantic gap between the optimization objectives of the main and auxiliary tasks may bring noise to pre-trained language models. To address these challenges, we propose an efficient BERT-based neural network model with local context comprehension (BERT-LCC) for multi-turn response selection. First, we propose a self-supervised learning strategy, which introduces an auxiliary task named Response Prediction in Random Sliding Windows (RPRSW). In a multi-turn dialogue, the RPRSW task takes utterances falling within a random sliding window as input and predicts whether the last utterance within the sliding window is the appropriate response for the local dialogue context. This auxiliary task can enhance BERT’s understanding of local semantic information. Second, we propose a local information fusion (LIF) mechanism that collects multi-granularity local features at different dialogue stages and employs a gating function to fuse global features with local features. Third, we introduce a simple but effective domain learning strategy to learn rich semantic information at different dialogue stages during pre-training. Experimental results on two public benchmark datasets show that BERT-LCC outperforms other state-of-the-art models.

BERT-based response selection in dialogue systems using utterance attention mechanisms

Dialogue systems, one of the core research fields of natural language processing, attempt to understand the utterances of a user and generate an appropriate response. Response selection in a retrieval-based dialogue system involves searching for the most context-appropriate subsequent utterance. Conversations are usually composed of multiple turns; therefore, the intention of the speaker must be properly understood prior to the response selection. To accurately capture such intended meaning, we propose a retrieval-based response selection model that effectively comprehends the relationships among words and utterances in a conversation and a response candidate with word and utterance attention. Word representation is generated by using the self-attention mechanism to reflect the contextual information between intentional words in an overall conversation or individual utterance, while utterance representation is by the cross-attention mechanism to reflect the contextual information among utterances. Furthermore, since our model does not need much computation and memory size, it can be easily combined with existing other response selection models or pre-trained language models. Experiments on various utterance embedding methods were also conducted to find a proper representation of the utterance information. Our proposed model exhibits an improvement in performance of approximately 2.1%p in hit@1 in the DSTC8 ubuntu dataset compared to baseline models, as well as significant performance improvements for other datasets.

Response Ranking with Multi-types of Deep Interactive Representations in Retrieval-based Dialogues

Building an intelligent dialogue system with the ability to select a proper response according to a multi-turn context is challenging in three aspects: (1) the meaning of a context–response pair is built upon language units from multiple granularities (e.g., words, phrases, and sub-sentences, etc.); (2) local (e.g., a small window around a word) and long-range (e.g., words across the context and the response) dependencies may exist in dialogue data; and (3) the relationship between the context and the response candidate lies in multiple relevant semantic clues or relatively implicit semantic clues in some real cases. However, existing approaches usually encode the dialogue with mono-type representation and the interaction processes between the context and the response candidate are executed in a rather shallow manner, which may lead to an inadequate understanding of dialogue content and hinder the recognition of the semantic relevance between the context and response. To tackle these challenges, we propose a representation [ K ] -interaction [ L ] -matching framework that explores multiple types of deep interactive representations to build context-response matching models for response selection. Particularly, we construct different types of representations for utterance–response pairs and deepen them via alternate encoding and interaction. By this means, the model can handle the relation of neighboring elements, phrasal pattern, and long-range dependencies during the representation and make a more accurate prediction through multiple layers of interactions between the context–response pair. Experiment results on three public benchmarks indicate that the proposed model significantly outperforms previous conventional context-response matching models and achieve slightly better results than the BERT model for multi-turn response selection in retrieval-based dialogue systems.

Topic-Aware Multi-turn Dialogue Modeling

In the retrieval-based multi-turn dialogue modeling, it remains a challenge to select the most appropriate response according to extracting salient features in context utterances. As a conversation goes on, topic shift at discourse-level naturally happens through the continuous multi-turn dialogue context. However, all known retrieval-based systems are satisfied with exploiting local topic words for context utterance representation but fail to capture such essential global topic-aware clues at discourse-level. Instead of taking topic-agnostic n-gram utterance as processing unit for matching purpose in existing systems, this paper presents a novel topic-aware solution for multi-turn dialogue modeling, which segments and extracts topic-aware utterances in an unsupervised way, so that the resulted model is capable of capturing salient topic shift at discourse-level in need and thus effectively track topic flow during multi-turn conversation. Our topic-aware modeling is implemented by a newly proposed unsupervised topic-aware segmentation algorithm and Topic-Aware Dual-attention Matching (TADAM) Network, which matches each topic segment with the response in a dual cross-attention way. Experimental results on three public datasets show TADAM can outperform the state-of-the-art method, especially by 3.3% on E-commerce dataset that has an obvious topic shift.

Filling the Gap of Utterance-aware and Speaker-aware Representation for Multi-turn Dialogue

A multi-turn dialogue is composed of multiple utterances from two or more different speaker roles. Thus utterance- and speaker-aware clues are supposed to be well captured in models. However, in the existing retrieval-based multi-turn dialogue modeling, the pre-trained language models (PrLMs) as encoder represent the dialogues coarsely by taking the pairwise dialogue history and candidate response as a whole, the hierarchical information on either utterance interrelation or speaker roles coupled in such representations is not well addressed. In this work, we propose a novel model to fill such a gap by modeling the effective utterance-aware and speaker-aware representations entailed in a dialogue history. In detail, we decouple the contextualized word representations by masking mechanisms in Transformer-based PrLM, making each word only focus on the words in current utterance, other utterances, two speaker roles (i.e., utterances of sender and utterances of receiver), respectively. Experimental results show that our method boosts the strong ELECTRA baseline substantially in four public benchmark datasets, and achieves various new state-of-the-art performance over previous methods. A series of ablation studies are conducted to demonstrate the effectiveness of our method.

Leveraging Linguistic Coordination in Reranking N-Best Candidates For End-to-End Response Selection Using BERT

Retrieval-based dialogue systems select the best response from many candidates. Although many state-of-the-art models have shown promising performance in dialogue response selection tasks, there is still quite a gap between R@1 and R@10 performance. To address this, we propose to leverage linguistic coordination (a phenomenon that individuals tend to develop similar linguistic behaviors in conversation) to rerank the N-best candidates produced by BERT, a state-of-the-art pre-trained language model. Our results show an improvement in R@1 compared to BERT baselines, demonstrating the utility of repairing machine-generated outputs by leveraging a linguistic theory.

Toward hardware-aware deep-learning-based dialogue systems

In the past few years, the use of transformer-based models has experienced increasing popularity as new state-of-the-art performance was achieved in several natural language processing tasks. As these models are often extremely large, however, their use for applications within embedded devices may not be feasible. In this work, we look at one such specific application, retrieval-based dialogue systems, that poses additional difficulties when deployed in environments characterized by limited resources. Research on building dialogue systems able to engage in natural sounding conversation with humans has attracted increasing attention in recent years. This has led to the rise of commercial conversational agents, such as Google Home, Alexa and Siri situated on embedded devices, that enable users to interface with a wide range of underlying functionalities in a natural and seamless manner. In part due to memory and computational power constraints, these agents necessitate frequent communication with a server in order to process the users’ queries. This communication may act as a bottleneck, resulting in delays as well as in the halt of the system should the network connection be lost or unavailable. We propose a new framework for hardware-aware retrieval-based dialogue systems based on the Dual-Encoder architecture, coupled with a clustering method to group candidates pertaining to a same conversation, that reduces storage capacity and computational power requirements.

Randomly Wired Network Based on RoBERTa and Dialog History Attention for Response Selection

Recently, chatbot research for conversation has been actively conducted in the field of natural language processing. Among the dialogue systems, a retrieval-based system has emerged as an important one. Because, it showed better performance than the response generation models and it actually seemed applicable. The response selection task is used to develop a retrieval-based system. Thus, we did research to improve the performance of retrieval-based dialogue systems for response selection. We used Google's bidirectional encoder representations from transformers (BERT) natural-language processor with the robustly optimized BERT pre-training model (RoBERTa) as the base model to improve performance. The Recall@N of our model was improved by 1.7–4.2% using two methods: First, instead of using a simple feed-forward network at the end of the model, we employed a randomly wired neural network that contained multiple wiring paths, thus achieving better performance than that of feed-forward networks. The experimental results of this study demonstrated that feed-forward networks can be replaced by randomly wired neural networks. Second, we calculated the correlation between dialog history (i.e., context) and the last utterance of the context, reflecting this correlation as an attention network that yielded an answer prediction. Using first and second method achieved higher performance than using only the first method in dialog response selection tasks.

Towards Simple but Efficient Next Utterance Ranking

Retrieval-based dialogue systems converse with humans by ranking candidate responses according to their relevance to the history of the conversation (context). Recent studies either match the context with the response on only sequence level or use complex architectures to match them on the word and sequence levels. We show that both information levels are important and that a simple architecture can capture them effectively. We propose an end to endmulti level response retrieval dialogue system. Our model learns to match the context with the best response by computing their semantic similarity on the word and sequence levels. Empirical evaluation on two dialogue datasets shows that our model outperforms several state of the art systems and performs as good as the best system while being conceptually simpler.