Representation Of Utterance Research Articles

ГРАММАТИКА СЛУШАЮЩЕГО: ПРОГНОЗИРОВАНИЕ ПОНИМАНИЯ СВЯЗНОГО ТЕКСТА И СПОСОБЫ ЕГО КОРРЕКЦИИ

Статья продолжает тематику работ по созданию грамматики слушающего и посвящена пониманию связного текста в аспекте прогнозирования его продолжения слушающим. Выявляются основные стратегии развертывания текста. Показывается возможность использования частиц, союзов и других дискурсивных слов для коррекции прогноза. Основные выводы делаются на основе русскоязычного материала, однако, поскольку предполагается их универсальный характер, приводятся данные сравнения русского языка с рядом европейских. The article makes part of the investigations focused on the grammar of Addressee: (the Hearer). The investigations in the Active Grammar (the Grammar of the Speaker) made it possible to describe various phenomena in grammar and lexis of many languages. Such branch of the functional linguistics as Receptive grammar or the Grammar of Addressee (the Hearer) is now being developed. It concerns the description of language entities that enable proper understanding the text in step-by-step analyzing it. The consequence of the Addresse’s activity is supposed to be the following: perception, deciphering meanings, reference, revealing implicatures, forming semantic representation of utterances and texts, revealing intentions of the Speaker, marking sentimental characteristics. These hypothetic actions form the framework of this Grammar. And the categories related to these stages are formed by the language entities necessary for this activity. Such categories as ‘Orientation’ (referring an utterance to the situation), ‘explication of implicatures (inferences)’ were investigated and described. This article concerns the activity of the Hearer while understanding the content of a text, namely taking into consideration it’s possible continuation. This article concerns the activity of the Hearer while understanding the content of a text, namely taking into consideration it’s possible continuation. The principal components of the text composition and three types of texts: narrative, description, dissertation — are taken into consideration. The most common strategies of the continuation of narrative are listed. The speaker has means that help him to correct possible inferences and expectations for the continuation of the text. He uses the markers (conjunctions, particles, adverbials) that enable the proper understanding of texts. Some of them: conjunctions and, but and their translations into other languages, Russian modal particles vot, tam, adverbials, etc. are supposed to be described. The principles of the Hearer’s activity are supposed to be more or less universal and valid for all languages (or the majority Теория языка на материале лингвистической компаративистики 263 of them). So the basic rules are elaborated on Russian examples while comparison with other languages, English, first of all — were also provided. The further investigations in the sphere of understanding and the grammar of the Hearer were offered

Contextual and Cross-Modal Interaction for Multi-Modal Speech Emotion Recognition

Speech emotion recognition combining linguistic content and audio signals in the dialog is a challenging task. Nevertheless, previous approaches have failed to explore emotion cues in contextual interactions and ignored the long-range dependencies between elements from different modalities. To tackle the above issues, this letter proposes a multimodal speech emotion recognition method using audio and text data. We first present a contextual transformer module to introduce contextual information via embedding the previous utterances between interlocutors, which enhances the emotion representation of the current utterance. Then, the proposed cross-modal transformer module focuses on the interactions between text and audio modalities, adaptively promoting the fusion from one modality to another. Furthermore, we construct associative topological relation over mini-batch and learn the association between deep fused features with graph convolutional network. Experimental results on the IEMOCAP and MELD datasets show that our method outperforms current state-of-the-art methods.

A multi-view network for real-time emotion recognition in conversations

Real-time emotion recognition in conversations (RTERC), the task of using the historical context to identify the emotion of a query utterance in a conversation, is important for opinion mining and building empathetic machines. Existing works mainly focus on obtaining each utterance representation separately and then utilizing utterance-level features to model the emotion representation of the query. These approaches treat each utterance as a unit and capture the utterance-level dependencies in the context, but ignore the word-level dependencies among different utterances. In this paper, we propose a multi-view network (MVN) to explore the emotion representation of a query from two different views, i.e., word- and utterance-level views. For the word-level view, MVN takes the context and query as word sequences and then models the word-level dependencies among utterances. For the utterance-level view, MVN extracts each utterance representation separately and then models the utterance-level dependencies in the context. Experimental results on two public emotion conversation datasets show that the proposed model outperforms the state-of-the-art baselines.

H-VECTORS: Improving the robustness in utterance-level speaker embeddings using a hierarchical attention model

In this paper, a hierarchical attention network is proposed to generate robust utterance-level embeddings (H-vectors) for speaker identification and verification. Since different parts of an utterance may have different contributions to speaker identities, the use of hierarchical structure aims to learn speaker related information locally and globally. In the proposed approach, frame-level encoder and attention are applied on segments of an input utterance and generate individual segment vectors. Then, segment level attention is applied on the segment vectors to construct an utterance representation. To evaluate the quality of the learned utterance-level speaker embeddings on speaker identification and verification, the proposed approach is tested on several benchmark datasets, such as the NIST SRE2008 Part1, the Switchboard Cellular (Part1), the CallHome American English Speech ,the Voxceleb1 and Voxceleb2 datasets. In comparison with some strong baselines, the obtained results show that the use of H-vectors can achieve better identification and verification performances in various acoustic conditions.

Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection

Real-life applications, heavily relying on machine learning, such as dialog systems, demand for out-of-domain detection methods. Intent classification models should be equipped with a mechanism to distinguish seen intents from unseen ones so that the dialog agent is capable of rejecting the latter and avoiding undesired behavior. However, despite increasing attention paid to the task, the best practices for out-of-domain intent detection have not yet been fully established. This paper conducts a thorough comparison of out-of-domain intent detection methods. We prioritize the methods, not requiring access to out-of-domain data during training, gathering of which is extremely time- and labor-consuming due to lexical and stylistic variation of user utterances. We evaluate multiple contextual encoders and methods, proven to be efficient, on three common datasets for intent classification, expanded with out-of-domain utterances. Our main findings show that fine-tuning Transformer-based encoders on in-domain data leads to superior results. Mahalanobis distance, together with utterance representations, derived from Transformer-based encoders, outperform other methods by a wide margin(1-5% in terms of AUROC) and establish new state-of-the-art results for all datasets. The broader analysis shows that the reason for success lies in the fact that the fine-tuned Transformer is capable of constructing homogeneous representations of in-domain utterances, revealing geometrical disparity to out of domain utterances. In turn, the Mahalanobis distance captures this disparity easily.

Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

Most recently, there has been significant interest in learning contextual representations for various NLP tasks, by leveraging large scale text corpora to train powerful language models with self-supervised learning objectives, such as Masked Language Model (MLM). Based on a pilot study, we observe three issues of existing general-purpose language models when they are applied in the text-to-SQL semantic parsers: fail to detect the column mentions in the utterances, to infer the column mentions from the cell values, and to compose target SQL queries when they are complex. To mitigate these issues, we present a model pretraining framework, Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterance and table schemas, by leveraging generation models to generate high-quality pre-train data. GAP Model is trained on 2 million utterance-schema pairs and 30K utterance-schema-SQL triples, whose utterances are generated by generation models. Based on experimental results, neural semantic parsers that leverage GAP Model as a representation encoder obtain new state-of-the-art results on both Spider and Criteria-to-SQL benchmarks.

Time to Transfer: Predicting and Evaluating Machine-Human Chatting Handoff

Is chatbot able to completely replace the human agent? The short answer could be – ``it depends...''. For some challenging cases, e.g., dialogue's topical spectrum spreads beyond the training corpus coverage, the chatbot may malfunction and return unsatisfied utterances. This problem can be addressed by introducing the Machine-Human Chatting Handoff (MHCH) which enables human-algorithm collaboration. To detect the normal/transferable utterances, we propose a Difficulty-Assisted Matching Inference (DAMI) network, utilizing difficulty-assisted encoding to enhance the representations of utterances. Moreover, a matching inference mechanism is introduced to capture the contextual matching features. A new evaluation metric, Golden Transfer within Tolerance (GT-T), is proposed to assess the performance by considering the tolerance property of the MHCH. To provide insights into the task and validate the proposed model, we collect two new datasets. Extensive experimental results are presented and contrasted against a series of baseline models to demonstrate the efficacy of our model on MHCH.

Tracking Interaction States for Multi-Turn Text-to-SQL Semantic Parsing

The task of multi-turn text-to-SQL semantic parsing aims to translate natural language utterances in an interaction into SQL queries in order to answer them using a database which normally contains multiple table schemas. Previous studies on this task usually utilized contextual information to enrich utterance representations and to further influence the decoding process. While they ignored to describe and track the interaction states which are determined by history SQL queries and are related with the intent of current utterance. In this paper, two kinds of interaction states are defined based on schema items and SQL keywords separately. A relational graph neural network and a non-linear layer are designed to update the representations of these two states respectively. The dynamic schema-state and SQL-state representations are then utilized to decode the SQL query corresponding to current utterance. Experimental results on the challenging CoSQL dataset demonstrate the effectiveness of our proposed method, which achieves better performance than other published methods on the task leaderboard.

Topic-Aware Multi-turn Dialogue Modeling

In the retrieval-based multi-turn dialogue modeling, it remains a challenge to select the most appropriate response according to extracting salient features in context utterances. As a conversation goes on, topic shift at discourse-level naturally happens through the continuous multi-turn dialogue context. However, all known retrieval-based systems are satisfied with exploiting local topic words for context utterance representation but fail to capture such essential global topic-aware clues at discourse-level. Instead of taking topic-agnostic n-gram utterance as processing unit for matching purpose in existing systems, this paper presents a novel topic-aware solution for multi-turn dialogue modeling, which segments and extracts topic-aware utterances in an unsupervised way, so that the resulted model is capable of capturing salient topic shift at discourse-level in need and thus effectively track topic flow during multi-turn conversation. Our topic-aware modeling is implemented by a newly proposed unsupervised topic-aware segmentation algorithm and Topic-Aware Dual-attention Matching (TADAM) Network, which matches each topic segment with the response in a dual cross-attention way. Experimental results on three public datasets show TADAM can outperform the state-of-the-art method, especially by 3.3% on E-commerce dataset that has an obvious topic shift.

Learning to Respond with Your Favorite Stickers

Stickers with vivid and engaging expressions are becoming increasingly popular in online messaging apps, and some works are dedicated to automatically select sticker response by matching the stickers image with previous utterances. However, existing methods usually focus on measuring the matching degree between the dialog context and sticker image, which ignores the user preference of using stickers. Hence, in this article, we propose to recommend an appropriate sticker to user based on multi-turn dialog context and sticker using history of user. Two main challenges are confronted in this task. One is to model the sticker preference of user based on the previous sticker selection history. Another challenge is to jointly fuse the user preference and the matching between dialog context and candidate sticker into final prediction making. To tackle these challenges, we propose a Preference Enhanced Sticker Response Selector (PESRS) model. Specifically, PESRS first employs a convolutional-based sticker image encoder and a self-attention-based multi-turn dialog encoder to obtain the representation of stickers and utterances. Next, deep interaction network is proposed to conduct deep matching between the sticker and each utterance. Then, we model the user preference by using the recently selected stickers as input and use a key-value memory network to store the preference representation. PESRS then learns the short-term and long-term dependency between all interaction results by a fusion network and dynamically fuses the user preference representation into the final sticker selection prediction. Extensive experiments conducted on a large-scale real-world dialog dataset show that our model achieves the state-of-the-art performance for all commonly used metrics. Experiments also verify the effectiveness of each component of PESRS.

A Multiple-Integration Encoder for Multi-Turn Text-to-SQL Semantic Parsing

This paper studies multi-turn text-to-SQL generation, which is a new but important task in semantic parsing. In order to deal with its two challenges, i.e., multi-turn interaction and cross-domain evaluation, this paper proposes a multiple-integration encoder, which derives the vector representations of user utterances and database schemas using three custom-designed modules for information integration. First, an utterance representation enhancing module is built to integrate the information of history utterances into the representation of each token in current utterance by attentive selection. Second, a schema discrepancy enhancing module is designed to integrate previous predicted SQL query into the representation of schema items. Third, a latent schema linking module is employed to integrate schema information into utterance representations for better dealing with unseen database schemas. These three modules are all implemented based on a lightweight multi-head attention mechanism, which reduces the number of parameters in conventional multi-head attention. Experimental results on the SParC dataset show that our method achieved better accuracy of multi-turn text-to-SQL generation than the most advanced benchmarks. Further ablations studies and analysis also demonstrate the effectiveness of the three modules designed for information integration in the encoder.

Estimating Uniqueness of I-Vector-Based Representation of Human Voice

We study the individuality of the human voice with respect to a widely used feature representation of speech utterances, namely, the i-vector model. As a first step toward this goal, we compare and contrast uniqueness measures proposed for different biometric modalities. Then, we introduce a new uniqueness measure that evaluates the entropy of i-vectors while taking into account speaker level variations. Our measure operates in the discrete feature space and relies on accurate estimation of the distribution of i-vectors. Therefore, i-vectors are quantized while ensuring that both the quantized and original representations yield similar speaker verification performance. Uniqueness estimates are obtained from two newly generated datasets and the public VoxCeleb dataset. The first custom dataset contains more than one and a half million speech samples of 20,741 speakers obtained from TEDx Talks videos. The second one includes over twenty one thousand speech samples from 1,595 actors that are extracted from movie dialogues. Using this data, we analyzed how several factors, such as the number of speakers, number of samples per speaker, sample durations, and diversity of utterances affect uniqueness estimates. Most notably, we determine that the discretization of i-vectors does not cause a reduction in speaker recognition performance. Our results show that the degree of distinctiveness offered by i-vector-based representation may reach 43–70 bits considering 5-second long speech samples; however, under less constrained variations in speech, uniqueness estimates are found to reduce by around 30 bits. We also find that doubling the sample duration increases the distinctiveness of the i-vector representation by around 20 bits.

Deep Contextualized Utterance Representations for Response Selection and Dialogue Analysis

The NOESIS II challenge, as the Track 2 in the Eighth Dialogue System Technology Challenge (DSTC 8), is the extension of Track 1 in DSTC 7. Three new elements are incorporated into the extended track, i.e., dialogue with multiple participants, dialogue success, and dialogue disentanglement. These are vital for the creation of a deployed task-oriented dialogue system. This track is divided into four subtasks, the first two of which are evaluated in the form of response selection and the last two focus on dialogue analysis. This paper describes our methods developed for these four subtasks, which all employ deep contextualized utterance representations to make models aware of contextual information and to keep the intrinsic property of multi-turn dialogue systems. In the released evaluation results of Track 2 in DSTC 8, our proposed methods ranked fourth in subtask 1, third in subtask 2, and first in subtask 3 and subtask 4 respectively. In addition to the challenge tasks, we also compare our proposed methods with previous ones on public benchmark datasets. Experimental results show that our proposed methods outperform existing ones by large margins and achieve new state-of-the-art performances on multi-turn response selection and dialogue disentanglement.

On the Distribution of Speaker Verification Scores: Generative Models for Unsupervised Calibration

Speaker verification systems whose outputs can be interpreted as log-likelihood ratios (LLR) allow for cost-effective decisions by comparing the system outputs to application-defined thresholds depending only on prior information. Classifiers often produce uncalibrated scores, and require additional processing to produce well-calibrated LLRs. Recently, generative score calibration models have been proposed, which achieve calibration performance close to that of state-of-the-art discriminative techniques for supervised scenarios, while also allowing for unsupervised training. The effectiveness of these methods, however, strongly depends on their capabilities to correctly model the target and non-target score distributions. In this work we propose theoretically grounded and accurate models for characterizing the distribution of scores of speaker verification systems. Our approach is based on tied Generalized Hyperbolic distributions and overcomes many limitations of Gaussian models. Experimental results on different NIST benchmarks, using different utterance representation front-ends and different back-end classifiers, show that our method is effective not only in supervised scenarios, but also in unsupervised tasks characterized by very low proportion of target trials.

Multi-turn intent determination and slot filling with neural networks and regular expressions

Intent determination and slot filling are two prominent research areas related to natural language understanding (NLU). In a multi-turn NLU system, contextual information from dialogue history is exploited to mitigate the ambiguity of user utterance. State-of-the-art models employ memory networks to encode dialogue context, which is used by neural networks for determining user intent and associated slots. However, these methods rely on a large amount of labelled data, whereas we often have limited labelled data. To address this problem, we propose a multi-task learning model based on neural networks and regular expressions (REs), to jointly perform intent determination and slot filling tasks. The proposed model integrates neural networks with REs to encode domain knowledge and handle cases with a limited amount of labelled data in an end-to-end trainable manner. More specifically, the model employs a pre-trained BERT model to obtain contextual word representations of user utterances. These representations are utilized by a memory network to encode multi-turn information which is shared by the tasks. Furthermore, the convolutional neural network (CNN) and the recurrent neural network (RNN) are applied to contextual word representations and dialogue context for intent determination and slot filling tasks, respectively. These neural networks are then combined with REs which encode domain knowledge about a particular intent or slot value. Finally, the two neural networks are trained simultaneously by minimizing the joint loss. Extensive experiments on Key-Value Retrieval and Frames datasets show that the proposed model outperforms baseline methods in both tasks while requiring modest human effort.

Sentiment Classification in Customer Service Dialogue with Topic-Aware Multi-Task Learning

Sentiment analysis in dialogues plays a critical role in dialogue data analysis. However, previous studies on sentiment classification in dialogues largely ignore topic information, which is important for capturing overall information in some types of dialogues. In this study, we focus on the sentiment classification task in an important type of dialogue, namely customer service dialogue, and propose a novel approach which captures overall information to enhance the classification performance. Specifically, we propose a topic-aware multi-task learning (TML) approach which learns topic-enriched utterance representations in customer service dialogue by capturing various kinds of topic information. In the experiment, we propose a large-scale and high-quality annotated corpus for the sentiment classification task in customer service dialogue and empirical studies on the proposed corpus show that our approach significantly outperforms several strong baselines.

Learning Hierarchical Representations of Stories by Using Multi-layered Structures in Narrative Multimedia.

Narrative works (e.g., novels and movies) consist of various utterances (e.g., scenes and episodes) with multi-layered structures. However, the existing studies aimed to embed only stories in a narrative work. By covering other granularity levels, we can easily compare narrative utterances that are coarser (e.g., movie series) or finer (e.g., scenes) than a narrative work. We apply the multi-layered structures on learning hierarchical representations of the narrative utterances. To represent coarser utterances, we consider adjacency and appearance of finer utterances in the coarser ones. For the movies, we suppose a four-layered structure (character roles ∈ characters ∈ scenes ∈ movies) and propose three learning methods bridging the layers: Char2Vec, Scene2Vec, and Hierarchical Story2Vec. Char2Vec represents a character by using dynamic changes in the character’s roles. To find the character roles, we use substructures of character networks (i.e., dynamic social networks of characters). A scene describes an event. Interactions between characters in the scene are designed to describe the event. Scene2Vec learns representations of a scene from interactions between characters in the scene. A story is a series of events. Meanings of the story are affected by order of the events as well as their content. Hierarchical Story2Vec uses sequential order of scenes to represent stories. The proposed model has been evaluated by estimating the similarity between narrative utterances in real movies.

A Study of Irony in Political Discourse

This study examines the ideological and evaluative nature of irony in discourse. It aims to reveal the linguistic constructions involved in ironic utterances and to uncover ideologies underpinning such structures. It draws upon the echo and pretense theories to account for utterance representation along with Wodak’s (2007) framework of self and other presentation strategies. The study examines the incongruous construction of an attitude, and the objectives attempted to achieve by taking a cynical stance. The main research questions center on the role played by ironic utterances and how they contribute to preserving the image of the speaker and to the negative representation of the other. The study reveals the way politicians mark their verbal irony, i.e., to classify and categorize the co-textual markers including hyperboles, and repetitive follow-up commentaries. The findings in this study support Bertuccelli (2018) view that irony is not merely saying the opposite of what one means but instead communicating an attitude. It adds to the existing literature that irony involves not a reversal of evaluation, but a reversal of attitudes about social actors to draw a positive self-image. Furthermore, the study shows that echoing the opponent's utterance involves a representation of acts and speech acts along with free direct and indirect speech.

Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations

This article presents a method of sequence-to-sequence (seq2seq) voice conversion using non-parallel training data. In this method, disentangled linguistic and speaker representations are extracted from acoustic features, and voice conversion is achieved by preserving the linguistic representations of source utterances while replacing the speaker representations with the target ones. Our model is built under the framework of encoder-decoder neural networks. A recognition encoder is designed to learn the disentangled linguistic representations with two strategies. First, phoneme transcriptions of training data are introduced to provide the references for leaning linguistic representations of audio signals. Second, an adversarial training strategy is employed to further wipe out speaker information from the linguistic representations. Meanwhile, speaker representations are extracted from audio signals by a speaker encoder. The model parameters are estimated by two-stage training, including a pre-training stage using a multi-speaker dataset and a fine-tuning stage using the dataset of a specific conversion pair. Since both the recognition encoder and the decoder for recovering acoustic features are seq2seq neural networks, there are no constrains of frame alignment and frame-by-frame conversion in our proposed method. Experimental results showed that our method obtained higher similarity and naturalness than the best non-parallel voice conversion method in Voice Conversion Challenge 2018. Besides, the performance of our proposed method was closed to the state-of-the-art parallel seq2seq voice conversion method.

A Study of Irony in Political Discourse

This study examines the ideological and evaluative nature of irony in discourse. It aims to reveal the linguistic constructions involved in ironic utterances and to uncover ideologies underpinning such structures. It draws upon the echo and pretense theories to account for utterance representation along with Wodak’s (2007) framework of self and other presentation strategies. The study examines the incongruous construction of an attitude, and the objectives attempted to achieve by taking a cynical stance. The main research questions center on the role played by ironic utterances and how they contribute to preserving the image of the speaker and to the negative representation of the other. The study reveals the way politicians mark their verbal irony, i.e., to classify and categorize the co-textual markers including hyperboles, and repetitive follow-up commentaries. The findings in this study support Bertuccelli (2018) view that irony is not merely saying the opposite of what one means but instead communicating an attitude. It adds to the existing literature that irony involves not a reversal of evaluation, but a reversal of attitudes about social actors to draw a positive self-image. Furthermore, the study shows that echoing the opponent's utterance involves a representation of acts and speech acts along with free direct and indirect speech.