Conversational Speech Synthesis Research Articles

Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling

Conversational Speech Synthesis (CSS) aims to accurately express an utterance with the appropriate prosody and emotional inflection within a conversational setting. While recognising the significance of CSS task, the prior studies have not thoroughly investigated the emotional expressiveness problems due to the scarcity of emotional conversational datasets and the difficulty of stateful emotion modeling. In this paper, we propose a novel emotional CSS model, termed ECSS, that includes two main components: 1) to enhance emotion understanding, we introduce a heterogeneous graph-based emotional context modeling mechanism, which takes the multi-source dialogue history as input to model the dialogue context and learn the emotion cues from the context; 2) to achieve emotion rendering, we employ a contrastive learning-based emotion renderer module to infer the accurate emotion style for the target utterance. To address the issue of data scarcity, we meticulously create emotional labels in terms of category and intensity, and annotate additional emotional information on the existing conversational dataset (DailyTalk). Both objective and subjective evaluations suggest that our model outperforms the baseline models in understanding and rendering emotions. These evaluations also underscore the importance of comprehensive emotional annotations. Code and audio samples can be found at: https://github.com/walker-hyf/ECSS.

End-to-end conversational speech synthesis with controllable emotions in the dimensions of pleasantness and arousal

End-to-end conversational speech synthesis with controllable emotions in the dimensions of pleasantness and arousal

Conversational speech synthesis and the need for some laughter

This paper reports progress in the synthesis of conversational speech, from the viewpoint of work carried out on the analysis of a very large corpus of expressive speech in normal everyday situations. With recent developments in concatenative techniques, speech synthesis has overcome the barrier of realistically portraying extra-linguistic information by using the actual voice of a recognizable person as a source for units, combined with minimal use of signal processing. However, the technology still faces the problem of expressing paralinguistic information, i.e., the variety in the types of speech and laughter that a person might use in everyday social interactions. Paralinguistic modification of an utterance portrays the speaker's affective states and shows his or her relationships with the speaker through variations in the manner of speaking, by means of prosody and voice quality. These inflections are carried on the propositional content of an utterance, and can perhaps be modeled by rule, but they are also expressed through nonverbal utterances, the complexity of which may be beyond the capabilities of many current synthesis methods. We suggest that this problem may be solved by the use of phrase-sized utterance units taken intact from a large corpus

Generation and perception of F0 markedness for communicative speech synthesis

Aiming at natural F 0 control for conversational speech synthesis using attributes of constituent output words, F 0 characteristics are analyzed from both generation and perception viewpoints. We recorded commonly used two-phrase utterances consisting of Japanese adjective and adverb phrases expressing different degree of markedness under designed conversational situations, and compared their F 0 characteristics. The comparison showed the consistent F 0 control dependencies not only on adverbs themselves but also on the attribute of following adjective phrases. Strong positive or negative correlation is observed between the markedness of adverbs and F 0 height when an adjective phrase showing positiveness or negativeness is followed to the current adverb phrase. These consistencies have been perceptually confirmed by naturalness evaluation tests using the same two-phrase samples with different F 0 heights. Finally, a computational model of conversational F 0 control is proposed using lexical information of adjectives showing positiveness or negativeness and adverbs expressing markedness. F 0 estimation experiments quantitatively showed the possibility of F 0 control for natural conversational speech synthesis using the attribute of constituent output words.

Conversational speech synthesis model for virtualized-egos

Conversational speech synthesis model for virtualized-egos