Concatenative Units Research Articles

Emotional Intelligence in Text-To-Speech Synthesis in Pali Language Using Fuzzy Logic

The field of emotional text-to-speech (TTS) synthesis is making swift progress within the realm of artificial intelligence, holding immense promise to transform our interaction with technology. By using advanced algorithms to analyze and understand the emotional content of text, these systems are able to produce spoken language that accurately conveys the intended emotional tone of the message. Despite the existence of several Text-To-Speech systems across various languages, Pali language is yet to have its own. As a result, we have taken the initiative to create a Text-To-Speech synthesizer exclusively for Pali. Our system offers an end-to-end solution for emotional speech synthesis via Text-To-Speech. We address the problem by incorporating disentangled, well-grained prosody features with global, sentence-level emotion implanting. These well-grained features learn to denote local prosodic differences disentangled from the speaker, tone, and worldwide emotion label. Prosody is usually modeled by rules, so we have implemented the fuzzy logic system to develop a controller for the prosody of Pali speech. The fuzzy controller handles different linguistic parameters in three types of sentences: interrogative, exclamatory, and declarative. The final system produces comprehensible speech that mimics the appropriate intonation for every type of sentence. In this paper, we introduce and outline the application of a fuzzy paradigm to incorporate a Text-To-Speech system for the Pali language while preserving a rule-based Concatenative synthesizer. In the outline of classic Concatenative TTS systems, we recommend a new method in order to increase Concatenative unit selection computation, directed at increasing synthetic speech perceptual superiority. In order to tackle the problem of phonemes that are prone to multiple descriptions in rule-based speech synthesis, the proposed solution involves a fuzzy system. In the introductory section, we offer a concise description of the current context surrounding the challenge of emotional speech synthesis. The second section of this paper outlines the notable advancements made in emotional speech synthesis, acknowledging the contributions of various researchers in this field. The third section delves into the technical details of implementing a fuzzy system The last section of the paper presents the main conclusions and future research scope.

Video-realistic expressive audio-visual speech synthesis for the Greek language

High quality expressive speech synthesis has been a long-standing goal towards natural human-computer interaction. Generating a talking head which is both realistic and expressive appears to be a considerable challenge, due to both the high complexity in the acoustic and visual streams and the large non-discrete number of emotional states we would like the talking head to be able to express. In order to cover all the desired emotions, a significant amount of data is required, which poses an additional time-consuming data collection challenge. In this paper we attempt to address the aforementioned problems in an audio-visual context. Towards this goal, we propose two deep neural network (DNN) architectures for Video-realistic Expressive Audio-Visual Text-To-Speech synthesis (EAVTTS) and evaluate them by comparing them directly both to traditional hidden Markov model (HMM) based EAVTTS, as well as a concatenative unit selection EAVTTS approach, both on the realism and the expressiveness of the generated talking head. Next, we investigate adaptation and interpolation techniques to address the problem of covering the large emotional space. We use HMM interpolation in order to generate different levels of intensity for an emotion, as well as investigate whether it is possible to generate speech with intermediate speaking styles between two emotions. In addition, we employ HMM adaptation to adapt an HMM-based system to another emotion using only a limited amount of adaptation data from the target emotion. We performed an extensive experimental evaluation on a medium sized audio-visual corpus covering three emotions, namely anger, sadness and happiness, as well as neutral reading style. Our results show that DNN-based models outperform HMMs and unit selection on both the realism and expressiveness of the generated talking heads, while in terms of adaptation we can successfully adapt an audio-visual HMM set trained on a neutral speaking style database to a target emotion. Finally, we show that HMM interpolation can indeed generate different levels of intensity for EAVTTS by interpolating an emotion with the neutral reading style, as well as in some cases, generate audio-visual speech with intermediate expressions between two emotions.

Screen Readers for Linux and Windows – Concatenation Methods and Unit Selection based Marathi Text to Speech System

The research paper briefs about the implementation of screen readers for Marathi in Windows and Linux platform using unrestricted domain Marathi Text To Speech with Indian English support. The application is an integration of MTTS with open source Screen readers NVDA and ORCA. MTTS is a syllable based unit selection concatenative system, built around open source festival engine. IE support is provided for the smooth navigation and handling the English words occurring while accessing internet and other applications. The TTS is a concatenative based system in which syllable is the highest unit for concatenation. The TTS output resembles natural human voice since it uses the original speech segments for concatenation. Testing has been done with normal and differently abled users. Tuning of the system for improving the user friendliness has been done based on the feedback from the DA The system gets a Mean Opinion Score of 86.4% when evaluated by a group of DA.

Syllable based text to speech synthesis system using auto associative neural network prosody prediction

This paper presents the design and development of an Auto Associative Neural Network (AANN) based unrestricted prosodic information synthesizer. Unrestricted Text To Speech System (TTS) is capable of synthesize different domain speech with improved quality. This paper deals with a corpus-driven text-to speech system based on the concatenative synthesis approach. Concatenative speech synthesis involves the concatenation of the basic units to synthesize an intelligent, natural sounding speech. A corpus-based method (unit selection) uses a large inventory to select the units and concatenate. The prosody prediction is done with the help of five layer auto associative neural network which helps us to improve the quality of speech synthesis. Here syllables are used as basic unit of speech synthesis database. The database consisting of the units along with their annotated information is called annotated speech corpus. A clustering technique is used in annotated speech corpus that provides way to select the appropriate unit for concatenation, based on the lowest total join cost of the speech unit. Discontinuities present at the unit boundaries are lowered by using the mel-LPC smoothing technique. The experiment has been made for the Dravidian language Tamil and the results reveal to demonstrate the improved intelligibility and naturalness of the proposed method. The proposed system is applicable to all the languages if the syllabification rules has been changed.

Durational Evidence for Syllable Boundary of /n/ and /l/ in Text-to-Speech Synthesis

The Text-to-Speech (TTS) system does rely on syllable boundary information for segmental duration. However, ambisyllabic consonants always pose a problem to TTS because the system requires clear syllable boundaries to segment and concatenate. In order to provide a possible solution to this problem, /n/ and /l/ in V L C A V R are chosen in this paper as the target to be examined whether their durations behave more like the syllabic onset or coda when comparing with the durational properties of /n/ and /l/ both as onsets in CV and codas in VC . As the syllable boundaries, onset C shows much more sensitivity to stress than coda C while coda C shows more sensitivity to syllabic position than onset C . Moreover, C A in V L C A V R is also influenced by two variables of stress and position as C in CV and VC . The results show that the intervocalic C A holds the properties of both the syllabic onset and coda, which states the possibility that intervocalic consonants should be considered as a rather independent concatenative unit in TTS synthesis.

Design and evaluation of prosodically-sensitive concatenative units for a Korean TTS system

This paper describes the design and evaluation of prosodically-sensitive concatenative units for a Korean text-to-speech (TTS) synthesis system. The diphones used are prosodically conditioned in the sense that a single conventional diphone is stored as different versions taken directly from the different prosodic domains of the prosodically labeled, read sentences. The four levels of the Korean prosodic hierarchy were observed in the diphone selection process, thereby selecting four different versions of each diphone: three edge diphones from the prosodic domains of the intonational phrase (IP), accentual phrase (AP) and prosodic word (PW), and a non-edge diphone from the domain of the prosodic word. Due to the size of the corpus that we employed, our system covers only 36.4% of the 6503 possible diphones. A listening experiment designed to evaluate the quality of the diphone database showed that listeners preferred stimuli composed of prosodically appropriate diphones. We interpret this as supporting the view that segments carry prosodic domain information.

The Contribution of Various Sources of Spectral Mismatch to Audible Discontinuities in a Diphone Database

One of the major problems in concatenative synthesis is the occurrence of audible discontinuities between two successive concatenative units. Several studies have attempted to discover objective distance measures that predict the audibility of these discontinuities. In this paper, we investigate mid-vowel joins for three vowels with a range of post-vocalic consonant contexts typical for diphone databases. A first perceptual experiment uses a pairwise comparison procedure to find two subsets of unit combinations: Those with versus without audible discontinuities. A second perceptual experiment uses these two subsets in a procedure where formant resynthesis is used to manipulate three sources of discontinuity separately: formant frequencies, formant bandwidths, and overall energy. Results show mismatch in formant frequencies provides the largest contribution to audible discontinuity, followed by mismatch in overall energy

Induction in the Timed Interval Calculus

The Timed Interval Calculus, a timed-trace formalism based on set theory, is introduced. It is extended with an induction law and a unit for concatenation, which facilitates the proof of properties over trace histories. The effectiveness of the extended Timed Interval Calculus is demonstrated via a benchmark case study, the mine pump. Specifically, a safety property relating to the operation of a mine shaft is proved, based on an implementation of the mine pump and assumptions about the environment of the mine.

Domain-specific prominence-based concatenation

Some recent approaches to unlimited text-to-speech conversion do not use ‘‘designed’’ inventories of concatenative units like diphones. Instead, an annotated corpus of read speech is searched for the realization of a speech segment whose features best match the corresponding ones demanded by the synthesis input. Crucial for this approach are the size and the variety of the corpus, the number and kind of annotations, and the definition of ‘‘best match’’ based on these annotations. In this implementation, the parameter ‘‘perceived prominence’’ is relied on as the most important parameter for the selection of a prosodically appropriate realization. A previous investigation with a German corpus indicated, however, that a corpus has to be very large in order to allow prosodic variations (e.g., different focus placements) of one utterance while retaining an acceptable intelligibility, if no post-selection signal modification is applied. The application of the same procedure to an American English corpus with domain-specific utterances (travel planning) indicates that the restriction to specific domains yields better results, but that signal manipulation is inevitable in order to obtain intelligible speech.

Determining concatenative units for speech synthesis

The purpose of this research is to determine the best method for deciding on concatenative units (diphones and other polyphones) for speech synthesis. Four different databases are used and a greedy algorithm [J. Van Santen (1991)] is applied in order to extract n-gram phonemic frequencies; this serves as a basis for comparison between dictionary-derived frequencies and corpus-derived frequencies. Two dictionaries, a large one (the Robert Encyclopedic French dictionary with 85 000 headwords) and a small one (the Collins Gem containing 15 000 words) and two phonetically transcribed corpora, a large one (the Hansard, of about 2.5 million words) and a smaller one (the Tubach and Boe of 8 000 words) were used. The hypothesis was that a dictionary could provide essential concatenative unit data without recourse to corpora. Nevertheless, results showed that for interword phenomena, the hypothesis was correct, whereas for intraword effects, such as liaison and assimilation, the dictionary field information falls short. This research presents two interesting aspects: on the practical side, a speech synthesis system for French [E. Tzoukermann (1993)] is being built at AT&T Bell Laboratories and the output of this research will determine the most frequent concatenative units for putting into the working system. On the theoretical side, the goal is to explore the question of what constitutes adequate data for inducing concatenative units.

Issues in French text-to-speech synthesis

In this paper, the new French text-to-speech system being developed at AT&T is presented. The system, based on diphone and triphone concatenation, follows the general framework of the Bell Laboratories English TTS system [J. Olive (1977); J. Olive and M. Liberman (1979)]. Concatenative units have been recorded within real words from dictionary extraction and from logatomes, or nonexisting words. Researchers disagree as to whether to use logatomes or real words for synthesis. The argument for using logatomes is that it is better to collect neutral words so that the diphone is recorded as neutrally as possible and does not undergo any real word stress. Those against argue that the diphone is over-articulated in a logatome environment and that it reduces the naturalness of the synthesized speech. Experiments are shown that prove the strengths and weaknesses of the two approaches. Methods for automatic segmentation and for the choice of concatenative units are presented, as well as durational and prosodic issues. Directions for future improvement are described, including morphological analysis, part-of-speech tagging, and partial syntactic analysis. Some problems in French text-to-speech, such as nasalization, liaison, realization of schwas, lengthening of silences, prosodic groupings, as well as stable versus unstable phonemes are discussed.

Perceptual experiments for diagnostic testing of text-to-speech systems

This paper describes perceptual methods for diagnosing problems in text-to-speech systems. Special attention is paid to two issues. First, coverage of the domain of a text-to-speech system. Since this domain involves an enormous range of contexts, it is criticial for diagnostics, and also for overall evaluation, that test materials cover this range to the fullest extent possible. Automatic text generation algorithms that make extensive use of "greedy" algorithms are described that serve this purpose. Second, speech generated by text-to-speech systems tends to have a great variety of problems . A battery of experimental paradigms is discussed that address different facets of speech quality and intelligibility. Included are: (a) "word pointing" method for detection of problematic concatenative units, (b) "minimal pairs intelligibility test"—an expanded diagnostic rhyme test; (c) automatically scored orthographic name transcription task; (d) mean opinion score paradigm with problem categorization; and (e) paired comparison paradigm with strength-of-choice rating. The methods are applied in a series of experiments on high-end text-to-speech systems.

Phonology and phonetics. A syllable-based model of articulatory organization.

A new model for describing articulatory organization is proposed, demonstrating the possibility to basically deviate from the traditional segment-concatenation coarticula tiontheory. This Converter/Distributor (C/D) Model is based on syllables as the concatenative units, and combines feature-based specifications of demisyllables with a novel concept of syllable and boundary strength and timing represented by abstract impulses. Consonantal gestures, superimposed upon a vocalic base function, are implementedas a combination of stored impulse response functions for individual elementalgestures, in respective articulatory dimensions associating actions with articulators.The C/D model is intended for description and explanation of natural speech, and emphasis is placed on its ability to handle articulatory variability due to a wide variety of extraphonological as well as prosodic effects according to simple principles.

Some observations on the processing of mute “e” in a French diphone-based speech synthesis system

Based on qualitative experimental observations and the results of intelligibility tests on synthetic speech utterances, this study tries to explain why some of the mute “e” pronunciation rules observed in standard French cannot be replicated with the current version of the CNET diphone-based TTS system without degrading intelligibility. It highlights the necessity of improving the type of speech unit to be stored, in place of diphones, in concatenation-based systems. It also shows that this choice of concatenative unit cannot be adequately made without gaining more knowledge about some scarcely investigated phonetic issues, such as the acoustic correlates of pronounced vs. elided mute “e”.

Development of a female synthetic voice using concatenative synthesis

Although there is considerable demand for synthetic female voices in text-to-speech applications, current analysis and synthesis algorithms are more successful for male than for female speech. Difficulties include voice source characteristics and other acoustic differences between female and male speech. Telephone bandwidth constraints present an additional disadvantage for the female voice, because relatively more phonetically relevant high-frequency acoustic information is filtered out of the telephone signal for female speakers than for males. A synthetic female voice is currently being developed for use with an experimental AT&T concatenative text-to-speech system for which only male voices have been available previously [J.P. Olive, Proc. ICASSP, 568–570 (1977); J.P. Olive and M. Y. Liberman, J. Acoust. Soc. Am. Suppl. 1 78, S6 (1985)]. In this synthesis-by-rule system, segments used for synthesis are obtained from natural speech and concatenated to synthesize any arbitrary English utterance. Issues such as speaker selection, preparation and recording of speech materials, inventor? of concatenative units, and analysis and synthesis methods will be discussed. Diagnostic use of intelligibility test results and preliminary evaluations also will be described.

A neuronal model for syllable representation

A speculative neuronal template, equivalent to canonical syllable forms and independent of segmental representations, is offered to help account for (1) the inviolate nature of phonotactic constraints in aphasic speech output, and (2) left hemisphere specialization for speech sound access and output. The model, which attempts to relate plausible neuronal systems to linguistic function, is based on cell assemblies that are thought to develop by way of genetic predisposition and ontogenetic language experience, into configurations that can represent canonical slot positions for the consonants and vowel comprising a syllable. The syllable is assumed to be the basic organizational rhythmic unit for serial concatenation of sublexical segments. A scheme for neurological differentiation of vowels and consonants is offered. Phonotactic constraints can become “hard-wired” to help create the automaticity underlying phonological sound organization. Testable predictions are offered to substantiate the claims of the model.

A set of concatenative units for speech synthesis

In a previous paper [J. P. Olive, “Rule Synthesis of Speech From Dyadic Units”, IEEE International Conference on Acoustics, Speech and Signal Processing, 568–570, (1977)], we discussed a speech synthesis by rule scheme where the segments used for synthesis were obtained from natural speech. These segments included the consonants and the transitions from consonants to vowels, vowels to vowels, and vowels to consonants. Each synthesis parameter was defined by two sets of LPC area parameters, and in the concatenative process, straight line interpolation was used to obtain the complete set of area parameters. Informal listening and some formal intelligibility testing revealed that this simplified description of the synthesis segments was not sufficient to produce the kind of speech that would satisfy us. Consequently, it was decided to expand the definition of the set of concatenative units. Specifically: (1) the number of phonetic segments defining a synthesis unit can vary from one to three; (2) arbitrary number of area parameter vectors can be stored for a given synthesis unit; and (3) a facility is provided for expressing transition durations and amplitude rules for synthesis units or classes of units. This paper will discuss the details of a more complete and accurate set of concatenative units.