HMM-based Speech Synthesis System Research Articles

RETRACTED ARTICLE: Powerful basic frequency extraction from monophonic signs utilizing versatile sub-band separating

Basic frequency (f0) extraction assumes a significant function in the handling of monophonic signals, for example, discourse and tune. It is fundamental in different ongoing applications like feeling acknowledgment, discourse/performing voice segregation, etc. A few f0 extraction techniques have been proposed throughout the long term, yet nobody calculation functions admirably for both discourse and tune. Previously an efficient method for extracting pitch from speech signals for HMM-based speech synthesis system, voicing detection and pitch estimation is performed using the mean signal obtained from Continuous Wavelet Transform (CWT) coefficients. Both objective and subjective evaluation results show that the quality of speech synthesized with the proposed pitch estimation method is much better compared with HMM-based speech synthesis systems developed using the state-of-the-art pitch extraction methods, namely, Robust Algorithm for Pitch Tracking (RAPT) and Speech Transformation and Representation using Adaptive Interpolation of weighted spectrum (STRAIGHT) employed in HTS. Zero-frequency filter (ZFF) is used to derive the locations of impulse excitation. An algorithm is presented for the estimation of the fundamental frequency (F0) of speech or musical sounds, the well-known autocorrelation method with a number of modifications that combine to prevent errors. The algorithm has several desirable features. Error rates are about three times lower than the best competing methods, as evaluated over a database of speech recorded together with a laryngograph signal. In this paper, we propose a novel methodology that can precisely appraise f0 from discourse just as tunes. To begin with, voiced/unvoiced recognition is performed utilizing a novel RNN-LSTM based methodology. At that point, each voiced edge is disintegrated into a few sub-groups. From each sub-band of a voiced casing, the applicant pitch periods are distinguished utilizing autocorrelation and non-straight activities. At last, Viterbi interpreting is utilized to shape the last pitch forms. The presentation of the proposed technique is assessed utilizing mainstream discourse (Keele, CMU-Cold), and melody (MIR-1K, Verses) information bases. The assessment results show that the proposed strategy performs similarly well for discourse and monophonic melodies and is superior to the best in class techniques. Further, the viability of proposed f0 extraction strategy is exhibited by building up an intelligent SARGAM learning apparatus. The most generally utilized f0 extraction strategies are Praat and Riveted. The proposed method increases the performance nearly 81.8% and 35.2% compared to existing methods.

Language-independent acoustic cloning of HTS voices

Speaker adaptation techniques can be classified as intra-lingual or cross-lingual depending on whether or not the source model and the target speaker employ the same language. Most of the work in this field has been focused on the first case, while the second one has been less explored. In this paper we address the cross-lingual paradigm in the framework of a HMM-based speech synthesis system by further developing a formerly proposed approach. This method is able to clone a given speaker into a different language by combining the linguistic structure and the acoustic characteristics of two HTS models. In this work, we discuss the extension of the adaptation procedure to some other source model parameters that were kept unmodified in the initial version, and compare the performance of both versions by means of subjective and objective tests. These results are also contrasted with those obtained by a KLD-based technique proposed in the literature for a similar purpose. While no significant preference for any of the versions of our method is observed, our approach clearly outperforms the KLD-based technique.

Hidden-Markov-model based statistical parametric speech synthesis for Marathi with optimal number of hidden states

Hidden Markov Model and Deep Neural Networks based Statistical Parametric Speech Synthesis systems, gain a significant attention from researchers because of their flexibility in generating speech waveforms in diverse voice qualities as well as in styles. This paper describes HMM-based speech synthesis system (SPSS) for the Marathi language. In proposed synthesis method, speech parameter trajectories used for synthesis are generated from the trained hidden Markov models (HMM). We have recorded our database of 5300 phonetically balanced Marathi sentences to train the context-dependent HMM with five, seven and nine hidden states. The subjective quality measures (MOS and PWP) shows that the HMMs with seven hidden states are capable of giving an adequate quality of synthesized speech as compared to five state and with less time complexity than seven state HMMs. The contextual features used for experimentation are inclusive of a position of an observed phoneme in a respective syllable, word, and sentence.

Evaluation of speech unit modelling for HMM-based speech synthesis for Arabic

This paper investigates the use of hidden Markov models (HMM) for Modern Standard Arabic speech synthesis. HMM-based speech synthesis systems require a description of each speech unit with a set of contextual features that specifies phonetic, phonological and linguistic aspects. To apply this method to Arabic language, a study of its particularities was conducted to extract suitable contextual features. Two phenomena are highlighted: vowel quantity and gemination. This work focuses on how to model geminated consonants (resp. long vowels), either considering them as fully-fledged phonemes or as the same phonemes as their simple (resp. short) counterparts but with a different duration. Four modelling approaches have been proposed for this purpose. Results of subjective and objective evaluations show that there is no important difference between differentiating modelling units associated to geminated consonants (resp. long vowels) from modelling units associated to simple consonants (resp. short vowels) and merging them as long as gemination and vowel quantity information is included in the set of features.

Robust Pitch Extraction Method for the HMM-Based Speech Synthesis System

This letter proposes an efficient method for extracting pitch from speech signals for the hidden Markov model (HMM)-based speech synthesis system (HTS). In the proposed method, voicing detection and pitch estimation is performed using the mean signal obtained from continuous wavelet transform coefficients. The proposed pitch extraction method is integrated in the HMM-based speech synthesis system. The Performance of the proposed method is evaluated on CMU Arctic and Keele databases. Both objective and subjective evaluation results show that the quality of speech synthesized with the proposed pitch estimation method is much better compared with HMM-based speech synthesis systems developed using the state-of-the-art pitch extraction methods, namely, robust algorithm for pitch tracking and speech transformation and representation using adaptive interpolation of weighted spectrum employed in the HTS.

Parameterization of Excitation Signal for Improving the Quality of HMM-Based Speech Synthesis System

This paper proposes a new approach of parameterizing the excitation signal for improving the quality of HMM-based speech synthesis system. The proposed method tries to model the excitation or residual signal by segregating the regions of the residual signal based on their perceptual importance. Initially, a study on the characteristics of the residual signal around glottal closure instant (GCI) is performed using principal component analysis (PCA). Based on the present study, and from the previous literature (Adiga and Prasanna in Proceedings of Interspeech, pp 1677–1681, 2013; Cabral in Proceedings of Interspeech, pp 1082–1086, 2013), it is concluded that the segment of the residual signal around GCI which carries perceptually important information is considered as the deterministic component and the remaining part of the residual signal is considered as the noise component. The deterministic component is compactly represented using PCA coefficients (with about 95% accuracy), and the noise component is parameterized in terms of spectral and amplitude envelopes. The proposed excitation modeling approach is incorporated in the HMM-based speech synthesis system. Subjective evaluation results show a significant improvement of quality for both female and male speakers’ speech synthesized by the proposed method, compared to three existing excitation modeling methods. Accurate parameterization of the segment of the residual signal around GCI resulted in the improvement of the quality of the synthesized speech. Synthesized speech samples of the proposed and existing source models are made available online at http://www.sit.iitkgp.ernet.in/~ksrao/parametric-hts/pcd-hts.html.

Generation of creaky voice for improving the quality of HMM-based speech synthesis

This paper aims at developing an HMM-based speech synthesis system capable of generating creaky voice in addition to modal voice. Generation of creaky voice is carried out by addressing two main issues, namely, an automatic prediction of creaky voice and appropriate modelling of the excitation signal of creaky voice. An automatic creaky voice detection method is proposed based on the analysis of variation of epoch parameters for different voicing regions. A neural network classifier is trained using the variances of epoch parameters for detection of creaky regions. A hybrid source model which is an extension of recently developed time-domain deterministic plus noise model is proposed for modelling creaky excitation signal. In the proposed hybrid source model, the pitch-synchronous analysis is performed on the creaky excitation signal of every phone. From the creaky residual frames of every phonetic class, the deterministic and noise components are estimated. The creaky deterministic components of all phonetic classes are stored in the database. The noise components are parameterized in terms of spectral and amplitude envelopes and are modelled by HMMs. During synthesis, the appropriate deterministic component is selected from the database, and the noise component is constructed from the parameters generated from HMMs. The creaky deterministic and noise components are pitch-synchronously overlap-added to produce the creaky excitation signal. Subjective evaluation results indicate that the incorporation of creaky voice has improved the naturalness of the synthetic speech of two male speakers, and the quality is slightly better than the basic time-domain deterministic plus noise model meant for only modal excitation.

Corpus-Based Hidden Markov Modelling of the Fundamental Frequency of Lithuanian

This paper presents the corpus-driven approach in building the computational model of fundamental frequency, or F0, for Lithuanian language. The model was obtained by training the HMM-based speech synthesis system HTS on six hours of speech coming from multiple speakers. Several gender specific models, using different parameters and different contextual factors, were investigated. The models were evaluated by synthesizing F0 contours and by comparing them to the original F0 contours using criteria of root mean square error (RMSE) and voicing classification error. The HMM-based models showed an improvement of the RMSE over the mean-based model that predicted F0 of the vowel on the basis of its average normalized pitch.

Towards automatic cross-lingual acoustic modelling applied to HMM-based speech synthesis for under-resourced languages

Nowadays Human Computer Interaction (HCI) can also be achieved with voice user interfaces (VUIs). To enable devices to communicate with humans by speech in the user's own language, low-cost language portability is often discussed and analysed. One of the most time-consuming parts for the language-adaptation process of VUI- capable applications is the target-language speech-data acquisition. Such data is further used in the development of VUIs subsystems, especially of speech-recognition and speech-production systems. The tempting idea to bypass a long-term process of data acquisition is considering the design and development of an automatic algorithms, which can extract the similar target-language acoustic from different language speech databases. This paper focus on the cross-lingual phoneme mapping between an under-resourced and a well-resourced language. It proposes a novel automatic phoneme-mapping technique that is adopted from the speaker-verification field. Such a phoneme mapping is further used in the development of the HMM-based speech-synthesis system for the under-resourced language. The synthesised utterances are evaluated with a subjective evaluation and compared by the expert knowledge cross-language method against to the baseline speech synthesis based just from the under-resourced data. The results reveals, that combining data from well-resourced and under-resourced language with the use of the proposed phoneme-mapping technique, can improve the quality of under-resourced language speech synthesis.

Hierarchical stress modeling and generation in mandarin for expressive Text-to-Speech

Expressive speech synthesis has received increased attention in recent times. Stress (or pitch accent) is the perceptual prominence within words or utterances, which contributes to the expressivity of speech. This paper summarizes our contribution to Mandarin expressive speech synthesis. A novel hierarchical stress modeling and generation method for Mandarin is proposed and further integrated into HMM-based speech synthesis (HTS) and Fujisaki model-based speech synthesis systems to accurately model the undulation of pitch contour. In HMM-based expressive speech synthesis, stress-related contextual features obtained from the hierarchical model are introduced in modeling the prosodic variation caused by stress, in addition to the traditional prosodic features used in HTS. A rule-based and a Deep Belief Network based prosodic variation models are proposed and then used in stress adaptation module in HTS. The other approach uses the Fujisaki model to improve the expressiveness of synthetic speech. The hierarchical stress model is introduced into the phrase and tone command control mechanisms of the model. The pitch contour is then directly generated by the superposition of two-level commands of the Fujisaki model. Experimental results using the proposed hierarchical stress modeling and generation methods showed that the macro- and microcharacteristics of stress could be successfully captured. The methodology proposed in this paper has application to a range of areas such as conveying attitude and indicating focus in spoken dialog systems.

Robust Voicing Detection and $$F_{0}$$ F 0 Estimation for HMM-Based Speech Synthesis

This paper proposes a robust voicing detection and $$F_{0}$$F0 estimation method for Hidden Markov model (HMM)-based speech synthesis system. Impulse-like excitation present in voiced speech is utilized for extracting the fundamental frequency. Zero-frequency filter (ZFF) is used to derive the locations of impulse excitation. The main contribution of this paper is exploitation of size of window used in ZFF for accurate voicing detection and $$F_{0}$$F0 estimation. By adaptively choosing appropriate window size, the strength of excitation for voiced speech is significantly higher compared with unvoiced speech. With suitable threshold on the strength of excitation, accurate voicing detection is performed. In this method, smooth and accurate $$F_{0}$$F0 contour is extracted by frame-wise zero-frequency filtering of speech with appropriate window size. Performance of the proposed method is compared with other existing voicing detection and $$F_{0}$$F0 estimation methods. The proposed voicing detection and $$F_{0}$$F0 estimation method is implemented in HMM-based speech synthesis system. Both objective and subjective evaluation results show that the proposed method is capable of generating good quality speech compared with HMM-based speech synthesis systems developed using voicing detection and $$F_{0}$$F0 estimation methods based on Robust algorithm for pitch tracking and Speech Transformation and Representation using Adaptive Interpolation of weiGHTed spectrum.

$$\hbox {F}_{0}$$ F 0 contour generation and synthesis using Bengali Hmm-based speech synthesis system

HMM based Bengali speech synthesis system (Bengali-HTS) generates highly intelligible synthesized speech but its naturalness is not adequate even though it is trained with a very good amount of speech corpus. In case of interrogative, imperative and exclamatory sentences, naturalness of the synthesized speech falls drastically. This paper proposes a method to overcome this problem by modifying the $$\hbox {F}_{0}$$F0 contour of synthetic speech based on Fujisaki model. The Fujisaki model features for different types of Bengali sentences are analyzed for the generation of $$\hbox {F}_{0}$$F0 contour. These features depend on prosodic word/phrase boundary of the sentence. So a two layer supervised classification and regression tree is trained to predict the prosodic word/phrase boundary. Fujisaki model then generates $$\hbox {F}_{0}$$F0 contour from input text using the prosodic word/phrase boundary and segmental duration information from HMM-based speech synthesis system. Moreover, for HMM training purpose, prosodic structure of sentence has been employed rather than lexical structure. From MOS and preference test it is found that proposed method significantly improved the overall quality of synthesized speech than that of Bengali-HTS.

Developing an HMM-Based Speech Synthesis System for Malay: A Comparison of Iterative and Isolated Unit Training

Developing an HMM-Based Speech Synthesis System for Malay: A Comparison of Iterative and Isolated Unit Training

Modeling Spectral Envelopes Using Restricted Boltzmann Machines and Deep Belief Networks for Statistical Parametric Speech Synthesis

This paper presents a new spectral modeling method for statistical parametric speech synthesis. In the conventional methods, high-level spectral parameters, such as mel-cepstra or line spectral pairs, are adopted as the features for hidden Markov model (HMM)-based parametric speech synthesis. Our proposed method described in this paper improves the conventional method in two ways. First, distributions of low-level, un-transformed spectral envelopes (extracted by the STRAIGHT vocoder) are used as the parameters for synthesis. Second, instead of using single Gaussian distribution, we adopt the graphical models with multiple hidden variables, including restricted Boltzmann machines (RBM) and deep belief networks (DBN), to represent the distribution of the low-level spectral envelopes at each HMM state. At the synthesis time, the spectral envelopes are predicted from the RBM-HMMs or the DBN-HMMs of the input sentence following the maximum output probability parameter generation criterion with the constraints of the dynamic features. A Gaussian approximation is applied to the marginal distribution of the visible stochastic variables in the RBM or DBN at each HMM state in order to achieve a closed-form solution to the parameter generation problem. Our experimental results show that both RBM-HMM and DBN-HMM are able to generate spectral envelope parameter sequences better than the conventional Gaussian-HMM with superior generalization capabilities and that DBN-HMM and RBM-HMM perform similarly due possibly to the use of Gaussian approximation. As a result, our proposed method can significantly alleviate the over-smoothing effect and improve the naturalness of the conventional HMM-based speech synthesis system using mel-cepstra.

HMM 기반의 TTS를 위한 상호유사도 비율을 이용한 결정트리 기반의 문맥 군집화

본 논문은 HMM 기반의 TTS 시스템을 위하여 상호유사도 비율을 이용한 결정트리 기반의 문맥 군집화 알고리즘을 제안한다. 기존의 알고리즘들은 유사한 통계적 특성을 가지는 문맥종속 HMM을 하나로 묶고 있다. 그러나 기존의 알고리즘들은 결정트리의 나누어진 노드간의 통계적 유사도를 고려하지 않음으로 인하여 최종 노드 사이의 통계적인 차이를 보장하지 못한다. 제안한 알고리즘은 분리된 노드들 간의 통계적 유사도를 최소화하여 모델 파라미터의 신뢰도를 향상시킨다. 실험 결과를 통해 제안한 알고리즘이 기존의 알고리즘들에 비해 우수한 성능을 나타낸다는 것을 확인할 수 있다. This paper proposes a decision tree based context clustering algorithm for HMM-based speech synthesis systems using the cross likelihood ratio with a hierarchical prior (CLRHP). Conventional algorithms tie the context-dependent HMM states that have similar statistical characteristics, but they do not consider the statistical similarity of split child nodes, which does not guarantee the statistical difference between the final leaf nodes. The proposed CLRHP algorithm improves the reliability of model parameters by taking a criterion of minimizing the statistical similarity of split child nodes. Experimental results verify the superiority of the proposed approach to conventional ones.

Design of Tree Structure in Context Clustering Process of Hidden Markov Model-Based Thai Speech Synthesis

Problem statement: In HMM-based Thai speech synthesis, the tone degradation due to the imbalance of training data of all tones. Some distortion of syllable duration is obviously noticeable when the system is trained with a small amount of data. These problems cause the decrement in naturalness and intelligibility of the synthesized speech. Approach: This study proposes an approach to improve the correctness of tone of the synthesized speech which is generated by an HMM-based Thai speech synthesis system. In the tree-based context clustering process, tone groups and tone types are used to design four different structures of decision tree including a single binary tree structure, a simple tone-separated tree structure, a constancy-based-tone-separated tree structure and a trend-based-tone-separated tree structure. Results: A subjective evaluation of tone correctness is conducted by using tone perception of eight Thai listeners. The simple tone-separated tree structure gives the highest level of tone correctness, while the single binary tree structure gives the lowest level of tone correctness. The additional contextual tone information which is applied to all structures of the decision tree achieves a significant improvement of tone correctness. Finally, the evaluation of syllable duration distortion among the four structures shows that the constancy-based-tone-separated and the trend-based-tone-separated tree structures can alleviate the distortions that appear when using the simple tone-separated tree structure. Conclusion: The appropriate structure of tree in context clustering process with the additional contextual tone information can improve the correctness of tones, while the constancy-based-tone-separated and the trend-based-tone-separated tree structures can alleviate the syllable duration distortions.

Thai Speech Phonology for Development of Speech Synthesis: A Review

Problem statement: To implementation of the Hidden Markov Model (HMM)-based Thai speech synthesis system, it is necessary to understand the phonology system for a language. Without the phonological information, the contextual factors of tree-based context clustering cannot be completed. Approach: The existing speech units in Thai are studied thoroughly so that the synthesis system can provide all of them in an appropriate way. In the study of speech in a specific language, we have to categorize the speech into sounds. Then summarize them in some specific ways including its function, how it appears in speech, the relation with other sounds and also how it contributes to syllable or word. Results: The speech units of the phoneme, syllable, word, phrase and sentence level are studied and explained, respectively. Conclusion: The important information of Thai phonology system has been summarized. It is expected to apply them efficiently in the HMM-based Thai speech synthesis system.

A Control of Fundamental Frequency Contour for Hidden Markov Model-Based Thai Speech Synthesis

Problem statement: In the conventional HMM-based speech synthesis system for Thai, there is no control of fundamental frequency control in the synthesis stage. The tone correctness of the synthesized speech is unacceptable due to the imbalance of training data of all tones. Approach: This study proposes a mathematical model to control the F0 contour of the synthesized speech. This control is proposed to correct only some distorted segments of the F0 contour which occur within some syllables due to lacking of training data for some tones. Results: An experimental result compares F0 contours between those of synthesized speech with and without tone-type questions; furthermore the size of Thai speech corpus is varied to investigate the synthesized speech quality. A mathematical model is applied to control the F0 contour. By using the proposed control, the correction of the F0 contour is obviously shown in the experimental results. Conclusion: The control of F0 contour has been proposed. It can noticeably improve the tone correctness of the synthesized speech.

Outlier Detection and Removal for HMM-Based Speech Synthesis with an Insufficient Speech Database

Decision tree-based clustering and parameter estimation are essential steps in the training part of an HMM-based speech synthesis system. These two steps are usually performed based on the maximum likelihood (ML) criterion. However, one of the drawbacks of the ML criterion is that it is sensitive to outliers which usually result in quality degradation of the synthesized speech. In this letter, we propose an approach to detect and remove outliers for HMM-based speech synthesis. Experimental results show that the proposed approach can improve the synthetic speech, particularly when the available training speech database is insufficient.

Analysis of Decision Trees in Context Clustering of Hidden Markov Model Based Thai Speech Synthesis

Problem statement: In Thai speech synthesis using Hidden Markov model (HMM) based synthesis system, the tonal speech quality is degraded due to tone distortion. This major problem must be treated appropriately to preserve the tone characteristics of each syllable unit. Since tone brings about the intelligibility of the synthesized speech. It is needed to establish the tone questions and other phonetic questions in tree-based context clustering process accordingly. Approach: This study describes the analysis of questions in tree-based context clustering process of an HMM-based speech synthesis system for Thai language. In the system, spectrum, pitch or F0 and state duration are modeled simultaneously in a unified framework of HMM, their parameter distributions are clustered independently by using a decision-tree based context clustering technique. The contextual factors which affect spectrum, pitch and duration, i.e., part of speech, position and number of phones in a syllable, position and number of syllables in a word, position and number of words in a sentence, phone type and tone type, are taken into account for constructing the questions of the decision tree. All in all, thirteen sets of questions are analyzed in comparison. Results: In the experiment, we analyzed the decision trees by counting the number of questions in each node coming from those thirteen sets and by calculating the dominance score given to each question as the reciprocal of the distance from the root node to the question node. The highest number and dominance score are of the set of phonetic type, while the second, third highest ones are of the set of part of speech and tone type. Conclusion: By counting the number of questions in each node and calculating the dominance score, we can set the priority of each question set. All in all, the analysis results bring about further development of Thai speech synthesis with efficient context clustering process in an HMM-based speech synthesis system.