Concatenation Cost Research Articles

An LSTM-based model for the compression of acoustic inventories for corpus-based text-to-speech synthesis systems

Large acoustic inventories must be used to produce speech close to natural quality. However, the concatenation cost space grows exponentially with the number of acoustic units in the acoustic inventory, increasing the latency of the unit selection algorithm, making algorithms unusable in real-time end-to-end systems. Even when data compression techniques are introduced, the model size is still high, representing a challenge for end-to-end systems. Thus, in this paper, we propose representing the concatenation cost space using LSTM (Long Short-Term Memory). The results show a 90% reduction in the size of the data space compared to all our previous techniques, and by an over 70% decrease in the look-up time. The proposed LSTM-based compression increases the responsiveness of the corpus-based text-to-speech systems significantly while keeping the overall speech quality at the same level.

Stereo Matching With Multiscale Hybrid Cost Volume

Stereo matching estimates the disparity between a pair of rectified left and right images, which plays an important role in robot navigation, autonomous driving, and other related tasks. Despite the remarkable progress of deep learning in stereo matching, there are still many challenges. One of the challenges is that current stereo models mostly generate a single scale cost volume based on costly 3D convolutions method or quick 2D convolutions method, but neither of these two methods can achieve a fair trade-off between quality and time. In this paper, we propose to construct multi-scale hybrid cost volume which aims at achieving fast speed while maintaining comparable accuracy. Concretely, we generate the correlation cost volume and the concatenation cost volume respectively, and then integrate together to form a hybrid cost volume which can significantly improve the accuracy and reduce the computational complexity. At the multi-scale level, we generate three hybrid cost volumes at different scales and then aggregate them by 2D convolutions which are faster than 3D convolutions. In addition, we adopt 2D CNN stacked hourglass with fused cost volume for cost aggregation. Specifically, the proposed method provides competitive performance with state-of-the-art methods, while being faster than most top-performing methods (e.g., 3.7× than PSMNet, 8.1× than GCNet, 16.3× than GANet). Extensive experiments of our model on the SceneFlow, KITTI 2012, and KITTI 2015 datasets demonstrate the competitiveness of our results.

Speech Synthesis System for the Holy Quran Recitation

This paper aims to develop a Text-To-Speech (TTS) synthesis system for the holly quran recitation, to properly helps reciters and facilitates its use. In this work, the unit selection method is adopted and improved to reach a good speech quality. The proposed approach consists mainly of two steps. In the first one, an Expert System (ES) module is integrated by employing Arabic, Quran language, phonetic and phonological features. This part was considered as a preselection to optimize the synthesis algorithm's speed. The second step is the final selection of units by minimizing a concatenation cost function and a forward-backward dynamic programming search. The system is evaluated by native and non-native Arabic speakers. The results show that the goal of a correct Quran recitation by respecting its reading rules was reached, with 97 % of speech intelligibility and 72.13% of naturalness

Learning and Modeling Unit Embeddings Using Deep Neural Networks for Unit-Selection-Based Mandarin Speech Synthesis

A method of learning and modeling unit embeddings using deep neutral networks (DNNs) is presented in this article for unit-selection-based Mandarin speech synthesis. Here, a unit embedding is defined as a fixed-length embedding vector for a phone-sized unit candidate in a corpus. Modeling phone-sized embedding vectors instead of frame-sized acoustic features can better measure the long-term dependencies among consecutive units in an utterance. First, a DNN with an embedding layer is built to learn the embedding vectors of all unit candidates in the corpus from scratch. In order to enable the extracted embedding vectors to carry both acoustic and linguistic information of unit candidates, a multitarget learning strategy is designed for the DNN. Its optional prediction targets include frame-level acoustic features, unit durations, monophone and tone identifiers, and context classes. Then, another two DNNs are constructed to map linguistic features toward the extracted embedding vectors. One of them employs the unit vectors of preceding phones besides the linguistic features of current phone as its input. At synthesis time, the distances between the unit vectors predicted by these two DNNs and the ones derived from unit candidates are used as a part of the target cost and a part of the concatenation cost, respectively. Our experiments on a Mandarin speech synthesis corpus demonstrate that learning and modeling unit embeddings improve the naturalness of hidden Markov model (HMM)-based unit selection speech synthesis. Furthermore, integrating multiple targets for learning unit embeddings achieves better performance than using only acoustic targets according to our subjective evaluation results.

A New Unit Selection Optimisation Algorithm for Corpus-Based TTS Systems Using the RBF-Based Data Compression Technique

A major drawback of corpus-based speech synthesis systems is the use of large acoustic inventories, and currently one of the main challenges is the optimal representation of concatenation costs associated with units in the acoustic inventory. These concatenation costs are used to evaluate spectral mismatches between the acoustic units to be concatenated. The combinatorics of costs grows exponentially with the size of the acoustic inventories and can result in hundreds of millions or even billions of concatenation costs to be processed. Therefore, in this paper, we represent a novel unit selection optimization algorithm, which minimizes the size of concatenation costs through the vector quantization-based compression technique and tuple structures. Furthermore, the proposed optimization algorithm is designed to be used as an objective measure to optimize the performance of the unit selection cost function regarding the quality of the speech output, and to evaluate the effect of the vector quantization-based compression technique on its performance. The results obtained show that even when data compression is above 50%, the effect on the quality of the synthesized speech is negligible.

Unit Selection Speech Synthesis Using Frame-Sized Speech Segments and Neural Network Based Acoustic Models

This paper proposes to select frame-sized speech segments for waveform concatenation speech synthesis using neural network based acoustic models. First, a deep neural network (DNN) based frame selection method is presented. In this method, three DNNs are adopted to calculate target costs and concatenation costs respectively for selecting candidate frames of 5ms length. One DNN is built in the same way as the DNN-based statistical parametric speech synthesis, which predicts target acoustic features given linguistic context inputs. The distance between the acoustic features of a candidate unit and the predicted ones for a target unit is calculated as the target cost. The other two DNNs are constructed to predict the acoustic features at current frame using its context features and the acoustic features of preceding frames. At synthesis time, these two DNNs are employed to calculate the concatenation cost for each candidate frame given its preceding ones. Furthermore, recurrent neural networks (RNNs) with long short-term memory (LSTM) cells are adopted to replace DNNs for acoustic modeling in order to make better use of the sequential information. A strategy of using multi-frame instead of single frame as the basic unit for selection is also presented to reduce the concatenation points within synthetic speech. Experimental results show that our proposed method can achieve better naturalness than the hidden Markov model (HMM)-based frame selection method and the HMM-based parametric speech synthesis method.

Building Text Corpus for Unit Selection Synthesis

The present paper deals with building the text corpus for unit selection text-to-speech synthesis. During synthesis the target and concatenation costs are calculated and these costs are usually based on the prosodic and acoustic features of sounds. If the cost calculation is moved to the phonological level, it is possible to simulate unit selection synthesis without any real recordings; in this case text transcriptions are sufficient. We propose to use the cost calculated during the test data synthesis simulation to evaluate the text corpus quality. The greedy algorithm that maximizes coverage of certain phonetic units will be used to build the corpus. In this work the corpora optimized to cover phonetic units of different size and weight are evaluated.

Syllable Specific Unit Selection Cost Functions for Text-to-Speech Synthesis

This paper presents the design and development of syllable specific unit selection cost functions for improving the quality of text-to-speech synthesis. Appropriate unit selection cost functions, namely concatenation cost and target cost, are proposed for syllable based synthesis. Concatenation costs are defined based on the type of segments present at the syllable joins. Proposed concatenation costs have shown significant reduction in perceptual discontinuity at syllable joins. Three-stage target cost formulation is proposed for selecting appropriate units from database. Subjective evaluation has shown improvement in the quality of speech at each stage.

Optimal weight tuning method for unit selection cost functions in syllable based text-to-speech synthesis

This paper proposes a method for tuning the weights of unit selection cost functions in syllable based text-to-speech (TTS) synthesis system. In this work, unit selection cost functions, namely target cost and concatenation cost, are designed appropriate to syllables. The method tunes the weights in such a way that perceptual preference patterns are appropriately considered while selecting the units. The method uses genetic algorithm to derive the optimal weights. Fitness function is designed to map perceptual preference patterns into weights of unit selection cost functions. The effectiveness of proposed method is evaluated by both subjective and objective measures. From the results, it is observed that the derived optimal weights can synthesize good quality speech compared to manually tuned weights.

Development of a silent speech interface driven by ultrasound and optical images of the tongue and lips

This article presents a segmental vocoder driven by ultrasound and optical images (standard CCD camera) of the tongue and lips for a “silent speech interface” application, usable either by a laryngectomized patient or for silent communication. The system is built around an audio–visual dictionary which associates visual to acoustic observations for each phonetic class. Visual features are extracted from ultrasound images of the tongue and from video images of the lips using a PCA-based image coding technique. Visual observations of each phonetic class are modeled by continuous HMMs. The system then combines a phone recognition stage with corpus-based synthesis. In the recognition stage, the visual HMMs are used to identify phonetic targets in a sequence of visual features. In the synthesis stage, these phonetic targets constrain the dictionary search for the sequence of diphones that maximizes similarity to the input test data in the visual space, subject to a concatenation cost in the acoustic domain. A prosody-template is extracted from the training corpus, and the final speech waveform is generated using “Harmonic plus Noise Model” concatenative synthesis techniques. Experimental results are based on an audiovisual database containing 1 h of continuous speech from each of two speakers.

Unit Selection Algorithm Using Bi-grams Model For Corpus-Based Speech Synthesis

In this paper, we present a novel statistical approach to corpus-based speech synthesis. Classically, phonetic information is defined and considered as acoustic reference to be respected. In this way, many studies were elaborated for acoustical unit classification. This type of classification allows separating units according to their symbolic characteristics. Indeed, target cost and concatenation cost were classically defined for unit selection. In Corpus-Based Speech Synthesis System, when using large text corpora, cost functions were limited to a juxtaposition of symbolic criteria and the acoustic information of units is not exploited in the definition of the target cost. In this manuscript, we token in our consideration the unit phonetic information corresponding to acoustic information. This would be realized by defining a probabilistic linguistic Bi-grams model basically used for unit selection. The selected units would be extracted from the English TIMIT corpora. Keywords—Unit selection, Corpus-based Speech Synthesis, Bigram model

Methods and apparatus for rapid acoustic unit selection from a large speech corpus

A speech synthesis system can select recorded speech fragments, or acoustic units, from a very large database of acoustic units to produce artificial speech. The selected acoustic units are chosen to minimize a combination of target and concatenation costs for a given sentence. However, as concatenation costs, which are measures of the mismatch between sequential pairs of acoustic units, are expensive to compute, processing can be greatly reduced by pre-computing and caching the concatenation costs. Unfortunately, the number of possible sequential pairs of acoustic units makes such caching prohibitive. However, statistical experiments reveal that while about 85% of the acoustic units are typically used in common speech, less than 1% of the possible sequential pairs of acoustic units occur in practice. A method for constructing an efficient concatenation cost database is provided by synthesizing a large body of speech, identifying the acoustic unit sequential pairs generated and their respective concatenation costs, and storing those concatenation costs likely to occur. By constructing a concatenation cost database in this fashion, the processing power required at run-time is greatly reduced with negligible effect on speech quality.

Concatenative speech synthesis using a finite-state transducer

A method for concatenative speech synthesis includes a processing stage that selects segments based on their symbolic labeling in an efficient graph-based search, which uses a finite-state transducer formalism. This graph-based search uses a representation of concatenation constraints and costs that does not necessarily grow with the size of the source corpus thereby limiting the increase in computation required for the search as the size of the source corpus increases. In one application of this method, multiple alternative segment sequences are generated and a best segment sequence is then be selected using characteristics that depend on specific signal characteristics of the segments.

Movement Epenthesis Generation Using NURBS-Based Spatial Interpolation

This study proposes a novel approach to the generation of video-based movement epenthesis for sign language and concentrates on a spatial interpolation approach using a nonuniform rational B-spline function to produce a smooth interpolation curve. To generate movement epenthesis, the beginning and end points are determined based on the concatenation cost, which is a linear combination of the distance, smoothness, and image distortion costs. The distance cost is defined as the normalized Euclidian distance between the palm locations at two cut points. The smoothness cost is determined by accumulating the second derivative of the curve. The image distortion cost is then estimated from the normalized Euclidian distance between the real and generated hand images. To evaluate the proposed approach, a set of sign video databases is collected and preprocessed with image calibration, content annotation, and principle component analysis. An image component overlapping procedure is also employed to yield a smooth sign video output. Evaluation results demonstrate that the synthesized trajectory is very close to the original trajectory. Moreover, the system was evaluated subjectively based on the results of ten hearing and five deaf people. The evaluation results also demonstrate the stability and feasibility of the proposed approach

Subjective evaluation of join cost and smoothing methods for unit selection speech synthesis

In unit selection-based concatenative speech synthesis, join cost (also known as concatenation cost), which measures how well two units can be joined together, is one of the main criteria for selecting appropriate units from the inventory. Usually, some form of local parameter smoothing is also needed to disguise the remaining discontinuities. This paper presents a subjective evaluation of three join cost functions and three smoothing methods. We also describe the design and performance of a listening test. The three join cost functions were taken from our previous study, where we proposed join cost functions derived from spectral distances, which have good correlations with perceptual scores obtained for a range of concatenation discontinuities. This evaluation allows us to further validate their ability to predict concatenation discontinuities. The units for synthesis stimuli are obtained from a state-of-the-art unit selection text-to-speech system: rVoice from Rhetorical Systems Ltd. In this paper, we report listeners' preferences for each join cost in combination with each smoothing method

On the computation of the Kullback-Leibler measure for spectral distances

Efficient algorithms for the exact and approximate computation of the symmetrical Kullback-Leibler (1998) measure for spectral distances are presented for linear predictive coding (LPC) spectra. A interpretation of this measure is given in terms of the poles of the spectra. The performances of the algorithms in terms of accuracy and computational complexity are assessed for the application of computing concatenation costs in unit-selection-based speech synthesis. With the same complexity and storage requirements, the exact method is superior in terms of accuracy.

A segmental speech coder based on a concatenative TTS

An extremely low bit rate speech coder based on a recognition/synthesis paradigm is proposed. In our speech coder, the speech signal is produced in a way which is similar to concatenative speech synthesis of text-to-speech (TTS). Hence, database construction, unit selection and prosody modification, which are the major parts of concatenative TTS, are employed to implement the speech coder. The synthesis units are automatically found in a large database using a joint segmentation/classification scheme. Dynamic programming (DP) is applied to unit selection in which two cost functions, an acoustic target cost and a concatenation cost are used to increase naturalness as well as intelligibility. Prosodic differences between the selected unit and the input segment are compensated for by time-scale and pitch modifications which are based on the harmonic plus noise (HNM) model framework. In single speaker tests, the proposed scheme gave intelligible and natural sounding speech at an average bit rate of about 580 b/s.

Concatenation of speech segments by use of a speech synthesizer

In a speech synthesizer apparatus, a weighting coefficient training controller calculates acoustic distances in second acoustic feature parameters between one target phoneme from the same phoneme and the phoneme candidates other than the target phoneme based on first acoustic feature parameters and prosodic feature parameters, and determines weighting coefficient vectors for respective target phonemes defining degrees of contribution to the second acoustic feature parameters for respective phoneme candidates by executing a predetermined statistical analysis therefor. Then, a speech unit selector searches for a combination of phoneme candidates which correspond to a phoneme sequence of an input sentence and which minimizes a cost including a target cost representing approximate costs between a target phoneme and the phoneme candidates and a concatenation cost representing approximate costs between two phoneme candidates to be adjacently concatenated, and outputs index information on the searched out combination of phoneme candidates. Further, a speech synthesizer synthesizes a speech signal corresponding to the input phoneme sequence by sequentially reading out speech segments of speech waveform signals corresponding to the index information and concatenating the read speech segments of the speech waveform signals.

A very low bit rate speech coder based on a recognition/synthesis paradigm

Previous studies have shown that a concatenative speech synthesis system with a large database produces more natural sounding speech. We apply this paradigm to the design of improved very low bit rate speech coders (sub 1000 b/s). The proposed speech coder consists of unit selection, prosody coding, prosody modification and waveform concatenation. The encoder selects the best unit sequence from a large database and compresses the prosody information. The transmitted parameters include unit indices and the prosody information. To increase naturalness as well as intelligibility, two costs are considered in the unit selection process: an acoustic target cost and a concatenation cost. A rate-distortion-based piecewise linear approximation is proposed to compress the pitch contour. The decoder concatenates the set of units, and then synthesizes the resultant sequence of speech frames using the harmonic+noise model (HNM) scheme. Before concatenating units, prosody modification which includes pitch shifting and gain modification is applied to match those of the input speech. With single speaker stimuli, a comparison category rating (CCR) test shows that the performance of the proposed coder is close to that of the 2400-b/s MELP coder at an average bit rate of about 800-b/s during talk spurts.

Concept-to-speech synthesis by phonological structure matching

This paper presents a new way of generating synthetic–speech waveforms from a linguistic description. The algorithm is presented as a proposed solution to the speech–generation problem in a concept–to–speech system. Off–line, a database of recorded speech is annotated so as to produce a phonological tree for each sentence in that database. Synthesis is performed by generating a phonological tree called the target tree, and searching the database of trees to find nodes that are the same in both trees. A search strategy using target and concatenation costs is then used to find the optimal sequence of units for the target sentence. This paper explains this algorithm, compares it with existing algorithms, and concludes with a discussion of future directions.