Speech Synthesis Processing Research Articles

Computational Methods in Speech Derived Music: Field Review

Abstract Recent years have seen the emergence of diverse computational methods used to derive musical material from speech. While speech analysis, synthesis, and signal processing advancements have informed works, a review of computational methods is absent. This review addresses the gap, categorising methods from 1980 to 2023 found in literature and contemporary music festival programmes, identifying trends relating to live computation, automated compositional processes, use of acoustic-mechanical instruments, and themes including human-machine interaction, obscuring of semantic meaning, and highlighting geographical speech qualities. These approaches have broadened compositional palettes and fostered novel composer-performer-electronic interactions.

Natural language processing in assistive technologies

Natural language processing expanded in the last 30 years, arising to cover a wide range of applications in various domains, among them assistive technologies, helping people with disabilities due to accidents, aging, or genetic heritance, to enhance their social integration and their daily life. The growing need in assistive techniques has led to the involvement in the usual devices of intelligent modules to improve and facilitate their use, natural language processing attaining nowadays a high applicability in this domain. The paper discusses first basic methodologies involved in natural language processing like speech recognition, speech synthesis and text processing. Some possible applications are then presented with accent on experimental realization of intelligent modules for assistive devices in the laboratory of our university.

Towards Intelligent Aviation: The Integration of Voice Assistants in Autonomous Aircraft

This paper explores the integration of a virtual assistant into an autonomous aircraft, focusing on the utilization of cutting-edge technologies for automatic speech recognition, speech synthesis, and natural language processing. By leveraging the power of Edge AI, particularly NVIDIA Jetson devices, we demonstrate how this amalgamation of technologies creates a lightweight, mobile, and efficient voice assistant system. We introduce a novel approach to enhance the operational capabilities of autonomous aircraft, facilitating human-machine interaction and enhancing overall user experience. This research paves the way for the application of edge-based voice assistants in autonomous vehicles, offering a glimpse into the future of intelligent and responsive in-flight systems.

Mlphon: A Multifunctional Grapheme-Phoneme Conversion Tool Using Finite State Transducers

In this article we present the design and the development of a knowledge based computational linguistic tool, Mlphon [em.el.foːɳ] for Malayalam language. Mlphon computationally models linguistic rules using finite state transducers and performs multiple functions including grapheme to phoneme (g2p) and phoneme to grapheme (p2g) conversions, syllabification, phonetic feature analysis and script grammar check. This open source software tool, released under MIT license, is developed as a one-stop solution to handle different speech related text processing tasks for automatic speech recognition, text to speech synthesis and non-speech natural language processing tasks including syllable subword based language modeling, phoneme diversity analysis and text sanity check. The tool is evaluated on a manually crafted gold standard lexicon. Mlphon performs orthographic syllabification with 99% accuracy with a syllable error rate of 0.62% on the gold standard lexicon. For grapheme to phoneme conversion task, overall phoneme recognition accuracy of 99% with a phoneme error rate of 0.55% is obtained on gold standard lexicon. Additionally an extrinsic evaluation of Mlphon is performed by employing the pronunciation lexicon created using Mlphon, in Malayalam automatic speech recognition (ASR) task. Performance analysis in terms of the computation time of lexicon creation process and the word error rate (WER) on ASR task are presented along with a comparison over other automated tools for lexicon creation. Pronunciation lexicons with more than 100k commonly used Malayalam words in phonemised and syllabified forms is created and they are published as open language resources along with this work. We also demonstrate the usage of Mlphon on different natural language processing applications - syllable subword ASR, assisted pronunciation learning, phoneme diversity analysis and text sanity check. Being a knowledge based solution with open source code, Mlphon can be adapted to other languages of similar script nature.

Research on Correction Method of Spoken Pronunciation Accuracy of AI Virtual English Reading

In order to improve the pronunciation accuracy of spoken English reading, this paper combines artificial intelligence technology to construct a correction model of the spoken pronunciation accuracy of AI virtual English reading. Moreover, this paper analyzes the process of speech synthesis with intelligent speech technology, proposes a statistical parametric speech based on hidden Markov chains, and improves the system algorithm to make it an intelligent algorithm that meets the requirements of the correction system of spoken pronunciation accuracy of AI virtual English reading. Finally, this paper combines the simulation research to analyze the English reading, spoken pronunciation, and pronunciation correction of the intelligent system. From the experimental research results, the correction system of spoken pronunciation accuracy of AI virtual English reading proposed in this paper basically meets the basic needs of this paper to build a system.

Stemmer and phonotactic rules to improve n-gram tagger-based indonesian phonemicization

A phonemicization or grapheme-to-phoneme conversion (G2P) is a process of converting a word into its pronunciation. It is one of the essential components in speech synthesis, speech recognition, and natural language processing. The deep learning (DL)-based state-of-the-art G2P model generally gives low phoneme error rate (PER) as well as word error rate (WER) for high-resource languages, such as English and European, but not for low-resource languages. Therefore, some conventional machine learning (ML)-based G2P models incorporated with specific linguistic knowledge are preferable for low-resource languages. However, these models are poor for several low-resource languages because of various issues. For instance, an Indonesian G2P model works well for roots but gives a high PER for derivatives. Most errors come from the ambiguities of some roots and derivative words containing four prefixes: 〈ber〉, 〈meng〉, 〈peng〉, and 〈ter〉. In this research, an Indonesian G2P model based on n-gram combined with stemmer and phonotactic rules (NGTSP) is proposed to solve those problems. An investigation based on 5-fold cross-validation, using 50 k Indonesian words, informs that the proposed NGTSP gives a much lower PER of 0.78% than the state-of-the-art Transformer-based G2P model (1.14%). Besides, it also provides a much faster processing time.

Predicting utterance pitch targets in Yorùbá for tone realisation in speech synthesis

Pitch is a fundamental acoustic feature of speech and as such needs to be determined during the process of speech synthesis. While a range of communicative functions are attributed to pitch variation in speech of all languages, it plays a vital role in distinguishing meaning of lexical items in tone languages. As a number of factors are assumed to affect the realisation of pitch, it is important to know which mechanisms are systematically responsible for pitch realisation in order to be able to model these effectively and thus develop robust speech synthesis systems in under-resourced environments. To this end, features influencing syllable pitch targets in continuous utterances in Yorùbá are investigated in a small speech corpus of 4 speakers. It is found that the previous syllable pitch level is strongly correlated with pitch changes between syllables and a number of approaches and features are evaluated in this context. The resulting models can be used to predict utterance pitch targets for speech synthesisers (whether it be concatenative or statistical parametric systems), and may also prove useful in speech-recognition systems.

Design of Tree Structure in Context Clustering Process of Hidden Markov Model-Based Thai Speech Synthesis

Problem statement: In HMM-based Thai speech synthesis, the tone degradation due to the imbalance of training data of all tones. Some distortion of syllable duration is obviously noticeable when the system is trained with a small amount of data. These problems cause the decrement in naturalness and intelligibility of the synthesized speech. Approach: This study proposes an approach to improve the correctness of tone of the synthesized speech which is generated by an HMM-based Thai speech synthesis system. In the tree-based context clustering process, tone groups and tone types are used to design four different structures of decision tree including a single binary tree structure, a simple tone-separated tree structure, a constancy-based-tone-separated tree structure and a trend-based-tone-separated tree structure. Results: A subjective evaluation of tone correctness is conducted by using tone perception of eight Thai listeners. The simple tone-separated tree structure gives the highest level of tone correctness, while the single binary tree structure gives the lowest level of tone correctness. The additional contextual tone information which is applied to all structures of the decision tree achieves a significant improvement of tone correctness. Finally, the evaluation of syllable duration distortion among the four structures shows that the constancy-based-tone-separated and the trend-based-tone-separated tree structures can alleviate the distortions that appear when using the simple tone-separated tree structure. Conclusion: The appropriate structure of tree in context clustering process with the additional contextual tone information can improve the correctness of tones, while the constancy-based-tone-separated and the trend-based-tone-separated tree structures can alleviate the syllable duration distortions.

Speech synthesis method and apparatus, and dictionary generation method and apparatus

In a speech synthesis process, micro-segments are cut from acquired waveform data and a window function. The obtained micro-segments are re-arranged to implement a desired prosody, and superposed data is generated by superposing the re-arranged micro-segments, so as to obtain synthetic speech waveform data. A spectrum correction filter is formed based on the acquired waveform data. At least one of the waveform data, micro-segments, and superposed data is corrected using the spectrum correction filter. In this way, “blur” of a speech spectrum due to the window function applied to obtain micro-segments is reduced, and speech synthesis with high sound quality is realized.

Note from the editor: “Expositions of Romanian scientists on the design of text-to-speech synthesis and natural language understanding and generation systems”

This special issue of the Journal is devoted to the assorted work of twelve Romanian speech scientists who have brought sharp analytical insights and new methodological approaches to the design of high performing text-to-speech (TTS) and natural language understanding and generation systems. Volume 12, No. 2 consists of five papers. Each paper, albeit addressing a different aspect of speech synthesis and natural language processing, is like the others in the rigorous diligence and precision of the author’s scientific scrutiny. The contributors to this special issue have served their Romanian academic and research institutions with distinction, while adding to the vast body of research in speech technology through diverse publications in the professional literature, not to mention their presentations at various academic conferences and workshops. The inspiration for this special issue comes from individuals such as Dan Tufis of the Romanian Academy, Institute for Artificial Intelligence. Back in 2004, I had the pleasure of hearing Dan give an enlightening presentation on multilingual word sense disambiguation (WSD) at the annual Natural Language Processing and Cognitive Science workshop, an event chaired by Bernadette Sharp of Staffordshire University. Several years later, when I became IJST’s Editor-in-Chief, I explored the Journal’s archives and found a compelling special issue dealing with speech technology research taking place in Romania (Volume 5, No. 3, September 2002). The guest editor of that issue, Dragos Burlieanu of Politehnica University of Bucharest, working under the auspices of the Journal’s then-Editor-in-Chief

Design, implementation and evaluation of the Czech realistic audio-visual speech synthesis

This paper presents the whole process of creation of audio-visual speech synthesis system. Such system consists of two main parts, the acoustic synthesis emulating human speech and the facial animation emulating the human lip articulation. The acoustic subsystem is based on concatenation-based speech synthesis. The visual subsystem is designed as a realistic, fully three-dimensional parametrically controllable facial animation model. To be able to parametrically control the animation to emulate human articulation, the set of visual parameters has to be obtained for all basic speech units. To provide realistic animation, the database of lip movements of a real person need to be recorded and expressed by suitable parameterization. The set of control parameters for visual animation is then derived from this database. The 3D model of a head based on a head of a real person also makes the animation more realistic. To obtain such model, a 3D scanning of a real person has to be adopted. We present the design and implementation of above-mentioned process. The aim is to obtain realistic audio-visual speech synthesis with possibility to change the 3D head model according to particular person. The design, acquisition and processing of audio-visual speech corpus for such purpose is presented. Next, the process of both acoustic and visual speech synthesis is described. The visual speech synthesis comprises the tasks of model training, animation control, and co-articulation modelling. A facial animation can also increase intelligibility of a telephone speech to people with hearing disabilities. In such case the textual information to control the animation is not available. Solution to the problem of mapping visual parameters from speech signal either directly or through recognized text is presented. Furthermore, the 3D scanning algorithm is presented. It allows to obtain realistic 3D model based on a head of a real person and thus to personalize the talking head. In the end of this paper, evaluation of intelligibility of the presented audio-visual speech synthesis and its possible applications are presented.

멀티미디어 신호처리에 기초한 스마트홈 가상대화 시스템

본 논문은 보다 편리한 가정 생활환경 구축을 목적으로 한 가상대화시스템 구현에 관한 연구이다. 이를 실현하기 위하여 본 논문은 음성인식, 음성합성, 비디오 신호 및 센서신호처리 등의 멀티미디어 신호처리에 그 기술적 기반을 두고 있다. 대화시스템의 중요한 모듈로서의 음성합성기, HM-Net(Hidden Markov Network)에 기반한 실시간 음성인식기, 픽셀의 밝기차를 이용한 실시간 움직임 검출 및 터치센서 등을 대화시스템에 통합함으로써 이루어진다. 실제 구동 실험에서 주위 노이즈 환경의 영향으로 시뮬레이션 결과보다는 성능이 떨어지나, 소파에 앉아있는 동안 자동되는 시스템의 실험 평가에서 가전제품 능의 컨트롤이 비교적 사용하기 쉬웠다는 결과를 얻었다. This paper focuses on the use of the virtual dialog system whose aim is to build more convenient living environments. In order to realize this, the main emphasis of the paper lies on the description of the multimedia signal processing on the basis of the technologies such as speech recognition, speech synthesis, video, or sensor signal processing. For essential modules of the dialog system, we incorporated the real-time speech recognizer based on HM-Net(Hidden Markov Network) as well as speech synthesis into the overall system. In addition, we adopted the real-time motion detector based on the changes of brightness in pixels, as well as the touch sensor that was used to start system. In experimental evaluation, the results showed that the proposed system was relatively easy to use for controlling electric appliances while sitting in a sofa, even though the performance of the system was not better than the simulation results owing to the noisy environments.

Microsegment-based speech-synthesis process

The invention concerns a digital speech-synthesis process whereby utterances in a language are recorded, the recorded utterances are divided into speech segments which are stored so as to allow their allocation to specific phonemes; a text which is to be output as speech is converted to a phoneme chain and the stored segments are output in a sequence defined by the phoneme chain; an analysis of the text to be output as speech is carried out and thus provides information which completes the phoneme chain and modifies the timing sequence signal for the speech segments which are to be strung together for output as speech. The invention is characterised by the use of, as speech segments, microsegments consisting of: segments for vowel halves and semi-vowel halves, vowels standing between consonants being split into two microsegments, a first vowel half beginning shortly before the start of the vowel and extending as far as the vowel middle, and a second vowel half from the vowel middle to just before the vowel end; segments for quasi-stationary vowel components cut from the middle of a vowel; consonant segments beginning shortly before the front phoneme boundary and ending shortly before the rear phoneme boundary; and segments for vowel-vowel sequences cut from the middle of a vowel-vowel transition.

Analysis of the Fiber Bragg Gratings using the Lattice Filter Model

We propose a new method for analyzing fiber Bragg gratings with an arbitrary index profile and an arbitrary individual grating length. The proposed method is based on the lattice filter model which is widely used in applications ranging from digital filtering to speech synthesis and explosive seismic signal processing. Lattice filter interpretation provides us with an accurate and simple tool for analyzing an arbitrary index profile and arbitrary aperiodic fiber grating structures, and gives us further insight into the understanding of fiber Bragg gratings. To verify the validity of the proposed model experimentally, we have fabricated two grating structures; the short-period (periodic) fiber Bragg grating structure and the chirped fiber Bragg grating structure. We have observed that the calculated transmission spectrum and the calculated reflectivity using our lattice filter model match very closely to the corresponding measured spectrum in the wavelength band of interest.

Speech synthesis using glottal closure instants determined from adaptively-threshold wavelet transforms

A speech synthesis system making use of a pitch-synchronous waveform overlap method to realize stable speech synthesis processing in which pitch shaking is negligible. The present invention is characterized in that glottal closure instants are used as reference points (pitch marks) for overlapping. Since the glottal closure instants can be extracted stably and accurately by using dyadic Wavelet conversion, speech in which pitch shaking is negligible and rumbling sounds are minimized can be synthesized stably. In addition, more flexible waveform separation becomes possible by setting the reference point for overlapping and the reference point for waveform separation to different positions. The extraction of glottal closure instants is performed by searching the local peaks of the dyadic Wavelet conversion, but preferably a threshold value for searching for the local peaks of the dyadic Wavelet conversion is adaptively controlled each time dyadic Wavelet conversion is obtained.

Voiced excitation functions calculated from micropower impulse radar information

Efforts underway at the Lawrence Livermore National Laboratory to use newly designed micropower impulse radars (MIR) to measure in real time the excitation function of the vocal tract will be presented. Studies undertaken in collaboration with the University of California at Davis and the University of Iowa with high-speed laryngoscopic cameras, electroglottographs, flow masks, and subglottal pressure transducers have solidified the relationship between the signal returned by the MIR and the voiced excitation function of the vocal tract. As a result, for the first time a transfer function of the vocal tract can be calculated in real time and with unprecedented clarity for voiced speech. This new capability could have significant implications for improvements in speech recognition and speech synthesis processing.

Processing device for speech synthesis by addition of overlapping waveforms

A process of speech synthesis from diphones stored in a dictionary as waveforms, for text-to-speech conversion, comprises supplying a sequence of phoneme codes and respective prosodic information, and, for each phoneme, analyzing and synthesizing each phoneme, and then concatenating the synthesized phonemes. For each phoneme, two diphones are selected among the stored diphones and the presence of voicing is determined. For voiced phonemes, the respective waveforms of the two diphones constituting the phoneme are filtered by a window which is centered on a point of the selected waveform representative of the beginning of a pulse response of vocal cords to excitation thereof. The window has a width substantially equal to twice the greater of the original fundamental period and the fundamental synthesis period and has an amplitude progressively decreasing from the center of the window. The signals resulting from the filtering and obtained for each diphone are time shifted so as to be spaced apart by a time equal to the fundamental synthesis period. Synthesis is achieved by adding the displaced overlapping signals.

Speech synthesis apparatus and method

A language processing unit (201) identifies a word by performing language analysis on a text supplied from a text holding unit (201). A synthesis selection unit (209) selects speech synthesis processing performed by a rule-based synthesis unit (204) or speech synthesis processing performed by a prerecorded-speech-based synthesis unit (206) for a word of interest extracted from the language analysis result. The selected rule-based synthesis unit or prerecorded-speech-based synthesis unit executes speech synthesis processing for the word of interest.

Poetry I: Teaching verse with CAI

The teaching of literature through CAI raises problems of both a linguistic and instructional nature; student involvement and creativity in studying literature, and especially poetry, is difficult to build into a computer-based lesson. We have confronted these difficulties in the lessonPoetry I, which introduces undergraduates to basic concepts of poetic verse in a design using screen display, speech synthesis, and verse processing to maximize interactivity and student involvement. The lesson contains instructional modules which include the student's composition of a limerick. Computational processing of the limerick's text enables the program to offer guidance as the student composes and revises the verse and gains first-hand experience with metrical language. The significant problems inherent in processing verse are addressed by adapting Digital Equipment Corporation's DECTalk speech synthesizer as an engine converting natural language verse text into accessible strings of phonemic symbols. Although the verse processor cannot “scan” verse, it can interpret DECTalk's symbols reliably enough to successfully stimulate students' thinking about their verse compositions. Preliminary responses to the program have been favorable and demonstrate its effectiveness in involving students more deeply in learning about poetic verse. This suggests that the techniques of verse processing prototyped inPoetry I might be usefully extended to other types of verse and levels of study.

Applications of Dynamic Programming to Speech and Language Processing

Dynamic programming (DP) is a mathematical programming technique for solving certain sequential optimization problems. In this paper, we focus our discussion on DP applications in the areas of speech and language processing. Because of the sequential nature of speech and language events, most applications can generally be formulated as a sequential decision process. DP can then be used to obtain the solution to these processes by decomposing them into a series of smaller local problems and solving them sequentially. It is impossible to give adequate treatment to each such application in this paper. Instead, we briefly illustrate the basic way in which DP techniques have been applied to several key areas of speech and language research in the Information Principles Research Laboratory. The applications discussed in this paper include speaker verification, speech recognition, speech synthesis, speech analysis, speech modeling, speech enhancement, speech segmentation, speech coding, speech transmission, language modeling, and natural language processing. These applications have been proven successful in enhancing human-machine communications.