Speech Synthesis Module Research Articles

A Deep Learning Approach for ASL Recognition and Text-to-Speech Synthesis using CNN

Abstract: Sign language is a visual language that is used by the deaf and hard-of-hearing community to communicate. However, sign language is not universally understood by non-signers, which can create communication barriers for the deaf and hard-ofhearing individuals. In this paper, we present a novel application for American Sign Language (ASL) to text to speech conversion using deep learning techniques. Our app aims to bridge the communication gap between hearing-impaired individuals who use ASL as their primary mode of communication and individuals who do not understand ASL. The app comprises three main components: a hand gesture recognition module that detects and recognizes ASL signs, a text-tospeech synthesis module that converts recognized ASL signs to text, and a speech synthesis module that converts the text to speech

Facilitating the communication with deaf people: Building a largest Saudi sign language dataset

Recently, several countries have been trying hard to facilitate the integration of disabled people into their societies by ensuring equal opportunities through ease of access to social services, daily human necessities, and the labor market. Deafness is considered one of the major disabilities separating the deaf from their society. To integrate the deaf fully into society, a two-way mode of communication is required: one from the deaf to the hearing people, and the other from the hearing to the deaf. Communication from the hearing person to the deaf is generally easy and can be done through speech recognition and text-to-sign representations, but communication from the deaf to the hearing is somewhat difficult and requires a sign recognition module that recognizes the sign motions from the deaf and translates it to a text; following this, a speech synthesis module will translate this text to speech. To build the sign recognition module, a sign language dataset is required. This paper contributes to the literature by introducing a comprehensive survey of 17 Arabic sign language datasets and by developing a well-organized framework that is used to build a sign language dataset. This paper also contributes to the literature by creating the largest Saudi Sign Language (SSL) database—the King Saud University Saudi-SSL (KSU-SSL data-base)—with 293 signs, 33 signers, 145,035 samples, and 10 domains (healthcare, common, alpha-bets, verbs, pronouns and adverbs, numbers, days, kings, family, and regions). This paper also contributes to the literature by introducing a convolutional graph neural network (CGCN) architecture for sign language recognition and applying the proposed architecture to the built KSU-SSL database. The architecture is made up of a small number of separable 3DGCN layers, and is augmented with a spatial attention mechanism. This study is a part of the project that aims to develop a two-way communication system for Saudi deaf individuals.

An Emotion Speech Synthesis Method Based on VITS

People and things can be connected through the Internet of Things (IoT), and speech synthesis is one of the key technologies. At this stage, end-to-end speech synthesis systems are capable of synthesizing relatively realistic human voices, but the current commonly used parallel text-to-speech suffers from loss of useful information during the two-stage delivery process, and the control features of the synthesized speech are monotonous, with insufficient expression of features, including emotion, leading to emotional speech synthesis becoming a challenging task. In this paper, we propose a new system named Emo-VITS, which is based on the highly expressive speech synthesis module VITS, to realize the emotion control of text-to-speech synthesis. We designed the emotion network to extract the global and local features of the reference audio, and then fused the global and local features through the emotion feature fusion module based on the attention mechanism, so as to achieve more accurate and comprehensive emotion speech synthesis. The experimental results show that the Emo-VITS system’s error rate went up a little bit compared with the network without emotionality and does not affect the semantic understanding. However, this system is superior to other networks in naturalness, sound quality, and emotional similarity.

Voice Guidance System for Color Recognition Based on IoT

We often need to identify the colors of such objects as buildings, clothing, photos, and traffic signs in everyday life. Some people cannot distinguish among such colors. This study proposes a method to help people, including colorblind people, identify colors. We establish a platform for color identification that is connected by the Internet of Things (IoT). The Espressif Systems 32 (ESP32) single chip is connected to a Wi-Fi communication network to transmit data to the color identification platform. The system interface is displayed in the form of color code and color name display. The speech synthesis module Speech Synthesizer Node 6288 (SYN6288) is used to broadcast color-related information. Since there are many color conversion technologies in modern times, or other color recognition methods are not applicable to the guide-blind function. Therefore, the authors use the method of reading out color-related information to directly enable colorblind people to conveniently identify the colors of objects. After experiments: the accuracy of various colors, the accuracy of environmental impact, the comparison of the original sensing experiment and the effect of adding interference light sources, the time required for color sensing to identify various colors, and the interference test results of any color light source, it is proved that the research mentioned in this research. The proposed method can not only improve the rate of recognition of the system in different environments, but can also accurately identify a variety of colors. The platform is integrated with the IoT to allow users to quickly monitor the displayed data.

한국어 text-to-speech(TTS) 시스템을 위한 엔드투엔드 합성 방식 연구*

A typical statistical parametric speech synthesis (text-to-speech, TTS) system consists of separate modules, such as a text analysis module, an acoustic modeling module, and a speech synthesis module. This causes two problems: 1) expert knowledge of each module is required, and 2) errors generated in each module accumulate passing through each module. An end-to-end TTS system could avoid such problems by synthesizing voice signals directly from an input string. In this study, we implemented an end-to-end Korean TTS system using Google

Preserving Word-Level Emphasis in Speech-to-Speech Translation

Speech-to-speech translation S2ST is a technology that translates speech across languages, which can remove barriers in cross-lingual communication. In the conventional S2ST systems, the linguistic meaning of speech was translated, but paralinguistic information conveying other features of the speech such as emotion or emphasis were ignored. In this paper, we propose a method to translate paralinguistic information, specifically focusing on emphasis. The method consists of a series of components that can accurately translate emphasis using all acoustic features of speech. First, linear-regression hidden semi-Markov models LRHSMMs are used to estimate a real-numbered emphasis value for every word in an utterance, resulting in a sequence of values for the utterance. After that the emphasis translation module translates the estimated emphasis sequence into a target language emphasis sequence using a conditional random field model considering the features of emphasis levels, words, and part-of-speech tags. Finally, the speech synthesis module synthesizes emphasized speech with LR-HSMMs, taking into account the translated emphasis sequence and transcription. The results indicate that our translation model can translate emphasis information, correctly emphasizing words in the target language with 91.6% F-measure by objective evaluation. A listening test with human subjects further showed that they could identify the emphasized words with 87.8% F-measure, and that the naturalness of the audio was preserved.

Recovering the sight to blind people in indoor environments with smart technologies

Assistive technologies for blind people are showing a fast growth, providing useful tools to support daily activities and to improve social inclusion. Most of these technologies are mainly focused on helping blind people to navigate and avoid obstacles. Other works emphasize on providing them assistance to recognize their surrounding objects. Very few of them however couple both aspects (i.e., navigation and recognition). With the aim to address the aforesaid needs, we describe in this paper an innovative prototype, which offers the capabilities to (i) move autonomously and to (ii) recognize multiple objects in public indoor environments. It incorporates lightweight hardware components (camera, IMU, and laser sensors), all mounted on a reasonably-sized integrated device to be placed on the chest. It requires the indoor environment to be ‘blind-friendly’, i.e., prior information about it should be prepared and loaded in the system beforehand. Its algorithms are mainly based on advanced computer vision and machine learning approaches. The interaction between the user and the system is performed through speech recognition and synthesis modules. The prototype offers to the user the possibility to (i) walk across the site to reach the desired destination, avoiding static and mobile obstacles, and (ii) ask the system through vocal interaction to list the prominent objects in the user's field of view. We illustrate the performances of the proposed prototype through experiments conducted in a blind-friendly indoor space equipped at our Department premises.

Implementation of the Intelligent Voice System for Kazakh

Modern speech technologies are highly advanced and widely used in day-to-day applications. However, this is mostly concerned with the languages of well-developed countries such as English, German, Japan, Russian, etc. As for Kazakh, the situation is less prominent and research in this field is only starting to evolve. In this research and application-oriented project, we introduce an intelligent voice system for the fast deployment of call-centers and information desks supporting Kazakh speech. The demand on such a system is obvious if the country's large size and small population is considered. The landline and cell phones become the only means of communication for the distant villages and suburbs. The system features Kazakh speech recognition and synthesis modules as well as a web-GUI for efficient dialog management. For speech recognition we use CMU Sphinx engine and for speech synthesis- MaryTTS. The web-GUI is implemented in Java enabling operators to quickly create and manage the dialogs in user-friendly graphical environment. The call routines are handled by Asterisk PBX and JBoss Application Server. The system supports such technologies and protocols as VoIP, VoiceXML, FastAGI, Java SpeechAPI and J2EE. For the speech recognition experiments we compiled and used the first Kazakh speech corpus with the utterances from 169 native speakers. The performance of the speech recognizer is 4.1% WER on isolated word recognition and 6.9% WER on clean continuous speech recognition tasks. The speech synthesis experiments include the training of male and female voices.

Design and Implementation of Voice Control System in Flexible Manufacturing Cell

On the voice development platform of Microsoft Speech SDK, speech recognition and speech synthesis modules based on command control mode is built in this paper. Ethernet-based remote voice control system of intelligent flexible manufacturing cell is developed for machine tools and industrial robots. This paper designs an intelligent voice control system based on LabVIEW development environment, which realizes the human-machine voice interaction of flexible manufacturing cell and remote voice control. Experimental studies have shown that intelligent voice control system has high speech recognition rate and system reliability in a relatively quiet environment.

Study on the Text Analysis Technology of Tibetan Text to Speech

T ext to speech system(TTS) ,which converts text into speech through computer and then output the speech, is a popular research object in the realm of speech processing in recent years.A successful TTS outputs speech with clear timbre and natural fluency.TTS is composed of Text analysis, prosody control and speech synthesis modules. Text analysis module is in the front end, which gives the linguistics and phonetics text detailed information for the speech synthesis. This paper, based on the Tibetan written, pronunciation characteristics and pronunciation mecha- nism, gives the basic structure of TTS that is based on large-scale speech corpus and study of standardization, word segmentation and voice tagging technologies for Tibetan TTS.

Research and Implementation of Embedded Voice Interaction System Based on ARM in Intelligent Space

This paper presents the design and implementation of voice interaction system based on ARM in service robots intelligent space. ARM Cortex-M3 core based STM32F103 is used for main controller. And real-time embedded operating system μC/OS-II is used to schedule different tasks and manage peripheral devices. LD3320 and XFS4041CN are used for speech recognition and speech synthesis, respectively. Establish a dialogue set by defining the set of two-dimensional array, and then intelligent space updates the dialogue set dynamically via ZigBee wireless network for multiple scenes. If the recognition result is got by speech recognition module, dialogue management module will send corresponding texts to speech synthesis module, enabling text-to-speech output. At the same time, intelligent space decision support system will send corresponding commands to equipments by ZigBee. From lots of experiments and practical applications, we can conclude that voice interaction systems designed in the paper can gracefully satisfy the current request of voice interaction in service robots intelligent space, having great application value.

Developing a reading tutor: Design and evaluation of dedicated speech recognition and synthesis modules

When a child learns to read, the learning process can be enhanced by significant reading practice with individual support from a tutor. But in reality, the availability of teachers or clinicians is limited, so the additional use of a fully automated reading tutor would be beneficial for the child. This paper discusses our efforts to develop an automated reading tutor for Dutch. First, the dedicated speech recognition and synthesis modules in the reading tutor are described. Then, three diagnostic and remedial reading tutor tools are evaluated in practice and improved based on these evaluations: (1) automatic assessment of a child’s reading level, (2) oral feedback to a child at the phoneme, syllable or word level, and (3) tracking where a child is reading, for automated screen advancement or for direct feedback to the child. In general, the presented tools work in a satisfactory way, including for children with known reading disabilities.

2A1-C12 道案内サービスのためのコミュニケーション知能モジュール群の開発

We proposes intelligent communication modules that enable the human-robot interaction for guidance robots in public environments such as shopping malls. These modules consist of 13 modules including the module which detects the position and the behavior of people, the module which estimates the utterance period based on the facial images, speech recognition module, speech synthesis module, interaction control module, gesture synthesis module, etc.

Training Baldi to be multilingual: A case study for an Arabic Badr

In this paper, we describe research to extend the capability of an existing talking head, Baldi, to be multilingual. We use parsimonious client/server architecture to impose autonomy in the functioning of an auditory speech module and a visual speech synthesis module. This scheme enables the implementation and the joint application of text-to-speech synthesis and facial animation in many languages simultaneously. Additional languages can be added to the system by defining a unique phoneme set and unique phoneme definitions for the visible speech for each language. The accuracy of these definitions is tested in perceptual experiments in which human observers identify auditory speech in noise presented alone or paired with the synthetic versus a comparable natural face. We illustrate the development of an Arabic talking head, Badr, and demonstrate how the empirical evaluation enabled the improvement of the visible speech synthesis from one version to another.

Speech synthesis for the new Pan-European traffic message control system RDS-TMC

This paper reports on the speech synthesis module used to present spoken traffic messages through the car radio, as part of the “Traffic Message Control” system RDS-TMC. One of the basic ideas of this intended Pan-European service is its ability to provide traffic information in the driver's native language, independent of the language used in the geographical area or used for broadcasting. To accomplish this for an unlimited set of location names, speech synthesis is a must. To achieve best speech quality, text-to-speech techniques are combined with phonetic input that has been optimised off-line, resulting in a flexible system with almost natural sounding speech output. A prototype has been developed for German. First TMC radios for German and Germany will be presented at Internationale Funkausstellung by end of August 1997 in Berlin, which also is the start date for a regular TMC broadcasting service. Other countries and languages will follow in 1998.