Text-to-speech Research Articles

Deep Speech Synthesis and Its Implications for News Verification: Lessons Learned in the RTVE-UGR Chair

This paper presents the multidisciplinary work carried out in the RTVE-UGR Chair within the IVERES project, whose main objective is the development of a tool for journalists to verify the veracity of the audios that reach the newsrooms. In the current context, voice synthesis has both beneficial and detrimental applications, with audio deepfakes being a significant concern in the world of journalism due to their ability to mislead and misinform. This is a multifaceted problem that can only be tackled adopting a multidisciplinary perspective. In this article, we describe the approach we adopted within the RTVE-UGR Chair to successfully address the challenges derived from audio deepfakes involving a team with different backgrounds and a specific methodology of iterative co-creation. As a result, we present several outcomes including the compilation and generation of audio datasets, the development and deployment of several audio fake detection models, and the development of a web audio verification tool addressed to journalists. As a conclusion, we highlight the importance of this systematic collaborative work in the fight against misinformation and the future potential of audio verification technologies in various applications.

Preventing type 2 diabetes in Portugal: a novel person-centered digital intervention

Abstract Introduction Diabetes, particularly type 2 diabetes (T2D), stands as one of the leading noncommunicable diseases worldwide. In Portugal, the prevalence of diabetes has surged, with T2D affecting 14% of adults in 2021. Its repercussions extend beyond individual health, deeply impacting familial, social, and community dynamics, presenting a significant public health challenge globally. Objectives The VA|PREVENTION project (https://doi.org/10.54499/2022.02969.PTDC) aims to assess the (cost-) effectiveness and implementation of an innovative person-centered digital intervention to prevent T2D among high-risk adults. Methodology A comprehensive approach encompasses the Behavior Change Wheel for intervention design, scalability principles, the WHO guidelines for digital health interventions, co-design, and the ISO framework for usability assessment. Evaluation will be pursued through a type 1 hybrid effectiveness-implementation study (randomized controlled trial), supplemented by qualitative and economic studies. Results The web application prototype features Ema, an automated virtual human coach (VHC), educational menus and a dashboard (version 1.0). Co-design included 2 workshops with 11 representatives of end-users and health professionals. Ema uses standardized techniques from an ontology to support behavior change, based on an initial assessment of low-level target behaviors (e.g. walking, fruit consumption, and reducing red meat intake), and barriers to performing these behaviors. Ema communicates in a helpful-cooperative style using a synthetic voice in European Portuguese, with its speech also appearing as subtitles on the interface. Users interact with Ema by pressing buttons or entering data, such as the daily number of steps. Ethics approval was granted for usability assessment with end-users. Conclusions The project is expected to yield health, scientific, technological, and policy impacts, powered by an exploitation and sustainability roadmap. Key messages • Innovative type 2 diabetes prevention through a scalable web app with a virtual human coach. • Comprehensive and robust approach for developing and evaluating the digital behavior change intervention.

Design of Interactive Artificial Intelligence for Early Cognitive Diagnosis

Due to the aging population in South Korea, the proportion of elderly people aged 65 and over is expected to increase from 14.9% in 2019 to 46.5% by 2067. The number of elderly people per 100 working-age population (15-64 years old) is also anticipated to rise to 102.4 by 2067. Population aging is recognized as a social issue, leading to problems such as increased chronic diseases, higher levels of elderly isolation, and insufficient medical infrastructure. To solve the problem of cognitive decline, such as dementia due to the aging of the population, research is actively being conducted in various fields, such as simulating cognitive ability and learning, inference, prediction, and problem-solving using artificial intelligence deep learning technology in the form of a fusion of artificial intelligence and realistic content technology. This study is on an interactive cognitive early diagnosis training system using artificial intelligence. The speech recognition technology for early cognitive diagnosis uses Selvas AI (Artificial Intelligence)'s speech recognition STT (Speech to Text)-TTS (Text to Speech). AI speech recognition interaction can increase psychological safety through conversation with users. It evaluates cognition (dementia) using MMSE (Mini-Mental State Examination)-K (DS), and it is a system that evaluates seven cognitive areas through the CERAD (Consortium to Establish a Registry for Alzheimer Disease)-K analysis system. The design of interactive artificial intelligence for early cognitive diagnosis aims to improve the cognitive function and daily living abilities of the elderly population.

Self‐Powered Wearable Pressure Sensors for Detection and Separation of Signals for Various Human Movements

A detailed study on the dynamic response of mountable pressure sensors is presented, with a focus on foot pressure sensors integrated with carbon nanotube (CNT)‐coated cotton fibers. The research explores the sensor's sensitivity to pressure changes, repeatability, hysteresis, and durability through rigorous modeling and experimental validation. Computational simulations using Python and experimental data demonstrate the sensor's nonlinear conductance response to applied force, attributed to the varying contact area and number of contact points among the fibers. Long‐term outdoor exposure tests confirm the material's resilience, maintaining electrical conductivity and structural integrity. The study also demonstrates the sensor's ability to monitor human activities like walking, running, stair climbing, and jumping, as well as detecting muscle movements during swallowing, coughing, and speech, highlighting potential applications in health monitoring and artificial voice synthesis. The MRMR algorithm is used for feature selection to distinguish activities, demonstrating the sensor's potential for activity recognition. Additionally, a piezoelectric sensor on the pressure sensors estimates harvested electric power to power wearable devices. This work contributes to the advancement of self‐powered wearable pressure sensors to monitor real‐time human activity, with implications for healthcare, sports performance, and assistive technologies.

An Antibullying and Therapeutic Interactive Visual Novel: TheraBuddy

This proceeding is an ongoing doctoral dissertation that will examine the effect of a serious game on bullying participant roles and social-emotional well-being. "This serious game, which is called TheraBuddy, is a Turkish interactive visual novel that is aimed at helping children and youth using cognitive-behavioral counseling and solution-focused counseling techniques." The protagonist of TheraBuddy is the player. According to story, the player is transfered to a school where bullying incidents occur. The player makes friends named Elma, Den, and Lu at this school. The player’s goal during the story is to offer solution-focused or cognitive-behavioral help to friends involved in bullying incidents, whether as victims or perpetrators. TheraBuddy consists of nine parts, including an introduction and eight chapters. Throughout the story, two characters named Ozguro and Bilgin help the player make choices, and the story of TheraBuddy continues according to the player’s choices. TheraBuddy was coded and hosted using the Ren'Py game engine. Characters and backgrounds were designed through Canva, and the TheraBuddy characters were voiced using Elevenlabs AI Voice Generator. As a result, an interactive anti-bullying game that is currently only playable offline has been developed. Work is currently being done to make this offline game a web-based game. It is believed that TheraBuddy will be helpful in combating bullying and strengthening students socially and emotionally. In the future, when the online version of TheraBuddy is completed, this belief will be tested under experimental conditions.

Development of a Children’s Educational Dictionary for a Low-Resource Language Using AI Tools

Today, various interactive tools or partially available artificial intelligence applications are actively used in educational processes to solve multiple problems for resource-rich languages, such as English, Spanish, French, etc. Unfortunately, the situation is different and more complex for low-resource languages, like Kazakh, Uzbek, Mongolian, and others, due to the lack of qualitative and accessible resources, morphological complexity, and the semantics of agglutinative languages. This article presents research on early childhood learning resources for the low-resource Kazakh language. Generally, a dictionary for children differs from classical educational dictionaries. The difference between dictionaries for children and adults lies in their purpose and methods of presenting information. A themed dictionary will make learning and remembering new words easier for children because they will be presented in a specific context. This article discusses developing an approach to creating a thematic children’s dictionary of the low-resource Kazakh language using artificial intelligence. The proposed approach is based on several important stages: the initial formation of a list of English words with the use of ChatGPT; identification of their semantic weights; generation of phrases and sentences with the use of the list of semantically related words; translation of obtained phrases and sentences from English to Kazakh, dividing them into bigrams and trigrams; and processing with Kazakh language POS pattern tag templates to adapt them for children. When the dictionary was formed, the semantic proximity of words and phrases to the given theme and age restrictions for children were taken into account. The formed dictionary phrases were evaluated using the cosine similarity, Euclidean similarity, and Manhattan distance metrics. Moreover, the dictionary was extended with video and audio data by implementing models like DALL-E 3, Midjourney, and Stable Diffusion to illustrate the dictionary data and TTS (Text to Speech) technology for the Kazakh language for voice synthesis. The developed thematic dictionary approach was tested, and a SUS (System Usability Scale) assessment of the application was conducted. The experimental results demonstrate the proposed approach’s high efficiency and its potential for wide use in educational purposes.

A Study on the Improvement of Pronunciation of AI Vocal Voice Synthesis based on Phonological Characteristics

A Study on the Improvement of Pronunciation of AI Vocal Voice Synthesis based on Phonological Characteristics

HiddenSinger: High-quality singing voice synthesis via neural audio codec and latent diffusion models

Recently, denoising diffusion models have demonstrated remarkable performance among generative models in various domains. However, in the speech domain, there are limitations in complexity and controllability to apply diffusion models for time-varying audio synthesis. Particularly, a singing voice synthesis (SVS) task, which has begun to emerge as a practical application in the game and entertainment industries, requires high-dimensional samples with long-term acoustic features. To alleviate the challenges posed by model complexity in the SVS task, we propose HiddenSinger, a high-quality SVS system using a neural audio codec and latent diffusion models. To ensure high-fidelity audio, we introduce an audio autoencoder that can encode audio into an audio codec as a compressed representation and reconstruct the high-fidelity audio from the low-dimensional compressed latent vector. Subsequently, we use the latent diffusion models to sample a latent representation from a musical score. In addition, our proposed model is extended to an unsupervised singing voice learning framework, HiddenSinger-U, to train the model using an unlabeled singing voice dataset. Experimental results demonstrate that our model outperforms previous models regarding audio quality. Furthermore, the HiddenSinger-U can synthesize high-quality singing voices of speakers trained solely on unlabeled data.

Improving foreign language reading technique using neural text-to-speech technology

Introduction. The intensification and technical development of the country, the COVID-19 pandemic and the post-crisis phase have led to the active integration of digital resources into the educational process, which has led to a new stage of development of the modern Russian education system – its digitalization. At present, new digital technologies, including artificial intelligence technologies, are being developed and introduced into the teaching of foreign languages. This article considers the use of text-to-speech technology functioning via neural networks to improve the reading technique in English and German of students of linguistic and non-linguistic specialties. Reading skills in a foreign language can be improved with the help of Natural Reader and TTSFree speech synthesizers, which students can use on their own. Due to application of the new FES HVE (Federal Educational Standards of Higher Vocational Education) of 2018 the volume of classroom work with students has decreased and the volume of independent work of students and students number in educational groups have increased, according to this matter the practical use of speech synthesizers in a foreign language classes seems relevant and important. Materials and Methods. Conducting an experiment to determine the effectiveness of the use of Natural Reader and TTSFree speech synthesizers to improve reading skills of linguistic and non-linguistic specialties students, calculating the number of phonetic errors when reading a foreign text before and after using of the voice generator, summarizing and interpreting the results obtained. Results. The list of publications studied in the process of scientific research makes 40 Russian and foreign works. These materials allowed to study the application of TTS technology with the use of neural networks, as well as Natural Reader and TTSFree speech synthesizers in foreign language classes at the university, which allowed to prove in practice the effectiveness of the analyzed services to improve the reading technique in English and German. Discussion and Conclusions. On the basis of theoretical research of scientific sources, as well as studying the experience of practical application of speech synthesizers Natural Reader and TTSFree, the conclusion about the great didactic potential of TTS technology is formulated. Possible applications of speech synthesis include the creation of an authentic language environment, the automation of reading, listening and speaking skills, as well as the self-control of students with their written language, voiced by TTS technology.

An instantaneous voice synthesis neuroprosthesis.

Brain computer interfaces (BCIs) have the potential to restore communication to people who have lost the ability to speak due to neurological disease or injury. BCIs have been used to translate the neural correlates of attempted speech into text1-3. However, text communication fails to capture the nuances of human speech such as prosody, intonation and immediately hearing one's own voice. Here, we demonstrate a "brain-to-voice" neuroprosthesis that instantaneously synthesizes voice with closed-loop audio feedback by decoding neural activity from 256 microelectrodes implanted into the ventral precentral gyrus of a man with amyotrophic lateral sclerosis and severe dysarthria. We overcame the challenge of lacking ground-truth speech for training the neural decoder and were able to accurately synthesize his voice. Along with phonemic content, we were also able to decode paralinguistic features from intracortical activity, enabling the participant to modulate his BCI-synthesized voice in real-time to change intonation, emphasize words, and sing short melodies. These results demonstrate the feasibility of enabling people with paralysis to speak intelligibly and expressively through a BCI.

AI-Integrated Smart Glasses for Enhancing Reading and Guidance Independence for the Visually Impaired

Since there are blind and partially sighted people all over the world, they face significant challenges when it comes to reading. In the past, Braille was the only solution for the blind and visually impaired to read and learn. However, there is a lack of training in Braille and a shortage of Braille machines and facilities. To address these issues, a smart reader for the blind integrated with Raspberry Pi technology has been proposed. The system utilizes YOLO (You only look once) for object detection, OCR (Optical Character Reader) for text recognition, and Google Text-to-Speech to capture and read text aloud through a speaker or headphone. Smart glasses offer real-time assistance and enhance the daily lives of visually impaired individuals. Through a combination of sensors, cameras, and audio feedback systems, these smart glasses enhance the reading capability of the visual impaired. Moreover, advancements in artificial intelligence and Deep learning algorithms further enhance the capabilities of these devices, enabling more accurate personalized assistance.

Hierarchical Sentiment Analysis Framework for Hate Speech Detection: Implementing Binary and Multiclass Classification Strategy

A significant challenge in automating hate speech detection on social media is distinguishing hate speech from regular and offensive language. These identify an essential category of content that web filters seek to remove. Only automated methods can manage this volume of daily data. To solve this problem, the community of Natural Language Processing is currently investigating different ways of hate speech detection. In addition to those, previous approaches (e.g., Convolutional Neural Networks, multi-channel BERT models, and lexical detection) have always achieved low precision without carefully treating other related tasks like sentiment analysis and emotion classification. They still like to group all messages with specific words in them as hate speech simply because those terms often appear alongside hateful rhetoric. In this research, our paper presented the hate speech text classification system model drawn upon deep learning and machine learning. In this paper, we propose a new multitask model integrated with shared emotional representations to detect hate speech across the English language. The Transformer-based model we used from Hugging Face and sentiment analysis helped us prevent false positives. Conclusion. We conclude that utilizing sentiment analysis and a Transformer-based trained model considerably improves hate speech detection across multiple datasets.

Student Progress Tracing of Arabic Learning: A Qualitative Investigation of Speaking Skills Formative Evaluation in Islamic Elementary School

Formative evaluation of Arabic language proficiency plays a crucial role in identifying students' progress in the learning process, particularly in speaking skills or maharatul kalam. The aim of this study is to explore the implementation of formative evaluation on Arabic speaking skills in the third-grade class of SDIT Harapan Bunda Manado. The research adopts a qualitative approach with data collected through interviews with Arabic language teachers and observations in three different classes. The findings indicate that the types of speaking skills tests implemented in this school align with the principles of Arabic language teaching for beginners. The formative evaluation method of maharatul kalam includes reading texts aloud and identifying vocabulary words presented by the teacher. This evaluation is conducted formatively throughout the learning process. Both evaluation methods aim to measure students' reading, pronunciation, and vocabulary mastery as a foundation for developing more complex Arabic language skills in subsequent levels. Interactive evaluation, utilizing varied technologies and approaches, enables teachers to provide constructive feedback and customize learning according to individual student needs.

Limitations of Disembodied Computer-Generated Voice to Convey Emotion in Multimedia Lessons

The goal of this study was to determine whether learning processes and outcomes are affected by the emotional tone of the computer-generated voice in a narrated slideshow. In a between-subjects design, participants viewed a narrated slideshow on lightning formation involving a computer-generated female (Experiment 1) or male voice (Experiment 2) that displayed happy, content, angry, or sad emotion. On subsequent surveys and post-tests, students could recognize positive or negative emotions conveyed by the female voice but not the male voice. The emotional tone of the instructor’s voices had minimal impact on students’ ratings of felt emotion during learning, ratings of social connection with the instructor, or scores on retention and transfer tests of learning outcome. The findings highlight the limitations of computer-generated voices to convey emotions that trigger affective, social, and cognitive processes when an onscreen instructor is absent, thereby suggesting a boundary condition for the cognitive-affective model of e-learning.

An Analysis of Grammatical Cohesion in TED Speech Texts: A Case Study of Teach Girls Bravery, Not Perfection

Based on the cohesion theory of Halliday and Hasan, this study analyzed the application of cohesion devices and explored the significant characteristics of their frequency and distribution in a TED speech Teach girls bravery, not perfection . Through quantitative and qualitative analysis, it is found that reference is the most frequently used grammatically cohesive device, mainly due to the high frequency of personal reference in the whole speech text, which refers to both the speaker and the audience. And in the subcategory of conjunctions, additive conjunctions are the most frequently used, and the simplest form and , is used to add some new information and promote the development of the speech content. Finally, substitution and omission were used in a limited way.

Exploration of Curriculum Reform of Main Japanese Courses Focusing on Curriculum Ideology and Politics and Online Teaching-- Take Business Japanese Listening as an example

The application of generative artificial intelligence tools in the field of education, especially in the online course construction, has attracted more and more attention. These tools can provide customized learning resources and feedback according to students personalized needs, so as to effectively improve students learning effect. At the same time, foreign language education can use digital technology to better serve the moral education of people, and AIGC technology gives us more technology and means. Taking the construction of Business Japanese Listening as an example, by using AIGC tools such as voice generation and AI digital human, we can quickly and conveniently produce dynamic business listening materials and animated sections of Chinese cultural stories, which can help students improve the efficiency of listening practice and promote cultural education. The production process itself can also help teachers grow up, so that teachers can realize the transformation from content teacher to content producer in the digital age. Therefore, innovation in course content through AIGC tool is an important path for the online teaching reform of foreign language courses, including Japanese.

AI-based avatars are changing the way we learn and teach: benefits and challenges

Advancements in the generative AI field have enabled the development of powerful educational avatars. These avatars embody a human and can, for instance, listen to users’ spoken input, generate an answer utilizing a large-language model, and reply by speaking with a synthetic voice. A theoretical introduction summarizes essential steps in developing AI-based educational avatars and explains how they differ from previously available educational technologies. Moreover, we introduce GPTAvatar, an open-source, state-of-the-art AI-based avatar. We then discuss the benefits of using AI-based educational avatars, which include, among other things, individualized and contextualized instruction. Afterward, we highlight the challenges of using AI-based educational avatars. Major problems concern incorrect and inaccurate information provided, as well as insufficient data protection. In the discussion, we provide an outlook by addressing advances in educational content and educational technology and identifying three crucial open questions for research and practice.

THE TRANSITIVITY SYSTEM ANALYSIS ON BANGTAN SONYEONDAN’S (BTS) SPEECH TEXT

The Love Myself Campaign, a collaborative project between BTS and UNICEF, is a campaign that voices the importance of loving yourself. As part of the campaign, BTS gave a speech at the 73rd UN General Assembly on September 28, 2018. The researcher used the speech text in this campaign as data in literature research, which was analyzed using a descriptive qualitative method. The study aims to discover how mental health awareness is conveyed through the transitivity system. Systemic Functional Linguistics by M.A.K. Halliday is used as a theory to analyze data, more precisely, the transitivity system in ideological meta functions. This meta function sees language as a symbol of human experience in the real world and the mind. Meanwhile, transitivity is a system that represents human experience as a process. The researcher found three dominators: the mental process (20) showed that the speaker tried to motivate the audience through his or her personal feelings and experiences; the first person "I" (42) showed that the speaker made himself the central subject of the speech. The non-circumstance clause (53) showed that the speech was global. Through these results, it can be concluded that the soft campaign method conveys the concept of mental health awareness in speech.

TelMedAI

Telemedicine is the practice of technology-enabled remote communication between patient and doctor. This phenomenon in healthcare has the potential to make services affordable and save time and money. Besides telemedicine allows care givers and family members to join conversations with doctors. Indian government initiated the National Telemedicine Network (NTN) to serve remote areas in healthcare by integrating existing healthcare facilities.Literature has revealed that existing works lack in an integrated approach for patient speech translation in language-independent fashion and automatic detection of disease and symptoms based on speech.There is a need for an automated system using Artificial Intelligence (AI) to recognize patient's speech and identify symptoms based on given audio description. We proposed a framework known as TelMedAI which is designed to recognize patient speech to comprehend disease symptoms besides converting the speech text into desired language. The framework is useful for realizing a telemedicine system. Speech to Speech (STS) module takes the patient's audio content into English audio. STS module exploits the Bi-LSTM model with an encoder, decoder and attention mechanism for translation. Then Google Speech API is used to convert English audio into English text. Then the framework exploits Natural Language Processing (NLP) to improve the quality of text. Afterwards, the disease and symptoms miner module eventually recognizes a list of diseases and corresponding symptoms. We proposed an algorithm known as Learning based Disease and Symptom Recognition from Patient Speech (LbDSRPS). This algorithm has the functionality to develop TelMedAI which helps doctors in telemedicine. Our empirical study has revealed that TelMedAI takes technology-driven telemedicine research forward significantly. The highest accuracy achieved by the proposed framework is 68.13% which is much better than the baseline LSTM model used for voice translation.

The development of synthetic child speech in three South African languages

It is well-known that children with expressive communication difficulties have the right to communicate, but they should also have the right to do so in whichever language they choose, with a voice that closely matches their age, gender, and dialect. This study aimed to develop naturalistic synthetic child speech, matching the vocal identity of three children with expressive communication difficulties, using Tacotron 2, for three under-resourced South African languages, namely South African English (SAE), Afrikaans, and isiXhosa. Due to the scarcity of child speech corpora, 2 hours of child speech data per child was collected from three 11- to 12-year-old children. Two adult models were used to “warm start” the child speech synthesis. To determine the naturalness of the synthetic voices, 124 listeners participated in a mean opinion score survey (Likert Score) and optionally gave qualitative feedback. Despite limited training data used in this study, we successfully developed a synthesized child voice of adequate quality in each language. This study highlights that with recent technological advancements, it is possible to develop synthetic child speech that matches the vocal identity of a child with expressive communication difficulties in different under-resourced languages.