Speech Feedback Research Articles

Convolutional Neural Networks to Facilitate the Continuous Recognition of Arabic Speech with Independent Speakers

Automatic speech recognition (ASR) is a field of research that focuses on the ability of computers to process and interpret speech feedback from humans and to provide the highest degree of accuracy in recognition. Speech is one of the simplest ways to convey a message in a basic context, and ASR refers to the ability of machines to process and accept speech data from humans with the greatest degree of accuracy. As the human-to-machine interface continues to evolve, speech recognition is expected to become increasingly important. However, the Arabic language has distinct features that set it apart from other languages, such as the dialect and the pronunciation of words. Until now, insufficient attention has been devoted to continuous Arabic speech recognition research for independent speakers with a limited database. This research proposed two techniques for the recognition of Arabic speech. The first uses a combination of convolutional neural network (CNN) and long short-term memory (LSTM) encoders, and an attention-based decoder, and the second is based on the Sphinx-4 recognizer, which includes pocket sphinx, base sphinx, and sphinx train, with various types and number of features to be extracted (filter bank and mel frequency cepstral coefficients (MFCC)) based on the CMU Sphinx tool, which generates a language model for different sentences spoken by different speakers. These approaches were tested on a dataset containing 7 hours of spoken Arabic from 11 Arab countries, covering the Levant, Gulf, and African regions, which make up the Arab world, and achieved promising results. CNN-LSTM achieved a word error rate (WER) of 3.63% using 120 features for filter bank and 4.04% WER using 39 features for MFCC, respectively, while the Sphinx-4 recognizer technique achieved 8.17% WER and an accuracy of 91.83% using 25 features for MFCC and 8 Gaussian mixtures, respectively, when tested on the same benchmark dataset.

Design and Implementation of Smart Blind Stick for Obstacle Detection and Navigation System

The design, advancement, and evaluation of an IoT -enabled smart stick that can detect and alert users to obstacles is presented in this study to help visually impaired persons traverse the outside world. A water sensor deployed to identify puddles and other wet surfaces in the user's route, and ultrasonic sensors for detecting obstacles are all used in the suggested design. Additionally, after correctly detecting and recognising items, the user receives voice feedback through earphones regarding various obstacles and objects. The suggested smart stick has two modes: the first detects and recognises obstacles and provides speech feedback; the second mode employs ultrasonic sensors for detecting and response through vibration motors to inform about the direction of the obstacle. The suggested method enables users to alternate between the two modes based on their preferences and the surroundings. Additionally, the user's latitude and longitude values are recorded and uploaded to the IoT platform for efficient tracking through global system for wireless communication modules, allowing the user or device to be tracked in real time on the dashboard. An IoT-enabled environment generates an alert signal in the form of an SMS with Google Maps link generated using latitude and longitude coordinates, and this serves as a panic button for emergency assistance. The smart stick's design includes features such a long battery life, size adjustment, waterproofing, and light weight. Robust features, portability, stability, ease of access, and energy efficiency are all guaranteed by the entire design.

Developmental stuttering may be caused by insufficient processing of auditory feedback

Developmental stuttering has been scientifically researched for about a hundred years, but the cause is still unknown. Here, a comprehensive causal hypothesis is presented, from the factors contributing to a predisposition for stuttering to the mechanism underlying the core symptoms: A deficit in attention regulation and/or auditory processing abets a misallocation of attention, i.e., of perceptual and processing capacity, during speech; this results in poor processing of the auditory feedback of speech. Insufficiently processed auditory feedback causes error signals in the speech network which, by an error-induced interaction between cerebellum and basal ganglia, interrupts the speech flow against the speaker’s will. The allocation of attention during speech is hypothesized to be a variable state that forms the interface between the physiological pathomechanism of stuttering, on the one hand, and situational, cognitive, and emotional factors influencing stuttering severity, on the other hand; the interaction between both accounts for the situational variability of the disorder. The hypothesis implicates that increased attention to the auditory feedback of speech (listening to one’s voice) reduces stuttering – this opens, both, a way to test the hypothesis and to improve the therapy.

Spectral Characteristics of Cardinal Vowels as Indicators of the Auditory Speech Feedback Control in Patients with Moderate and Moderately Severe Chronic Postlingual Sensorineural Hearing Loss

Chronic sensorineural hearing loss (SNHL) is characterized by an increase in hearing thresholds at basic speech frequencies, which implies the auditory speech feedback control worsening and, as a result, changes of speech characteristics. A hypothesis was tested that such worsening can manifest itself in an increase of F0, F1, F2 formants of speech vowel sounds in patients with moderate and moderately severe postlingual SNHL. Recordings of elicited speech were performed for young and middle age women (36–59 years): 7 women speakers with moderate SNHL who did not use hearing aids; 5 women speakers with moderately severe SNHL who were hearing aid users but were not using them during the recordings; a control group of 12 normally hearing women speakers. An assessment of F0, F1 and F2 of stressed vowels [a], [i], [u] and calculations of vowels’ centralization indices – vowel space area, vowel formant centralization ratio and the second formant ratio (F2i/F2u), were performed. All the studied spectral indices in groups of patients with postlingual SNHL were similar to those in the control group, no statistically reliable differences were revealed.

Spectral Characteristics of Cardinal Vowels as Indicators of the Auditory Speech Feedback Control in Patients with Moderate and Moderately Severe Chronic Postlingual Sensorineural Hearing Loss

Spectral Characteristics of Cardinal Vowels as Indicators of the Auditory Speech Feedback Control in Patients with Moderate and Moderately Severe Chronic Postlingual Sensorineural Hearing Loss

An IoT and Wearables-Based Smart Home for ALS Patients

In recent years, assistive wearables technologies based on Internet of Things (IoT) platforms for amyotrophic lateral sclerosis (ALS) patients trigger broad interests. Nevertheless, the user privacy leakage issue, owing to the scene camera installed on wearables to analyze environmental information, hinders further success use for ALS patients. To address this issue, in this article, a smart human-environment interactive (HEI) environment, including eye motion detection, radio-frequency identification (RFID), and speech feedback techniques, under the IoT framework is presented. Here, the users’ intentions are first interpreted by the eye motion classification, and then the target smart devices are reported and desired operations are confirmed by the RFID and speech feedback system in a hand-shaking manner. A high average accuracy of 93.2% is experimentally achieved, demonstrating the feasibility of the proposed method in obtaining satisfying performance while avoiding potential privacy leakage.

"Baby, You can Ride my Bike"

We envision a future where self-driving bicycles can take us to our destinations. This allows cyclists to use their time on the bike efficiently for work or relaxation without having to focus their attention on traffic. In the related field of self-driving cars, research has shown that communicating the planned route to passengers plays an important role in building trust in automation and situational awareness. For self-driving bicycles, this information transfer will be even more important, as riders will need to actively compensate for the movement of a self-driving bicycle to maintain balance. In this paper, we investigate maneuver indications for self-driving bicycles: (1) ambient light in a helmet, (2) head-up display indications, (3) speech feedback, (4) vibration on the handlebar, and (5) no assistance. To evaluate these indications, we conducted an outdoor experiment (N = 25) in a proposed tandem simulator consisting of a tandem bicycle with a steering and braking control on the back seat and a rider in full control of it. Our results indicate that riders respond faster to visual cues and focus comparably on the reading task while riding with and without maneuver indications. Additionally, we found that the tandem simulator is realistic, safe, and creates an awareness of a human cyclist controlling the tandem.

Perturbing the consistency of auditory feedback in speech.

Sensory information, including auditory feedback, is used by talkers to maintain fluent speech articulation. Current models of speech motor control posit that speakers continually adjust their motor commands based on discrepancies between the sensory predictions made by a forward model and the sensory consequences of their speech movements. Here, in two within-subject design experiments, we used a real-time formant manipulation system to explore how reliant speech articulation is on the accuracy or predictability of auditory feedback information. This involved introducing random formant perturbations during vowel production that varied systematically in their spatial location in formant space (Experiment 1) and temporal consistency (Experiment 2). Our results indicate that, on average, speakers’ responses to auditory feedback manipulations varied based on the relevance and degree of the error that was introduced in the various feedback conditions. In Experiment 1, speakers’ average production was not reliably influenced by random perturbations that were introduced every utterance to the first (F1) and second (F2) formants in various locations of formant space that had an overall average of 0 Hz. However, when perturbations were applied that had a mean of +100 Hz in F1 and −125 Hz in F2, speakers demonstrated reliable compensatory responses that reflected the average magnitude of the applied perturbations. In Experiment 2, speakers did not significantly compensate for perturbations of varying magnitudes that were held constant for one and three trials at a time. Speakers’ average productions did, however, significantly deviate from a control condition when perturbations were held constant for six trials. Within the context of these conditions, our findings provide evidence that the control of speech movements is, at least in part, dependent upon the reliability and stability of the sensory information that it receives over time.

Haptic Feedback to Assist Blind People in Indoor Environment Using Vibration Patterns.

Feedback is one of the significant factors for the mental mapping of an environment. It is the communication of spatial information to blind people to perceive the surroundings. The assistive smartphone technologies deliver feedback for different activities using several feedback mediums, including voice, sonification and vibration. Researchers 0have proposed various solutions for conveying feedback messages to blind people using these mediums. Voice and sonification feedback are effective solutions to convey information. However, these solutions are not applicable in a noisy environment and may occupy the most important auditory sense. The privacy of a blind user can also be compromised with speech feedback. The vibration feedback could effectively be used as an alternative approach to these mediums. This paper proposes a real-time feedback system specifically designed for blind people to convey information to them based on vibration patterns. The proposed solution has been evaluated through an empirical study by collecting data from 24 blind people through a mixed-mode survey using a questionnaire. Results show the average recognition accuracy for 10 different vibration patterns are 90%, 82%, 75%, 87%, 65%, and 70%.

Visually Impaired Person Assistance Based on Tensor FlowLite Technology

The most exciting thing about computer visualization is to detect a Real time object application system. This is abundantly used in many areas. With the more increase of development of deep learning such as self-driving cars, robots, safety tracking, and guiding visually impaired people, many algorithms have improved to find the relationship between video analysis and images analysis. Entire algorithms behave uniquely in the network architecture, and they have the same goal of detecting numerous objects in a composite image. It is very important to use our technology to train visually impaired people whenever they need them, as they are visually impaired and limit the movement of people in unknown places. This paper offers an application system that will identify all the possible day-to-day objects of our surroundings, and on the other side, it promotes speech feedback to the person about the sudden as well as far objects around them. This project was advanced using two different algorithms: Yolo and Yolo-v3, tested to the same criteria to measure its accuracy and performance. The SSD_MobileNet model is used in Yolo Tensor Flow and the Darknet model is used in Yolo_v3. Speech feedback: A Python library incorporated to convert statements to speech-to-speech. Both algorithms are analyzed using a web camera in a variety of circumstances to measure the correctness of the algorithm in every aspect.

Detection of Touchscreen-Based Urdu Braille Characters Using Machine Learning Techniques

Revolution in technology is changing the way visually impaired people read and write Braille easily. Learning Braille in its native language can be more convenient for its users. This study proposes an improved backend processing algorithm for an earlier developed touchscreen-based Braille text entry application. This application is used to collect Urdu Braille data, which is then converted to Urdu text. Braille to text conversion has been done on Hindi, Arabic, Bangla, Chinese, English, and other languages. For this study, Urdu Braille Grade 1 data were collected with multiclass (39 characters of Urdu represented by class 1, Alif (ﺍ), to class 39, Bri Yay (ے). Total (N = 144) cases for each class were collected. The dataset was collected from visually impaired students from The National Special Education School. Visually impaired users entered the Urdu Braille alphabets using touchscreen devices. The final dataset contained (N = 5638) cases. Reconstruction Independent Component Analysis (RICA)-based feature extraction model is created for Braille to Urdu text classification. The multiclass was categorized into three groups (13 each), i.e., category-1 (1–13), Alif-Zaal (ﺫ - ﺍ), category-2 (14–26), Ray-Fay (ﻒ - ﺮ), and category-3 (27–39), Kaaf-Bri Yay (ے - ﻕ), to give better vision and understanding. The performance was evaluated in terms of true positive rate, true negative rate, positive predictive value, negative predictive value, false positive rate, total accuracy, and area under the receiver operating curve. Among all the classifiers, support vector machine has achieved the highest performance with a 99.73% accuracy. For comparisons, robust machine learning techniques, such as support vector machine, decision tree, and K-nearest neighbors were used. Currently, this work has been done on only Grade 1 Urdu Braille. In the future, we plan to enhance this work using Grade 2 Urdu Braille with text and speech feedback on touchscreen-based android phones.

WristDial: An Eyes-Free Integer-Value Input Method by Quantizing the Wrist Rotation

ABSTRACT Smartwatches are a convenient substitute for smartphones in hand-busy situations, but there is limited interactivity to one-handed use of wrist-worn devices. In this paper, we propose WristDial, a one-handed integer input technique which uses quantization of the angular distance of the wrist rotation with nonvisual feedback. Using tactile feedback, we present four techniques based on counting of tactile stimuli. With one of them, 94.44% average accuracy and 2.81 s average completion time were acquired in the task of entering a number from the range 1 to 10. We also evaluate the quantitative performance of the combination of interruptible speech and non-speech auditory feedback. Our results show that when speech feedback exists, adding a continuous change of pitch or clicking sound at the boundaries does not have a significant effect on the performance. Using only speech feedback, 95.47% accuracy and 1.45 s completion time were acquired.

Optimization of Intelligent English Pronunciation Training System Based on Android Platform

Oral English, as a language tool, is not only an important part of English learning but also an essential part. For nonnative English learners, effective and meaningful voice feedback is very important. At present, most of the traditional recognition and error correction systems for oral English training are still in the theoretical stage. At the same time, the corresponding high-end experimental prototype also has the disadvantages of large and complex system. In the speech recognition technology, the traditional speech recognition technology is not perfect in recognition ability and recognition accuracy, and it relies too much on the recognition of speech content, which is easily affected by the noise environment. Based on this, this paper will develop and design a spoken English assistant pronunciation training system based on Android smartphone platform. Based on the in-depth study and analysis of spoken English speech correction algorithm and speech feedback mechanism, this paper proposes a lip motion judgment algorithm based on ultrasonic detection, which is used to assist the traditional speech recognition algorithm in double feedback judgment. In the feedback mechanism of intelligent speech training, a double benchmark scoring mechanism is introduced to comprehensively evaluate the speech of the speech trainer and correct the speaker’s speech in time. The experimental results show that the speech accuracy of the system reaches 85%, which improves the level of oral English trainers to a certain extent.

Neural Development of Speech Sensorimotor Learning.

The development of the human brain continues through to early adulthood. It has been suggested that cortical plasticity during this protracted period of development shapes circuits in associative transmodal regions of the brain. Here we considered how cortical plasticity during development might contribute to the coordinated brain activity required for speech motor learning. Specifically, we examined patterns of brain functional connectivity (FC), whose strength covaried with the capacity for speech audio-motor adaptation in children ages 5-12 and in young adults of both sexes. Children and adults showed distinct patterns of the encoding of learning in the brain. Adult performance was associated with connectivity in transmodal regions that integrate auditory and somatosensory information, whereas children rely on basic somatosensory and motor circuits. A progressive reliance on transmodal regions is consistent with human cortical development and suggests that human speech motor adaptation abilities are built on cortical remodeling, which is observable in late childhood and is stabilized in adults.SIGNIFICANCE STATEMENT A protracted period of neuro plasticity during human development is associated with extensive reorganization of associative cortex. We examined how the relationship between FC and speech motor learning capacity are reconfigured in conjunction with this cortical reorganization. Young adults and children aged 5-12 years showed distinctly different patterns. Mature brain networks related to learning included associative cortex, which integrates auditory and somatosensory feedback in speech, whereas the immature networks in children included motor regions of the brain. These patterns are consistent with the cortical reorganization that is initiated in late childhood. The result provides insights into the human biology of speech as well as to the mature neural mechanisms for multisensory integration in motor learning.

Error in the Superior Temporal Gyrus? A Systematic Review and Activation Likelihood Estimation Meta-Analysis of Speech Production Studies.

Evidence for perceptual processing in models of speech production is often drawn from investigations in which the sound of a talker's voice is altered in real time to induce "errors." Methods of acoustic manipulation vary but are assumed to engage the same neural network and psychological processes. This paper aims to review fMRI and PET studies of altered auditory feedback and assess the strength of the evidence these studies provide for a speech error correction mechanism. Studies included were functional neuroimaging studies of speech production in neurotypical adult humans, using natural speech errors or one of three predefined speech manipulation techniques (frequency altered feedback, delayed auditory feedback, and masked auditory feedback). Seventeen studies met the inclusion criteria. In a systematic review, we evaluated whether each study (1) used an ecologically valid speech production task, (2) controlled for auditory activation caused by hearing the perturbation, (3) statistically controlled for multiple comparisons, and (4) measured behavioral compensation correlating with perturbation. None of the studies met all four criteria. We then conducted an activation likelihood estimation meta-analysis of brain coordinates from 16 studies that reported brain responses to manipulated over unmanipulated speech feedback, using the GingerALE toolbox. These foci clustered in bilateral superior temporal gyri, anterior to cortical fields typically linked to error correction. Within the limits of our analysis, we conclude that existing neuroimaging evidence is insufficient to determine whether error monitoring occurs in the posterior superior temporal gyrus regions proposed by models of speech production.

Adaptation occurs following a single exposure to altered auditory feedback in speech

Auditory perturbations, or sensory errors caused by manipulating auditory feedback in near real-time, are known to induce instantaneous compensation as well as long-term adaptation responses in speech production. Adaptive responses are built through repetitive exposure to perturbations, but it is unknown how much learning occurs from a single exposure to an auditory perturbation. This trial-by-trial learning, known as one-shot learning, may provide insight into how individual events affect feed-forward control mechanisms and contribute to adaptation. One-shot learning is evident in reaching motor tasks, and the present study aims to understand its role in the speech sensorimotor system. Data were compiled from a set of studies which randomly exposed participants to isolated auditory perturbation events. On any given trial, the participants' first formant (F1) was either shifted up, shifted down, or remained unshifted while they pronounced a presented word. Data from unshifted trials that were preceded by a perturbation trial were extracted to examine if there was a learning effect that carried over from the previous, shifted trial. During unshifted trials that were preceded by a perturbation, participants adjusted their first formant in the opposite direction of the preceding shift, demonstrating that learning occurs even after a single auditory perturbation.

Visual feedback and self-monitoring in speech learning via hand movement

While human speech learning largely relies on perceiving sounds [Brainard and Doupe, Nat. Rev. Neurosci. 1(1), 31–40 (2000)], being able to see articulators also contributes to learning speech sounds [Gick et al. (2008), Phonology Second Language Acquisition 36, 315–328 (2000)]. However, seeing hands is not necessarily helpful for learning new hand movements [Emmorey et al., J. Memory Lang., 61(3), 398–411 (2009)]. This study investigates whether being able to see a hand, acting as a speech articulator, facilitates the learning of speech production via hand movements. Two groups of participants under different visual feedback conditions were asked to produce different vowel sequences via a system that maps hand movements to F1 and F2 of English vowels [Liu et al., Can. Acoust. 48(1) (2020)]. The results suggest that visual feedback contributes to the speed of speech learning and reaching vowel targets more accurately via hand movements. This study provides insight on the importance of visual feedback in monitoring speech, and supports the view that monitoring speech articulators visually can accelerate speech learning.

Stabilizing variability in the auditory feedback of speech

Auditory feedback is an essential part of speech motor control and speech learning. When feedback is perturbed in laboratory settings (e.g., Houde and Jourdan, 1998), speakers, on average, compensate for the perceived error. There is, however, considerable individual variability observed in natural speech and in speakers’ responses to auditory feedback manipulations (e.g., Purcell and Munhall, 2006). Here, we introduce a novel manipulation that stabilized the predictability of auditory feedback of 20 female speakers. Participants produced the English word “head” 95 times in two different conditions. In the Control condition, subjects produced all utterances with unaltered auditory feedback. In the Stabilization condition, subjects were presented with a recording of one of their own utterances of “head” synchronized with their speech on some trials. Auditory feedback was thus made constant by playing the same recording for a set of 30 trials. Trial-to-trial variability of the talkers’ speech did not change as a result of this constant feedback. Time-series analyses were performed to examine whether production variability differed among speakers in the two conditions. Results will be discussed regarding the possible role of variability in speech motor control and the importance of developing methods to detect state-change in individual time-series data.

Hippocampus plays a role in speech feedback processing

There is increasing evidence that the hippocampus is involved in language production and verbal communication, although little is known about its possible role. According to one view, hippocampus contributes semantic memory to spoken language. Alternatively, hippocampus is involved in the processing the (mis)match between expected sensory consequences of speaking and the perceived speech feedback. In the current study, we re-analysed functional magnetic resonance (fMRI) data of two overt picture-naming studies to test whether hippocampus is involved in speech production and, if so, whether the results can distinguish between a “pure memory” versus a “prediction” account of hippocampal involvement. In both studies, participants overtly named pictures during scanning while hearing their own speech feedback unimpededly or impaired by a superimposed noise mask. Results showed decreased hippocampal activity when speech feedback was impaired, compared to when feedback was unimpeded. Further, we found increased functional coupling between auditory cortex and hippocampus during unimpeded speech feedback, compared to impaired feedback. Finally, we found significant functional coupling between a hippocampal/supplementary motor area (SMA) interaction term and auditory cortex, anterior cingulate cortex and cerebellum during overt picture naming, but not during listening to one's own pre-recorded voice. These findings indicate that hippocampus plays a role in speech production that is in accordance with a “prediction” view of hippocampal functioning.

Differential contributions of the two cerebral hemispheres to temporal and spectral speech feedback control

Proper speech production requires auditory speech feedback control. Models of speech production associate this function with the right cerebral hemisphere while the left hemisphere is proposed to host speech motor programs. However, previous studies have investigated only spectral perturbations of the auditory speech feedback. Since auditory perception is known to be lateralized, with right-lateralized analysis of spectral features and left-lateralized processing of temporal features, it is unclear whether the observed right-lateralization of auditory speech feedback processing reflects a preference for speech feedback control or for spectral processing in general. Here we use a behavioral speech adaptation experiment with dichotically presented altered auditory feedback and an analogous fMRI experiment with binaurally presented altered feedback to confirm a right hemisphere preference for spectral feedback control and to reveal a left hemisphere preference for temporal feedback control during speaking. These results indicate that auditory feedback control involves both hemispheres with differential contributions along the spectro-temporal axis.