Speech Input Research Articles

Precision Adaptive MFCC Based on R2SDF-FFT and Approximate Computing for Low-Power Speech Keywords Recognition

This paper presents system-architecture-circuits co-designs for computing the MFCC feature extraction for speech keywords recognition. The trade-off between accuracy and power consumption under various background noises is achieved by using the 8-stage radix-2 single-path delay feedback FFT (R2SDF-FFT) and the precision self-adaptive architecture with approximate computing. The R2SDF-FFT structure with the fine-grained bit-width quantization can reduce 35.7% of memory size. Approximate multiplication and addition with Dual-Vdd are proposed to further improve the FFT computing energy efficiency. Finally, we present the precision self-adaptive MFCC architecture with the proposed FFT, which can be dynamically configured to use two calculation modes with different hardware settings according to the input speech background noise. Implemented and evaluated under 22 nm technology, the power consumption of the proposed design can be reduced by up to 76.3%, while the accuracy increased by 0.8%.

Listening to a Dysphonic Speaker in Noise May Impede Children's Spoken Language Processing in a Realistic Classroom Setting.

Purpose The aim of this study was to investigate children's processing of dysphonic speech in a realistic classroom setting, under the influence of added classroom noise. Method Typically developing 6-year-old primary school children performed two listening tasks in their regular classrooms: a phoneme discrimination task to assess speech perception and a sentence-picture matching task to assess listening comprehension. Speech stimuli were played back in either a typical or an impaired voice quality. Children performed the tasks in the presence of induced classroom noise at signal-to-noise ratios between +2 and +9 dB. Results Children's performance in the phoneme discrimination task decreased significantly when the speaker's voice was impaired. The effect of voice quality on sentence-picture matching depended on task demands: Easy sentences were processed more accurately in the impaired-voice condition than in the typical-voice condition. Signal-to-noise ratio effects are discussed in light of methodological constraints. Conclusions Listening to a dysphonic teacher in a noisy classroom may impede children's perception of speech, particularly when phonological discrimination is needed to disambiguate the speech input. Future research regarding the interaction of voice quality and task demands is necessary.

Modeling the Influence of Language Input Statistics on Children's Speech Production.

We trained a computational model (the Chunk-Based Learner; CBL) on a longitudinal corpus of child-caregiver interactions in English to test whether one proposed statistical learning mechanism-backward transitional probability-is able to predict children's speech productions with stable accuracy throughout the first few years of development. We predicted that the model less accurately reconstructs children's speech productions as they grow older because children gradually begin to generate speech using abstracted forms rather than specific "chunks" from their speech environment. To test this idea, we trained the model on both recently encountered and cumulative speech input from a longitudinal child language corpus. We then assessed whether the model could accurately reconstruct children's speech. Controlling for utterance length and the presence of duplicate chunks, we found no evidence that the CBL becomes less accurate in its ability to reconstruct children's speech with age.

Acoustic features of infant-directed speech to infants with hearing loss.

This study investigated the effects of hearing loss and hearing experience on the acoustic features of infant-directed speech (IDS) to infants with hearing loss (HL) compared to controls with normal hearing (NH) matched by either chronological or hearing age (experiment 1) and across development in infants with hearing loss as well as the relation between IDS features and infants' developing lexical abilities (experiment 2). Both experiments included detailed acoustic analyses of mothers' productions of the three corner vowels /a, i, u/ and utterance-level pitch in IDS and in adult-directed speech. Experiment 1 demonstrated that IDS to infants with HL was acoustically more variable than IDS to hearing-age matched infants with NH. Experiment 2 yielded no changes in IDS features over development; however, the results did show a positive relationship between formant distances in mothers' speech and infants' concurrent receptive vocabulary size, as well as between vowel hyperarticulation and infants' expressive vocabulary. These findings suggest that despite infants' HL and thus diminished access to speech input, infants with HL are exposed to IDS with generally similar acoustic qualities as are infants with NH. However, some differences persist, indicating that infants with HL might receive less intelligible speech.

A Live Speech-Driven Avatar-Mediated Three-Party Telepresence System: Design and Evaluation

AbstractIn this article, we present a live speech-driven, avatar-mediated, three-party telepresence system, through which three distant users, embodied as avatars in a shared 3D virtual world, can perform natural three-party telepresence that does not require tracking devices. Based on live speech input from three users, this system can real-time generate the corresponding conversational motions of all the avatars, including head motion, eye motion, lip movement, torso motion, and hand gesture. All motions are generated automatically at each user side based on live speech input, and a cloud server is utilized to transmit and synchronize motion and speech among different users. We conduct a formal user study to evaluate the usability and effectiveness of the system by comparing it with a well-known online virtual world, Second Life, and a widely-used online teleconferencing system, Skype. The user study results indicate our system can provide a measurably better telepresence user experience than the two widely-used methods.

Photorealistic Audio-driven Video Portraits.

Video portraits are common in a variety of applications, such as videoconferencing, news broadcasting, and virtual education and training. We present a novel method to synthesize photorealistic video portraits for an input portrait video, automatically driven by a person's voice. The main challenge in this task is the hallucination of plausible, photorealistic facial expressions from input speech audio. To address this challenge, we employ a parametric 3D face model represented by geometry, facial expression, illumination, etc., and learn a mapping from audio features to model parameters. The input source audio is first represented as a high-dimensional feature, which is used to predict facial expression parameters of the 3D face model. We then replace the expression parameters computed from the original target video with the predicted one, and rerender the reenacted face. Finally, we generate a photorealistic video portrait from the reenacted synthetic face sequence via a neural face renderer. One appealing feature of our approach is the generalization capability for various input speech audio, including synthetic speech audio from text-to-speech software. Extensive experimental results show that our approach outperforms previous general-purpose audio-driven video portrait methods. This includes a user study demonstrating that our results are rated as more realistic than previous methods.

Gated Recurrent Fusion With Joint Training Framework for Robust End-to-End Speech Recognition

The joint training framework for speech enhancement and recognition methods have obtained quite good performances for robust end-to-end automatic speech recognition (ASR). However, these methods only utilize the enhanced feature as the input of the speech recognition component, which are affected by the speech distortion problem. In order to address this problem, this paper proposes a gated recurrent fusion (GRF) method with joint training framework for robust end-to-end ASR. The GRF algorithm is used to dynamically combine the noisy and enhanced features. Therefore, the GRF can not only remove the noise signals from the enhanced features, but also learn the raw fine structures from the noisy features so that it can alleviate the speech distortion. The proposed method consists of speech enhancement, GRF and speech recognition. Firstly, the mask based speech enhancement network is applied to enhance the input speech. Secondly, the GRF is applied to address the speech distortion problem. Thirdly, to improve the performance of ASR, the state-of-the-art speech transformer algorithm is used as the speech recognition component. Finally, the joint training framework is utilized to optimize these three components, simultaneously. Our experiments are conducted on an open-source Mandarin speech corpus called AISHELL-1. Experimental results show that the proposed method achieves the relative character error rate (CER) reduction of 10.04% over the conventional joint enhancement and transformer method only using the enhanced features. Especially for the low signal-to-noise ratio (0 dB), our proposed method can achieves better performances with 12.67% CER reduction, which suggests the potential of our proposed method.

Design of Low Power Speech Processor-Based Cochlear Implants Using Modified FIR Filter with Variable Frequency Mapping

Generally, cochlear implant system consists of one or more electrodes which directly activate the auditory nerve. The microphone transforms external speech processor input into stimuli for each electrode and transcutaneous the connection between the electrodes and processor. As a result, cochlear implants continue to improve the performance of speech processor. After the implantation, preservation gains importance in acoustic hearing to prevent the neural degradation from the loss of cells. Eventually, the possibility of preserving low-frequency and high-frequency hearing using modified surgical technique has been explored for hearing in cochlear implantation. The primary focus on the architecture is to reduce the major potential thread performance and power trade-off. So, frequency mapping function with modified FIR filter has been proposed. For this implementation, the values relative to the frequency range defines upper and lower stimulus limits of each electrode. The proposed FIR filter is designed with distributed arithmetic function where the reconfigurable FIR filter can change the filter input coefficients. By scaling both the frequency ranges and by assessing the deriving frequency parameters, the mapping function will be yielded. Mapping function is manipulated to rectify any mapping deviations. This results in favor of the development of frequency specific mapping function with flexible system. From the proposed results, it is evident that the speech recognition performance is improved significantly by better minimization of overall energy consumption based on effective and pipelined FFT architecture. This implementation also reduces the power and improves the accuracy by 90% and above for different frequency ranges.

Preschoolers benefit from a clear sound-referent mapping to acquire nonnative phonology

AbstractPrevious studies have shown that visual information is a crucial input in early language learning. In the present study we examine what type of visual input helps preschoolers in acquiring nonnative phonological contrasts. Catalan/Spanish-speaking children (4–5 years, N = 47) participated in a task to assess their phonological discrimination abilities before and after a training. Three training conditions were presented: one with clear oral/visual speech information, one with an ostensive object-sound mapping, and one with a rich social interaction. Children’s looking patterns were tracked to examine their focus of interest while being trained. Results revealed that preschoolers’ discrimination abilities increase in all trained conditions, but the condition where the speaker created an ostensive object–sound mapping led to higher long-term gains (especially for younger children). Eye-tracking results further showed that children looked to the object of reference while being exposed to the novel phonological input, which may explain the higher learning gains in this condition. Our results indicate that preschoolers’ learning of nonnative phonological contrasts is particularly boosted when the speech input is accompanied by an object of reference that is signaled ostensively and contingently in the visual space, compared to when the visual space only contains clear oral/visual speech information or social interactivity cues.

Investigation on Large Vocabulary Continuous Kannada Speech Recognition

The proposed work carried out to convert the Kannada speech signal to a text document using Sphinx tools. The system is implemented for the large vocabulary of the Kannada speech signal by using the acoustic model (AM) and language model (LM) with the decoder. The AM extracted the Mel frequency cepstral coefficients (MFCC) of the speech signal successfully and trained these coefficients using hidden Markov model (HMM) and tweak the estimation of the AM using Baum-Welch method. The required language model (LM) format is built for the decoder using n-gram count. The decoder is configured to create a text file to the corresponding input speech signal using Sphinx3. The proposed automatic speech recognition (ASR) system achieves a better recognition rate with less word error rate (WER) with AM and LM adaptation using Sphinx3 for large vocabulary and Pocket Sphinx implemented on RasberryPi3 results better accuracy than the other ASR system.

Rapid computations of spectrotemporal prediction error support perception of degraded speech.

Human speech perception can be described as Bayesian perceptual inference but how are these Bayesian computations instantiated neurally? We used magnetoencephalographic recordings of brain responses to degraded spoken words and experimentally manipulated signal quality and prior knowledge. We first demonstrate that spectrotemporal modulations in speech are more strongly represented in neural responses than alternative speech representations (e.g. spectrogram or articulatory features). Critically, we found an interaction between speech signal quality and expectations from prior written text on the quality of neural representations; increased signal quality enhanced neural representations of speech that mismatched with prior expectations, but led to greater suppression of speech that matched prior expectations. This interaction is a unique neural signature of prediction error computations and is apparent in neural responses within 100 ms of speech input. Our findings contribute to the detailed specification of a computational model of speech perception based on predictive coding frameworks.

Object selection and scaling using multimodal interaction in mixed reality

Mixed Reality (MR) is the next evolution of human interacting with the computer as MR has the ability to combine the physical environment and digital environment and making them coexist with each other. Interaction is still a huge research area in Augmented Reality (AR) but very less in MR, this is due to current advanced MR display techniques still not robust and intuitive enough to let the user to naturally interact with 3D content. New techniques on user interaction have been widely studied, the advanced technique in interaction when the system able to invoke more than one input modalities. Multimodal interaction undertakes to deliver intuitive multiple objects manipulation with gestures. This paper discusses the multimodal interaction technique using gesture and speech which the proposed experimental setup to implement multimodal in the MR interface. The real hand gesture is combined with speech inputs in MR to perform spatial object manipulations. The paper explains the implementation stage that involves interaction using gesture and speech inputs to enhance user experience in MR workspace. After acquiring gesture input and speech commands, spatial manipulation for selection and scaling using multimodal interaction has been invoked, and this paper ends with a discussion.

AR-TO-KID: A speech-enabled augmented reality to engage preschool children in pronunciation learning

AR-TO-KID is an application produced for preschool children between ages five to six years old with an Augmented Reality (AR) application. The significant purpose of AR-TO-KID is to improve the pronunciation of the children in English. Hence, this paper discusses an AR application with speech input. The detection of the children speech input when they need to pronounce the words correctly, and they need to have critical thinking to identify the environment suit with the 3D objects that they will utter the word. Educational technology should be interactive and attractive for 5 to 6 years old preschool children learning; however, some at preschool teachers still used the conventional methods in teaching and children are not fully engaged with the method. Therefore, this project is to design and develop an interactive AR tool called AR-TO-KID for preschool children in pronunciation learning and teaching. This paper presents the evaluation and testing for preschool children with non-native English speaking. The article ends with results and discussion.

Intelligent Personal Assistant in Business-Context: Key-feature Evaluation for User Acceptance

Abstract Background The usage of intelligent personal assistants (IPA), such as Amazon Alexa or Google Assistant is increasing significantly, and voice-interaction is relevant for workflows in a business context. Objectives This research aims to determine IPA characteristics to evaluate the usefulness of specific functions in a simulated production system of an Enterprise Resource Planning (ERP) software. A new function called explanation-mode is introduced to the scientific community and business world. Methods/Approach As part of a design science research, an artefact, i.e. an add-on for speech-interaction in business software, was developed and evaluated using a survey among ERP users and researchers. Results In the area of IPA-features, the search-function and speech input for textual fields were recognised as most useful. The newly introduced feature, the explanation mode, was positively received too. There is no significant correlation between the usefulness of features and participant-characteristics, affinity to technology or previous experience with IPAs in a private context, which is in line with previous studies in the private environment leading to the conclusion that the task attraction is the most important element for usefulness. Conclusions Most of the participants agreed that the speech-input is not able to fully substitute standard input devices, such as a keyboard or a mouse, so the IPA is recognised as an addition to traditional input methods. The usefulness is rated high especially for speech-input for long text fields, calling up masks and search-functions.

NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries.

Natural language interfaces (NLls) have shown great promise for visual data analysis, allowing people to flexibly specify and interact with visualizations. However, developing visualization NLIs remains a challenging task, requiring low-level implementation of natural language processing (NLP) techniques as well as knowledge of visual analytic tasks and visualization design. We present NL4DV, a toolkit for natural language-driven data visualization. NL4DV is a Python package that takes as input a tabular dataset and a natural language query about that dataset. In response, the toolkit returns an analytic specification modeled as a JSON object containing data attributes, analytic tasks, and a list of Vega-Lite specifications relevant to the input query. In doing so, NL4DV aids visualization developers who may not have a background in NLP, enabling them to create new visualization NLIs or incorporate natural language input within their existing systems. We demonstrate NL4DV's usage and capabilities through four examples: 1) rendering visualizations using natural language in a Jupyter notebook, 2) developing a NLI to specify and edit Vega-Lite charts, 3) recreating data ambiguity widgets from the DataTone system, and 4) incorporating speech input to create a multimodal visualization system.

Comparisons of performance in pediatric bone conduction implant recipients using remote microphone technology

ObjectivesFor children with hearing loss, remote microphone (RM) technology can significantly improve access to speech in environments with poor signal-to-noise ratios (SNRs), such as classrooms. Yet, this has never been studied in bone conduction device (BCD) users, a common treatment for children with irresolvable conductive hearing loss resulting from anatomical malformations of the outer ear. The objective of this study was to investigate the benefits of RM technology on speech perception in noise in pediatric BCD users with Microtia/Atresia. A secondary aim was to assess parent and child perceptions of RM technology before and after exposure to RM technology. MethodsParticipants included 10 pediatric bone conduction implant users with unilateral conductive hearing loss ages 7–17 years, and their guardians. Speech perception in noise for soft and moderate inputs was assessed with and without RM technology. Guardians actively observed the child's hearing performance with and without the RM and were asked to complete a questionnaire assessing their perceptions about their child's performance. Children were also administered the questionnaire prior to and immediately following exposure to the RM technology. ResultsParticipants showed improved speech understanding in noise for both soft and moderate speech inputs when using the RM with their BCD compared to their BCD alone. Questionnaire results indicated good parent-child agreement. Further, significant improvements were reported for child understanding speech, parent hearing domains after exposure to the RM. No significant differences were noted for ease of use/likability. ConclusionsSignificant hearing in noise benefits were observed with RM technology for children using BCDs. Consistent with objective findings, children reported improved speech understanding with the RM. Improved parental perceptions of hearing benefit following exposure to the RM suggests that active participation may serve as an effective strategy to help improve parent understanding of the benefits of RM technology for their child.

A switched chaotic encryption scheme using multi-mode generalized modified transition map

This paper presents a multi-mode generalized modified transition chaotic map and a switched chaotic encryption scheme based on it. Eight different modes of operation can be selected based on the map graph (concave or convex), the range modification procedure (shrinking or widening) and the sign of one of its independent parameters. The generalization and modification preserve the controllability and continuous chaotic behavior properties, respectively. For the same encryption key and map equation, multi-mode operation occurs through switching between four alternatives of the dependent parameters. Switching between various modes is shown to overpass single modes in randomness by investigating the frequency distribution, autocorrelation function, histograms and 0-1 test. In addition, most bits, 75%, of the scaled and quantized chaotic output are shown to be suitable for pseudo-random number generation based on National Institute of Standards & Technology (NIST) statistical test suite. The performance of the proposed scheme is validated for several speech, grayscale and RGB images through various evaluation criteria, where good results are achieved. These criteria include basic perceptual and statistical aspects, the more advanced NIST tests, key sensitivity and robustness against brute force and differential attacks. The efficiency of the scheme is improved by up to three or six times faster than encrypting one sample or one channel of a pixel at a time for speech and image inputs, respectively. The proposed scheme demonstrates good security and efficiency with more applied evaluation criteria compared to similar recent works.

Speech and Language Skills of Low-Risk Preterm and Full-Term Late Talkers: The Role of Child Factors and Parent Input.

Among children in the third year of life, late talkers comprise from 9% to 20%. This range seems to increase when addressing preterm children. This study examined video-recorded child spontaneous speech during parent–child book sharing as well as linguistic skills reported through the MacArthur Bates Communicative Development Inventories (MB-CDI) Short Form in 61 late talkers aged 30 months old (26 low-risk preterm, 8 females; 35 full-term, 12 females). Differences between low-risk preterm and full-term late talkers in child language measures and parental speech input were tested, as were the roles of child and parent factors on child language. Low-risk preterm and full-term late talkers showed similar speech and language skills. Similarly, no differences were found in measures of parental speech between groups. Child cognitive score, chronological age, and low-risk preterm status were positively associated with lexical diversity, rate, and composition of child speech production, whereas family history for language and/or learning disorders as well as parent measures of lexical diversity, rate, and grammatical complexity were negatively associated with the above child variables. In addition, child cognitive score and low-risk preterm status were positively associated with the MB-CDI measures of word and sentence production. Findings are discussed in terms of the need of good practices when following up on low-risk preterm children and of interventions targeting parents’ input to preterm and full-term late talkers.

Effect of Various Visual Speech Units on Language Identification Using Visual Speech Recognition

In this paper, we describe our work in Spoken language Identification using Visual Speech Recognition (VSR) and analyze the effect of various visual speech units used to transcribe the visual speech on language recognition. We have proposed a new approach of word recognition followed by the word N-gram language model (WRWLM), which uses high-level syntactic features and the word bigram language model for language discrimination. Also, as opposed to the traditional visemic approach, we propose a holistic approach of using the signature of a whole word, referred to as a “Visual Word” as visual speech unit for transcribing visual speech. The result shows Word Recognition Rate (WRR) of 88% and Language Recognition Rate (LRR) of 94% in speaker dependent cases and 58% WRR and 77% LRR in speaker independent cases for English and Marathi digit classification task. The proposed approach is also evaluated for continuous speech input. The result shows that the Spoken Language Identification rate of 50% is possible even though the WRR using Visual Speech Recognition is below 10%, using only 1[Formula: see text]s of speech. Also, there is an improvement of about 5% in language discrimination as compared to traditional visemic approaches.

The Influence of Form- and Meaning-Based Predictions on Cortical Speech Processing Under Challenging Listening Conditions: A MEG Study.

Under adverse listening conditions, prior linguistic knowledge about the form (i.e., phonology) and meaning (i.e., semantics) help us to predict what an interlocutor is about to say. Previous research has shown that accurate predictions of incoming speech increase speech intelligibility, and that semantic predictions enhance the perceptual clarity of degraded speech even when exact phonological predictions are possible. In addition, working memory (WM) is thought to have specific influence over anticipatory mechanisms by actively maintaining and updating the relevance of predicted vs. unpredicted speech inputs. However, the relative impact on speech processing of deviations from expectations related to form and meaning is incompletely understood. Here, we use MEG to investigate the cortical temporal processing of deviations from the expected form and meaning of final words during sentence processing. Our overall aim was to observe how deviations from the expected form and meaning modulate cortical speech processing under adverse listening conditions and investigate the degree to which this is associated with WM capacity. Results indicated that different types of deviations are processed differently in the auditory N400 and Mismatch Negativity (MMN) components. In particular, MMN was sensitive to the type of deviation (form or meaning) whereas the N400 was sensitive to the magnitude of the deviation rather than its type. WM capacity was associated with the ability to process phonological incoming information and semantic integration.