Human Speech Recognition Research Articles

Modelling Human Word Learning and Recognition Using Visually Grounded Speech

Many computational models of speech recognition assume that the set of target words is already given. This implies that these models learn to recognise speech in a biologically unrealistic manner, i.e. with prior lexical knowledge and explicit supervision. In contrast, visually grounded speech models learn to recognise speech without prior lexical knowledge by exploiting statistical dependencies between spoken and visual input. While it has previously been shown that visually grounded speech models learn to recognise the presence of words in the input, we explicitly investigate such a model as a model of human speech recognition. We investigate the time course of noun and verb recognition as simulated by the model using a gating paradigm to test whether its recognition is affected by well-known word competition effects in human speech processing. We furthermore investigate whether vector quantisation, a technique for discrete representation learning, aids the model in the discovery and recognition of words. Our experiments show that the model is able to recognise nouns in isolation and even learns to properly differentiate between plural and singular nouns. We also find that recognition is influenced by word competition from the word-initial cohort and neighbourhood density, mirroring word competition effects in human speech comprehension. Lastly, we find no evidence that vector quantisation is helpful in discovering and recognising words, though our gating experiment does show that the LSTM-VQ model is able to recognise the target words earlier.

Improving College Assistance with the Help of Richer Human Computer Interaction and Speech Recognition

Abstract: Chat bots are intelligent systems that interpret and react to users questions in their native language. In a conversation, the chat bot reacts in the same way as a human would. It functions as a virtual assistant, and its accuracy is assessed by determining a correlation between user questions and chat bot responses. For a better user experience, the implemented Chat bot has two modes: text mode and audio mode. It provides an interactive approach of answering through voice messages when in audio mode. There is a long line at the inquiry window during the Institute's Academic Admission procedure. Even more challenging is the situation for parents who live in various cities, states, and nations. The purpose of the system is to give students and parents a place to ask questions and get answers using simple English language text messages or audio commands. Instead of queueing at an information desk to ask questions about the admission process students and parents will collaborate with the bot. The proposed application is easily available to students and responds to user any time. Keyword: Chat Bot, College bots, Interactive bots, Dialogue based bots, NLP, Text based bots

Electrospun nanofibrous yarn based piezoresistive flexible strain sensor for human motion detection and speech recognition

This work reports the development of highly sensitive Piezoresistive flexible strain sensor for human motion detection and speech recognition. Initially, a conductive polymer composite (CPC) solution comprising of thermoplastic polyurethane and Carbon Nanoparticles was prepared using Dimethylformamide as solvent and Chloroform as dispersant with the composition of 50% v/v. The solution was heated to a temperature of 60°C for evaporation of the solvent until it contained 13.5% w/v solvent for steady electrospinning. In this way, the CPC solution was used to develop Electrospun Nanofibrous Yarns (ENFYs) by applying a potential difference of 40 KV between the electrospinning needle and Aluminum collector. A cotton fabric was wrapped on the Aluminum collector to allow twisting of the deposited electrospun nanofibers. This novel collector configuration resulted in the formation of nanofibrous yarns due to the whirling action of the advancing jet of CPC solution. The cotton fabric on the collector facilitated twisting of fibers by allowing them to roll over the fabric. The fabricated ENFY sensors showed remarkable stretchability up to 102% strain while achieving a gauge factor of 70 at 100% strain. Long-term usage necessitates repeatability, which was demonstrated by cyclic loading at a crosshead speed of 40 mm/min for up to 1000 cycles using a custom-developed linear actuator, with no signs of fracture. ENFY strain sensor was attached to different parts of human body such as finger, fist, elbow, knee and ankle and was found capable of measuring and observing angle, position and frequency of motion. Owing to its ultrasensitive behavior, the developed sensor was able to measure heart rate as well. When the developed sensor was attached to Adam’s apple for speech recognition it showed remarkable response towards different utterances and breathing and gulping actions with clearly distinguishable signals. These results demonstrate that the developed novel ENFY flexible strain sensor can be employed for proprioceptive sensing and speech recognition for human-machine interaction, soft robotics and wearable devices etc.

An empirical analysis on the efficiency of five interlingual live subtitling workflows

Interlingual Live Subtitling (ILS) is an innovative translation and accessibility method where a written text in one language is produced live from an oral source in another language. ILS can be provided through different methods, some of which involve the participation of one or more humans, whereas others are fully automatic. Speech-to-text interpreting (STTI) is a form of human-mediated ILS that is situated at the crossroads of audiovisual translation, media accessibility and simultaneous interpreting, as well as between human-mediated translation and automatic language processing systems. One of the most promising forms of STTI is interlingual respeaking. It builds upon intralingual respeaking (the most common form of speech-to-text captioning, which does not include language transfer) and involves the participation of a human interpreter plus speech recognition software. Although interlingual respeaking is in great demand, there are other approaches to STTI -with different degrees of human intervention- which are currently being used by broadcasters and conference organizers. The purpose of this research is to test the efficiency of five of those methods, namely, (1) interlingual respeaking, (2) simultaneous interpreting plus intralingual respeaking, (3) simultaneous interpreting plus automatic speech recognition, (4) intralingual respeaking plus machine translation and (5) automatic speech recognition plus machine translation. The results provide a useful insight into the current efficiency of five different ILS methods and strengthen the idea that efficiency is not restricted to accuracy, but includes factors such as delay and the type of resources (either human or machine) required. It is hoped that this research may help provide the industry with tools to make informed choices between different forms of ILS (at least for the language combination English-Spanish) while offering employment opportunities for simultaneous interpreters and respeakers in the digital era.

A Project Based Learning Approach for Improving Students' Computational Thinking Skills.

An educational robotics lab has been planned for undergraduate students in an Electronic Engineering degree, using the Project Based Learning (PBL) approach and the NAO robot. Students worked in a research context, with the aim of making the functions of the NAO robot as social and autonomous as possible, adopting in the design process the Wolfram Language (WL), from the Mathematica software. Interfacing the programming environment of the NAO with Mathematica, they solved in part the problem of autonomy of the NAO, thus realizing enhanced functions of autonomous movement, recognition of human faces and speech for improving the system social interaction. An external repository was created to streamline processes and stow data that the robot can easily access. Self-assessment processes demonstrated that the course provided students with useful skills to cope with real life problems. Cognitive aspects of programming by WL have also been collected in the students’ feedback.

A model of speech recognition for hearing-impaired listeners based on deep learning

Automatic speech recognition (ASR) has made major progress based on deep machine learning, which motivated the use of deep neural networks (DNNs) as perception models and specifically to predict human speech recognition (HSR). This study investigates if a modeling approach based on a DNN that serves as phoneme classifier [Spille, Ewert, Kollmeier, and Meyer (2018). Comput. Speech Lang. 48, 51-66] can predict HSR for subjects with different degrees of hearing loss when listening to speech embedded in different complex noises. The eight noise signals range from simple stationary noise to a single competing talker and are added to matrix sentences, which are presented to 20 hearing-impaired (HI) listeners (categorized into three groups with different types of age-related hearing loss) to measure their speech recognition threshold (SRT), i.e., the signal-to-noise ratio with 50% word recognition rate. These are compared to responses obtained from the ASR-based model using degraded feature representations that take into account the individual hearing loss of the participants captured by a pure-tone audiogram. Additionally, SRTs obtained from eight normal-hearing (NH) listeners are analyzed. For NH subjects and three groups of HI listeners, the average SRT prediction error is below 2 dB, which is lower than the errors of the baseline models.

Review on human speech Recognition Techniques

Review on human speech Recognition Techniques

Arabic speech recognition by end-to-end, modular systems and human

Recent advances in automatic speech recognition (ASR) have achieved accuracy levels comparable to human transcribers, which led researchers to debate if the machine has reached human performance. Previous work focused on the English language and modular hidden Markov model-deep neural network (HMM–DNN) systems. In this paper, we perform a comprehensive benchmarking for end-to-end transformer ASR, modular HMM–DNN ASR, and human speech recognition (HSR) on the Arabic language and its dialects. For the HSR, we evaluate linguist performance and lay-native speaker performance on a new dataset collected as a part of this study. For ASR the end-to-end work led to 12.5%, 27.5% , 33.8% WER; a new performance milestone for the MGB2, MGB3, and MGB5 challenges respectively. Our results suggest that human performance in the Arabic language is still considerably better than the machine with an absolute WER gap of 3.5% on average.

Bioacoustic classification of avian calls from raw sound waveforms with an open-source deep learning architecture

The use of autonomous recordings of animal sounds to detect species is a popular conservation tool, constantly improving in fidelity as audio hardware and software evolves. Current classification algorithms utilise sound features extracted from the recording rather than the sound itself, with varying degrees of success. Neural networks that learn directly from the raw sound waveforms have been implemented in human speech recognition but the requirements of detailed labelled data have limited their use in bioacoustics. Here we test SincNet, an efficient neural network architecture that learns from the raw waveform using sinc-based filters. Results using an off-the-shelf implementation of SincNet on a publicly available bird sound dataset (NIPS4Bplus) show that the neural network rapidly converged reaching accuracies of over 65% with limited data. Their performance is comparable with traditional methods after hyperparameter tuning but they are more efficient. Learning directly from the raw waveform allows the algorithm to select automatically those elements of the sound that are best suited for the task, bypassing the onerous task of selecting feature extraction techniques and reducing possible biases. We use publicly released code and datasets to encourage others to replicate our results and to apply SincNet to their own datasets; and we review possible enhancements in the hope that algorithms that learn from the raw waveform will become useful bioacoustic tools.

Psycho-acoustics inspired automatic speech recognition

Understanding the human spoken language recognition process is still a far scientific goal. Nowadays, commercial automatic speech recognisers (ASRs) achieve high performance at recognising clean speech, but their approaches are poorly related to human speech recognition. They commonly process the phonetic structure of speech while neglecting supra-segmental and syllabic tracts integral to human speech recognition. As a result, these ASRs achieve low performance on spontaneous speech and require enormous costs to build up phonetic and pronunciation models and catch the large variability of human speech. This paper presents a novel ASR that addresses these issues and questions conventional ASR approaches. It uses alternative acoustic models and an exhaustive decoding algorithm to process speech at a syllabic temporal scale (100–250 ms) through a multi-temporal approach inspired by psycho-acoustic studies. Performance comparison on the recognition of spoken Italian numbers (from 0 to 1 million) demonstrates that our approach is cost-effective, outperforms standard phonetic models, and reaches state-of-the-art performance.

Make Patient Consultation Warmer: A Clinical Application for Speech Emotion Recognition

In recent years, many types of research have continued to improve the environment of human speech and emotion recognition. As facial emotion recognition has gradually matured through speech recognition, the result of this study provided more accurate recognition of complex human emotional performance, and speech emotion identification will be derived from human subjective interpretation into the use of computers to automatically interpret the speaker’s emotional expression. Focused on use in medical care, which can be used to understand the current feelings of physicians and patients during a visit, and improve the medical treatment through the relationship between illness and interaction. By transforming the voice data into a single observation segment per second, the first to the thirteenth dimensions of the frequency cestrum coefficients are used as speech emotion recognition eigenvalue vectors. Vectors for the eigenvalue vectors are maximum, minimum, average, median, and standard deviation, and there are 65 eigenvalues in total for the construction of an artificial neural network. The sentiment recognition system developed by the hospital is used as a comparison between the sentiment recognition results of the artificial neural network classification, and then use the foregoing results for a comprehensive analysis to understand the interaction between the doctor and the patient. Using this experimental module, the emotion recognition rate is 93.34%, and the accuracy rate of facial emotion recognition results can be 86.3%.

Thoughts on the potential to compensate a hearing loss in noise

Background: The effect of hearing impairment on speech perception was described by Plomp (1978) as a sum of a loss of class A, due to signal attenuation, and a loss of class D, due to signal distortion. While a loss of class A can be compensated by linear amplification, a loss of class D, which severely limits the benefit of hearing aids in noisy listening conditions, cannot. The hearing loss of class D is assumed to be the main reason why not few users of hearing aids keep complaining about the limited benefit of their devices in noisy environments. Working compensation strategies against it are unknown. Methods: Recently, in an approach to model human speech recognition by means of a re-purposed automatic speech recognition (ASR) system, the loss of class D was explained by introducing a level uncertainty which reduces the individual accuracy of spectro-temporal signal levels. Based on this finding, an implementation of a patented dynamic range manipulation scheme (PLATT) is proposed which aims to mitigate the effect of increased level uncertainty on speech recognition in noise by expanding spectral modulation patterns in the range of 2 to 4 ERB. This compensation approach is objectively evaluated regarding the benefit in speech recognition thresholds in noise using the ASR-based speech recognition model. Recommendations for an evaluation with human listeners are derived. Results: The objective evaluation suggests that approximately half of the class D loss due to an increased level uncertainty might be compensable. To measure the effect with human listeners, an experiment needs to be carefully designed to prevent the confusion class A and D loss compensations. Conclusions: A working compensation strategy for the class D loss could provide previously unexploited potential for relief. Evidence has to be provided in experiments with human listeners.

Thoughts on the potential to compensate a hearing loss in noise

Background: The effect of hearing impairment on speech perception was described by Plomp (1978) as a sum of a loss of class A, due to signal attenuation, and a loss of class D, due to signal distortion. While a loss of class A can be compensated by linear amplification, a loss of class D, which severely limits the benefit of hearing aids in noisy listening conditions, cannot. The hearing loss of class D is assumed to be the main reason why not few users of hearing aids keep complaining about the limited benefit of their devices in noisy environments. Working compensation strategies against it are unknown. Methods: Recently, in an approach to model human speech recognition by means of a re-purposed automatic speech recognition (ASR) system, the loss of class D was explained by introducing a level uncertainty which reduces the individual accuracy of spectro-temporal signal levels. Based on this finding, an implementation of a patented dynamic range manipulation scheme (PLATT) is proposed which aims to mitigate the effect of increased level uncertainty on speech recognition in noise by expanding spectral modulation patterns in the range of 2 to 4 ERB. This compensation approach is objectively evaluated regarding the benefit in speech recognition thresholds in noise using the ASR-based speech recognition model. Recommendations for an evaluation with human listeners are derived. Results: The objective evaluation suggests that approximately half of the class D loss due to an increased level uncertainty might be compensable. To measure the effect with human listeners, an experiment needs to be carefully designed to prevent the confusion class A and D loss compensations. Conclusions: A working compensation strategy for the class D loss could provide previously unexploited potential for relief. Evidence has to be provided in experiments with human listeners.

Learning Waveform-Based Acoustic Models Using Deep Variational Convolutional Neural Networks

We investigate the potential of stochastic neural networks for learning effective waveform-based acoustic models. The waveform-based setting, inherent to fully end-to-end speech recognition systems, is motivated by several comparative studies of automatic and human speech recognition that associate standard non-adaptive feature extraction techniques with information loss which can adversely affect robustness. Stochastic neural networks, on the other hand, are a class of models capable of incorporating rich regularization mechanisms into the learning process. We consider a deep convolutional neural network that first decomposes speech into frequency sub-bands via an adaptive parametric convolutional block where filters are specified by cosine modulations of compactly supported windows. The network then employs standard non-parametric 1D convolutions to extract relevant spectro-temporal patterns while gradually compressing the structured high dimensional representation generated by the parametric block. We rely on a probabilistic parametrization of the proposed neural architecture and learn the model using stochastic variational inference. This requires evaluation of an analytically intractable integral defining the Kullback-Leibler divergence term responsible for regularization, for which we propose an effective approximation based on the Gauss-Hermite quadrature. Our empirical results demonstrate a superior performance of the proposed approach over comparable waveform-based baselines and indicate that it could lead to robustness. Moreover, the approach outperforms a recently proposed deep convolutional neural network for learning of robust acoustic models with standard FBANK features.

Evidence for automated scoring and shorter passages of CBM-R in early elementary school.

Curriculum-based measurement of oral reading fluency (CBM-R) is widely used across the United States as a strong indicator of comprehension and overall reading achievement, but has several limitations including errors in administration and large standard errors of measurement. The purpose of this study is to compare scoring methods and passage lengths of CBM-R in an effort to evaluate potential improvements upon traditional CBM-R limitations. For a sample of 902 students in Grades 2 through 4, who collectively read 13,766 passages, we used mixed-effect models to estimate differences in CBM-R scores and examine the effects of (a) scoring method (comparing a human scoring criterion vs. traditional human or automatic speech recognition [ASR] scoring), and (b) passage length (25, 50, or 85 words, and traditional CBM-R length). We also examined differences in word score (correct/incorrect) agreement rates between human-to-human scoring and human-to-ASR scoring. Our results indicated that ASR can be applied in schools to score CBM-R, and that scores for shorter passages are comparable to traditional passages. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

A NOVEL SPEECH RECOGNITION SYSTEM USING FUZZY NEURAL NETWORK

An important field in digital speech processing is human voice. Speech recognition in humans has long been a hot topic in signal processing and artificial intelligence research. Natural Language Processing (NLP) offers an interdisciplinary subfield called speech recognition that makes it easier for machines to recognize spoken language and transform it into text. To communicate with machines, speech recognition machines has the potential to be helpful. Technology that can have communicates in real time has been made possible. However, there remain additional issues, such as speaker variance due to other factors like age, gender, signal speed, pronunciation variations, noise from the surrounding area etc. Classification of age and gender is important for speech processing. A lot of work has been done to enhance each of these phases in order to get better and more accurate results. The main goal of this analysis is on the integration of machine learning into the speech recognition system. Hence, this analysis presents a novel speech recognition system using Fuzzy neural network. Pre-processing, speech signal segmentation, speech feature extraction, and speaker recognition are the different phases of a speech recognition system. It uses a fuzzy neural network to identify the speaker's age and gender. Accuracy, F1-score, and Precision are used to evaluate the fuzzy model's performance.

Speech Recognition Using Elman Artificial Neural Network and Linear Predictive Coding

Background: Today, the automatic intelligent system requirement has caused an increasing consideration on the interactive modern techniques between human being and machine. These techniques generally consist of two types: audio and visual methods. Meanwhile, the need for developing the algorithms that enable the human speech recognition by machine is of high importance and frequently studied by the researchers. Objective: Using artificial intelligence methods has led to better results in human speech recognition, but the basic problem is the lack of an appropriate strategy to select the recognition data among the huge amount of speech information that practically makes it impossible for the available algorithms to work. Method: In this article, to solve the problem, the linear predictive coding coefficients extraction method is used to sum up the data related to the English digits pronunciation. After extracting the database, it is utilized to an Elman neural network to recognize the relation between the linear coding coefficients of an audio file with the pronounced digit. Results: The results show that this method has a good performance compared to other methods. According to the experiments, the obtained results of network training (99% recognition accuracy) indicate that the network still has better performance than RBF despite many errors. Conclusion: The results of the experiments showed that the Elman memory neural network has had an acceptable performance in recognizing the speech signal compared to the other algorithms. The use of the linear predictive coding coefficients along with the Elman neural network has led to higher recognition accuracy and improved the speech recognition system.

The Recognition of Persian Phonemes Using PPNet.

Background:In this paper, a novel approach is proposed for the recognition of Persian phonemes in the Persian consonant-vowel combination (PCVC) speech dataset. Nowadays, deep neural networks (NNs) play a crucial role in classification tasks. However, the best results in speech recognition are not yet as perfect as human recognition rate. Deep learning techniques show outstanding performance over many other classification tasks, such as image classification and document classification. Furthermore, the performance is sometimes better than a human. The reason why automatic speech recognition systems are not as qualified as the human speech recognition system, mostly depends on features of data which are fed to deep NNs.Methods:In this research, first, the sound samples are cut for the exact extraction of phoneme sounds in 50 ms samples. Then, phonemes are divided into 30 groups, containing 23 consonants, 6 vowels, and a silence phoneme.Results:The short-time Fourier transform is conducted on them, and the results are given to PPNet (a new deep convolutional NN architecture) classifier and a total average of 75.87% accuracy is reached which is the best result ever compared to other algorithms on separated Persian phonemes (like in PCVC speech dataset).Conclusion:This method not only can be used for recognizing mono-phonemes but it can also be adopted as an input to the selection of the best words in speech transcription

EARSHOT: A Minimal Neural Network Model of Incremental Human Speech Recognition.

Despite the lack of invariance problem (the many-to-many mapping between acoustics and percepts), human listeners experience phonetic constancy and typically perceive what a speaker intends. Most models of human speech recognition (HSR) have side-stepped this problem, working with abstract, idealized inputs and deferring the challenge of working with real speech. In contrast, carefully engineered deep learning networks allow robust, real-world automatic speech recognition (ASR). However, the complexities of deep learning architectures and training regimens make it difficult to use them to provide direct insights into mechanisms that may support HSR. In this brief article, we report preliminary results from a two-layer network that borrows one element from ASR, long short-term memory nodes, which provide dynamic memory for a range of temporal spans. This allows the model to learn to map real speech from multiple talkers to semantic targets with high accuracy, with human-like timecourse of lexical access and phonological competition. Internal representations emerge that resemble phonetically organized responses in human superior temporal gyrus, suggesting that the model develops a distributed phonological code despite no explicit training on phonetic or phonemic targets. The ability to work with real speech is a major advance for cognitive models of HSR.

Optimisation of phonetic aware speech recognition through multi-objective evolutionary algorithms

Recent advances in the availability of computational resources allow for more sophisticated approaches to speech recognition than ever before. This study considers Artificial Neural Network and Hidden Markov Model methods of classification for Human Speech Recognition through Diphthong Vowel sounds in the English Phonetic Alphabet rather than the classical approach of the classification of whole words and phrases, with a specific focus on both single and multi-objective evolutionary optimisation of bioinspired classification methods. A set of audio clips are recorded by subjects from the United Kingdom and Mexico and the recordings are transformed into a static dataset of statistics by way of their Mel-Frequency Cepstral Coefficients (MFCC) at sliding window length of 200ms as well as a reshaped MFCC timeseries format for forecast-based models. An deep neural network with evolutionary optimised topology achieves 90.77% phoneme classification accuracy in comparison to the best HMM that achieves 86.23% accuracy with 150 hidden units, when only accuracy is considered in a single-objective optimisation approach. The obtained solutions are far more complex than the HMM taking around 248 seconds to train on powerful hardware versus 160 for the HMM. A multi-objective approach is explored due to this. In the multi-objective approaches of scalarisation presented, within which real-time resource usage is also considered towards solution fitness, far more optimal solutions are produced which train far quicker than the forecast approach (69 seconds) with classification ability retained (86.73%). Weightings towards either maximising accuracy or reducing resource usage from 0.1 to 0.9 are suggested depending on the resources available, since many future IoT devices and autonomous robots may have limited access to cloud resources at a premium in comparison to the GPU used in this experiment.