Speech Input Research Articles

Easy come, easy go: Examining the stability of lexically guided perceptual learning over time

Listeners can use lexical information to accommodate ambiguity in speech input. Some evidence suggest that lexically guided perceptual learning persists over time. However, other evidence suggests that lexically guided perceptual learning attenuates throughout the test session, consistent with distributional learning that occurs given exposure to the test stimuli. Here we test the hypothesis that lexically guided and distributional learning may operate over different time scales. During exposure, listeners heard spectral energy ambiguous between /ʃ/ and /s/ in a lexically-biasing context. At test, listeners categorized tokens from an ashi-asi continuum. Test duration was manipulated between subjects to be either brief or long. Approximately 24 hours later, both groups completed a second test phase. The results to date show (1) robust lexically guided perceptual learning at the first test, (2) attenuation of learning for the long duration compared to the short duration test group at the first test, and (3) no robust evidence of lexically guided perceptual learning for either group at the second test. If these results hold in the full sample to be tested, then they would suggest that lexically guided perceptual learning is best characterized as dynamic adaption to recent input that dissipates within a day.

Linking lexically guided perceptual learning to statistical patterns in speech input

Listeners use lexical information to modify the mapping between speech acoustics and speech sound categories. Despite convention to consider lexically guided perceptual learning as a binary outcome, the magnitude of the learning effect varies in the extant literature. We hypothesize that graded learning outcomes can be linked, in part, to statistical characteristics of the to-be-learned input, consistent with the ideal adapter theory of speech adaptation. Following standard methods (i.e., waveform averaging to create ambiguous variants), a lexically guided perceptual learning stimulus set for the /ʃ/-/s/ contrast was created for each of 16 talkers, yielding variability in the statistical cues specifying this contrast across talkers. Experiment 1 will (a) measure lexically guided perceptual learning for each talker, (b) identify input characteristics that are associated with learning magnitude, and (c) examine whether a computational instantiation of the ideal adapter theory can model the input-learning link. Experiment 2 will provide a confirmatory test of the patterns observed in Experiment 1 by manipulating the to-be-learned input holding talker constant. The results will provide a critical test of the ideal adapter framework for speech adaptation, thus informing an understanding of the mechanisms that allow listeners to solve the lack of invariance problem for speech perception.

Statistical regularities of accented speech influence listeners’ own speech productions

Conversation partners’ speech acoustics sometimes converge, providing evidence for the transfer of information from speech perception to one’s own speech productions. The nature of this phonetic convergence has remained elusive, with variable experimental findings regarding the contexts under which phonetic convergence emerges, the acoustic speech features it affects, and the gender of talkers most influenced. Here, we approach phonetic convergence through the lens of dimension-based statistical learning whereby the statistical regularities of short-term speech input impact the perceptual weight of acoustic dimensions in speech categorization. Participants passively listened to a randomly ordered sequence of 4 pier and 4 beer utterances, sampled across voice onset time (VOT) and fundamental frequency (F0) in a manner that aligned with English norms (Canonical) or departed from typical English pronunciations as an ‘accent’ (Reverse). Immediately after, participants categorized and repeated aloud an ambiguous test stimulus varying only in F0, with ambiguous VOT. The Reverse input regularities led both male and female participants to down-weight F0 in both perceptual categorization and in the acoustics of word repetitions. The results indicate that statistical learning across passive listening to another speaker’s voice can lead to detailed acoustic-phonetic adjustments in one’s own speech productions.

Performance Comparison between RSA and El-Gamal Algorithms for Speech Data Encryption and Decryption

This article presents a performance comparison of two known public key cryptography techniques namely RSA (Rivest–Shamir–Adleman) and El-Gamal algorithms to encrypt/decrypt the speech signals during transferring over open networks. Specifically, this work is divided into two stages. The first stage is enciphering-deciphering the input speech file by employing the RSA method. The second stage is enciphering-deciphering the same input speech file by employing the El-Gamal method. Then, a comparative analysis is performed to test the performance of both cryptosystems using diverse experimental and statistical analyses for the ciphering and deciphering procedures like some known speech quality measures: histogram, spectrogram, correlation, differential, speed performance and noise effect analyses. The analyses outcomes reveal that the RSA and El-Gamal approaches are efficient and adequate for providing high degree of security, confidentiality and reliability. Additionally, the outcomes indicate that the RSA speech cryptosystem outperforms its counterpart the El-Gamal speech cryptosystem in most of ciphering/deciphering speech performance metrics.

Theta-gamma phase-amplitude coupling in auditory cortex is modulated by language proficiency.

The coordination between the theta phase (3-7 Hz) and gamma power (25-35 Hz) oscillations (namely theta-gamma phase-amplitude coupling, PAC) in the auditory cortex has been proposed as an essential neural mechanism involved in speech processing. However, it has not been established how this mechanism is related to the efficiency with which a listener processes speech. Speech processing in a non-native language offers a useful opportunity to evaluate if theta-gamma PAC is modulated by the challenges imposed by the reception of speech input in a non-native language. The present study investigates how auditory theta-gamma PAC (recorded with magnetoencephalography) is modulated in both native and non-native speech reception. Participants were Spanish native (L1) speakers studying Basque (L2) at three different levels: beginner (Grade 1), intermediate (Grade 2), and advanced (Grade 3). We found that during L2 speech processing (i) theta-gamma PAC was more highly coordinated for intelligible compared to unintelligible speech; (ii) this coupling was modulated by proficiency in Basque being lower for beginners, higher for intermediate, and highest for advanced speakers (no difference observed in Spanish); (iii) gamma power did not differ between languages and groups. These findings highlight how the coordinated theta-gamma oscillatory activity is tightly related to speech comprehension: the stronger this coordination is, the more the comprehension system will proficiently parse the incoming speech input.

A unified speech enhancement approach to mitigate both background noises and adversarial perturbations

Deep learning has been vastly applied in classification systems of speech signals. However, adversarial examples crafted by adding small perturbations to input speech signals can make those classification systems produce incorrect results. Beyond detecting and discarding adversarial examples when securing the classification systems, we consider to purify the input adversarial examples to normal ones in this paper. Speech enhancement is a technique known as removing background noises to enhance voice quality. By treating adversarial perturbations as noises, we hereby propose a unified speech enhancement approach to mitigate both background noises and adversarial perturbations. The approach is named by Speech Enhancement and Adversarial Defense Network (SEADNet), which performs model-level fusion of conventional speech enhancement and adversarial defenses. In SEADNet, a learnable noise-adding module is firstly created to adaptively putting noises to the input speech to destroy the functions of adversarial perturbations. Subsequently, a speech enhancement network is deployed to reconstruct clean speech from noisy inputs. Finally, an adversarial example detection module is designed as a discriminator to guide the reconstructed speech in the previous speech enhancement network to mitigate adversarial perturbations. The overall network is optimized by minimizing a joint loss function. Experiments are conducted on the tasks of speech enhancement and automatic speaker verification with benchmark datasets. The proposed SEADNet performs well on both tasks of conventional noise mitigation and adversarial defenses compared to the related competitive baselines.

Recognition of English speech – using a deep learning algorithm

Abstract The accurate recognition of speech is beneficial to the fields of machine translation and intelligent human–computer interaction. After briefly introducing speech recognition algorithms, this study proposed to recognize speech with a recurrent neural network (RNN) and adopted the connectionist temporal classification (CTC) algorithm to align input speech sequences and output text sequences forcibly. Simulation experiments compared the RNN-CTC algorithm with the Gaussian mixture model–hidden Markov model and convolutional neural network-CTC algorithms. The results demonstrated that the more training samples the speech recognition algorithm had, the higher the recognition accuracy of the trained algorithm was, but the training time consumption increased gradually; the more samples a trained speech recognition algorithm had to test, the lower the recognition accuracy and the longer the testing time. The proposed RNN-CTC speech recognition algorithm always had the highest accuracy and the lowest training and testing time among the three algorithms when the number of training and testing samples was the same.

Hearing is believing: Lexically guided perceptual learning is graded to reflect the quantity of evidence in speech input

There is wide variability in the acoustic patterns that are produced for a given linguistic message, including variability that is conditioned on who is speaking. Listeners solve this lack of invariance problem, at least in part, by dynamically modifying the mapping to speech sounds in response to structured variation in the input. Here we test a primary tenet of the ideal adapter framework of speech adaptation, which posits that perceptual learning reflects the incremental updating of cue-sound mappings to incorporate observed evidence with prior beliefs. Our investigation draws on the influential lexically guided perceptual learning paradigm. During an exposure phase, listeners heard a talker who produced fricative energy ambiguous between /ʃ/ and /s/. Lexical context differentially biased interpretation of the ambiguity as either /s/ or /ʃ/, and, across two behavioral experiments (n = 500), we manipulated the quantity of evidence and the consistency of evidence that was provided during exposure. Following exposure, listeners categorized tokens from an ashi – asi continuum to assess learning. The ideal adapter framework was formalized through computational simulations, which predicted that learning would be graded to reflect the quantity, but not the consistency, of the exposure input. These predictions were upheld in human listeners; the magnitude of the learning effect monotonically increased given exposure to four, 10, or 20 critical productions, and there was no evidence that learning differed given consistent versus inconsistent exposure. These results (1) provide support for a primary tenet of the ideal adapter framework, (2) establish quantity of evidence as a key determinant of adaptation in human listeners, and (3) provide critical evidence that lexically guided perceptual learning is not a binary outcome. In doing so, the current work provides foundational knowledge to support theoretical advances that consider perceptual learning as a graded outcome that is tightly linked to input statistics in the speech stream.

Understanding and Applying Deep Learning.

The past 10 years have witnessed an explosion in deep learning neural network model development. The most common perceptual models with vision, speech, and text inputs are not general-purpose AI systems but tools. They automatically extract clues from inputs and compute probabilities of class labels. Successful applications require representative training data, an understanding of the limitations and capabilities of deep learning, and careful attention to a complex development process. The goal of this view is to foster an intuitive understanding of convolutional network deep learning models and how to use them with the goal of engaging a wider creative community. A focus is to make it possible for experts in areas such as health, education, poverty, and agriculture to understand the process of deep learning model development so they can help transition effective solutions to practice.

Bispectral feature speech intelligibility assessment metric based on auditory model

A bispectral feature based predictive speech intelligibility metric (GMBSIM) using a more refined functional auditory model of ear is proposed. In the auditory model of ear, Gammatone filter banks and Meddis inner hair cell auditory model is combined to simulate the ear function. With input speech signal divided into 32 auditory subbands, and each subband signal passed through the inner hair cell model, the bispectrum of each subband signal in time domain is estimated by frames. And then bispectral features are extracted and chosen to calculate the speech intelligibility. The proposed GMBSIM has relative low computational complexity by omitting the spectrogram or neurogram image transformation. Considering the ear's perception and processing of speech signals makes the metric is advantageous to the classical metrics. And the last but not the least, the proposed GMBSIM metric is verified favorably across a range of conditions spanning reverberation, additive noise, and distortion such as jitter, which means it can be applied in most kinds of complex background noise environment.

Smart Multi-Model Emotion Recognition System with Deep learning

Emotion recognition is added a new dimension to the sentiment analysis. This paper presents a multi-modal human emotion recognition web application by considering of three traits includes speech, text, facial expressions, to extract and analyze emotions of people who are giving interviews. Now a days there is a rapid development of Machine Learning, Artificial Intelligence and deep learning, this emotion recognition is getting more attention from researchers. These machines are said to be intelligent only if they are able to do human recognition or sentiment analysis. Emotion recognition helps in spam call detection, blackmailing calls, customer services, lie detectors, audience engagement, suspicious behavior. In this paper focus on facial expression analysis is carried out by using deep learning approaches with speech signals and input text.

CAQoE: A Novel No-Reference Context-aware Speech Quality Prediction Metric

The quality of speech degrades while communicating over Voice over Internet Protocol applications, for example, Google Meet, Microsoft Skype, and Apple FaceTime, due to different types of background noise present in the surroundings. It reduces human perceived Quality of Experience (QoE). Along this line, this article proposes a novel speech quality prediction metric that can meet human’s desired QoE level. Our motivation is driven by the lack of evidence showing speech quality metrics that can distinguish different noise degradations before predicting the quality of speech. The quality of speech in noisy environments is improved by speech enhancement algorithms, and for measuring and monitoring the quality of speech, objective speech quality metrics are used. With the integration of these components, a novel no-reference context-aware QoE prediction metric (CAQoE) is proposed in this article, which initially identifies the context or noise type or degradation type of the input noisy speech signal and then predicts context-specific speech quality for that input speech signal. It will have of great importance in deciding the speech enhancement algorithms if the types of degradations causing poor speech quality are known along with the quality metric. Results demonstrate that the proposed CAQoE metric outperforms in different contexts as compared to the metric where contexts are not identified before predicting the quality of speech, even in the presence of limited size speech corpus having different contexts available from the NOIZEUS speech database.

Sleep-Based Memory Consolidation Stabilizes Perceptual Learning of Noise-Vocoded Speech.

Sleep-based memory consolidation has been shown to facilitate perceptual learning of atypical speech input including nonnative speech sounds, accented speech, and synthetic speech. The current research examined the role of sleep-based memory consolidation on perceptual learning for noise-vocoded speech, including maintenance of learning over a 1-week time interval. Because comprehending noise-vocoded speech requires extensive restructuring of the mapping between the acoustic signal and prelexical representations, sleep consolidation may be critical for this type of adaptation. Thus, the purpose of this study was to investigate the role of sleep-based memory consolidation on adaptation to noise-vocoded speech in listeners without hearing loss as a foundational step toward identifying parameters that can be useful to consider for auditory training with clinical populations. Two groups of normal-hearing listeners completed a transcription training task with feedback for noise-vocoded sentences in either the morning or the evening. Learning was assessed through transcription accuracy before training, immediately after training, 12 hr after training, and 1 week after training for both trained and novel sentences. Both the morning and evening groups showed improved comprehension of noise-vocoded sentences immediately following training. Twelve hours later, the evening group showed stable gains (following a period of sleep), whereas the morning group demonstrated a decline in gains (following a period of wakefulness). One week after training, the morning and evening groups showed equivalent performance for both trained and novel sentences. Sleep-consolidated learning helps stabilize training gains for degraded speech input, which may hold clinical utility for optimizing rehabilitation recommendations.

Auditory Word Comprehension Is Less Incremental in Isolated Words.

Partial speech input is often understood to trigger rapid and automatic activation of successively higher-level representations of words, from sound to meaning. Here we show evidence from magnetoencephalography that this type of incremental processing is limited when words are heard in isolation as compared to continuous speech. This suggests a less unified and automatic word recognition process than is often assumed. We present evidence from isolated words that neural effects of phoneme probability, quantified by phoneme surprisal, are significantly stronger than (statistically null) effects of phoneme-by-phoneme lexical uncertainty, quantified by cohort entropy. In contrast, we find robust effects of both cohort entropy and phoneme surprisal during perception of connected speech, with a significant interaction between the contexts. This dissociation rules out models of word recognition in which phoneme surprisal and cohort entropy are common indicators of a uniform process, even though these closely related information-theoretic measures both arise from the probability distribution of wordforms consistent with the input. We propose that phoneme surprisal effects reflect automatic access of a lower level of representation of the auditory input (e.g., wordforms) while the occurrence of cohort entropy effects is task sensitive, driven by a competition process or a higher-level representation that is engaged late (or not at all) during the processing of single words.

Speech Enhancement Algorithm Based on a Convolutional Neural Network Reconstruction of the Temporal Envelope of Speech in Noisy Environments

Temporal modulation processing is a promising technique for improving the intelligibility and quality of speech in noise. We propose a speech enhancement algorithm that constructs the temporal envelope (TEV) in the time-frequency domain by means of an embedded convolutional neural network (CNN). To accomplish this, the input speech signals are divided into sixteen parallel frequency bands (subbands) with bandwidths approximating 1.5 times that of auditory filters. The corrupted TEVs in each subband are extracted and then fed to the 1-dimensional CNN (1-D CNN) model to restore the TEVs distorted by noise. The method is evaluated using 2,700 words from nine different talkers, which are mixed with speech-spectrum shaped random noise (SSN), and babble noise, at different signal-to-noise ratios. The Short-time Objective Intelligibility (STOI) and Perceptual Evaluation of Speech Quality (PESQ) metrics are used to evaluate the performance of the 1-D CNN algorithm. Results suggest that the 1-D CNN model improves STOI scores on average by 27% and 34% for SSN and babble noise, respectively, and PESQ scores on average by 19% and 18%, respectively, compared to unprocessed speech. The 1-D CNN model is also shown to outperform a conventional TEV-based speech enhancement algorithm.

Bayesian Self-Attentive Speaker Embeddings for Text-Independent Speaker Verification

Learning effective and discriminative speaker embeddings is a crucial task in speaker verification. Usually, speaker embeddings are extracted from a speaker-classification DNN that averages the hidden vectors over all the spoken frames of a speaker; the hidden vectors produced from all the frames are assumed to be equally important. In our previous work, we relaxed this assumption and computed the speaker embedding as a weighted average of a speaker's frame-level hidden vectors, and their weights were automatically determined by a self-attention mechanism. The effect of multiple attention heads have also been investigated to capture different aspects of a speaker's input speech. One challenge for multi-head attention is the information redundancy problem. If there is no constraint during the training of multi-head attention, different heads may extract similar attentive features, leading to the attention redundancy problem. In this paper, we generalize the deterministic multi-head attention to a Bayesian attention framework, and provide a new understanding of multi-head attention from a Bayesian perspective. Under the Bayesian framework, we adopt the recently developed sampling method in optimization, which explicitly enforces the repulsiveness among the multiple heads. Systematic evaluation of the proposed Bayesian self-attentive speaker embeddings is performed on VoxCeleb and SITW evaluation sets. Significant and consistent improvements over other multi-head attention systems are achieved on all the evaluation datasets. The best Bayesian system with eight heads improves the EER by around 26% on VoxCeleb and 9% on SITW over the single-head baseline.

Deep Learning-Based Non-Intrusive Multi-Objective Speech Assessment Model With Cross-Domain Features

This study proposes a cross-domain multi-objective speech assessment model, called MOSA-Net, which can simultaneously estimate the speech quality, intelligibility, and distortion assessment scores of an input speech signal. MOSA-Net comprises a convolutional neural network and bidirectional long short-term memory architecture for representation extraction, and a multiplicative attention layer and a fully connected layer for each assessment metric prediction. Additionally, cross-domain features (spectral and time-domain features) and latent representations from self-supervised learned (SSL) models are used as inputs to combine rich acoustic information to obtain more accurate assessments. Experimental results show that in both seen and unseen noise environments, MOSA-Net can improve the linear correlation coefficient (LCC) scores in perceptual evaluation of speech quality (PESQ) prediction, compared to Quality-Net, an existing single-task model for PESQ prediction, and improve LCC scores in short-time objective intelligibility (STOI) prediction, compared to STOI-Net, an existing single-task model for STOI prediction. Moreover, MOSA-Net can be used as a pre-trained model to be effectively adapted to an assessment model for predicting subjective quality and intelligibility scores with a limited amount of training data. Experimental results show that MOSA-Net can improve LCC scores in mean opinion score (MOS) predictions, compared to MOS-SSL, a strong single-task model for MOS prediction. We further adopt the latent representations of MOSA-Net to guide the speech enhancement (SE) process and derive a quality-intelligibility (QI)-aware SE (QIA-SE) approach. Experimental results show that QIA-SE outperforms the baseline SE system with improved PESQ scores in both seen and unseen noise environments over a baseline SE model.

Real-Time Speech Enhancement Based on Convolutional Recurrent Neural Network

Speech enhancement is the task of taking a noisy speech input and producing an enhanced speech output. In recent years, the need for speech enhancement has been increased due to challenges that occurred in various applications such as hearing aids, Automatic Speech Recognition (ASR), and mobile speech communication systems. Most of the Speech Enhancement research work has been carried out for English, Chinese, and other European languages. Only a few research works involve speech enhancement in Indian regional Languages. In this paper, we propose a two-fold architecture to perform speech enhancement for Tamil speech signal based on convolutional recurrent neural network (CRN) that addresses the speech enhancement in a real-time single channel or track of sound created by the speaker. In the first stage mask based long short-term memory (LSTM) is used for noise suppression along with loss function and in the second stage, Convolutional Encoder-Decoder (CED) is used for speech restoration. The proposed model is evaluated on various speaker and noisy environments like Babble noise, car noise, and white Gaussian noise. The proposed CRN model improves speech quality by 0.1 points when compared with the LSTM base model and also CRN requires fewer parameters for training. The performance of the proposed model is outstanding even in low Signal to Noise Ratio (SNR).

Bisayan Dialect Short-time Fourier Transform Audio Recognition System using Convolutional and Recurrent Neural Network

Speech is a form of oral communication that reinforces thoughts and ideas that have general purpose and meaning. In the Philippines, Filipinos can speak at least three languages. English, Filipino, native language. The Philippine government says the Philippines has more than 150 regional native languages, one of which he says is Cebuano. This research aims to implement automatic speech recognition (ASR) specifically for the Bisayan dialect, and researchers use machine learning techniques to create and operate the system. ASR has served its purpose in recent years not only in the official language of the Philippines, but also in various foreign languages. The required datasets were collected throughout the study to train and build the models selected for the speech recognition engine. Audio files are recorded in waveform file format and contain Visayan phrases and sentences. Audio was captured through hours of recorded audio and process using Tensorflow short time Fourier transform (STFT) algorithm to ensure the accurate representation. In order to analyze the audio data, the recordings were specially converted to digital format, specifically .wav and making it sure all records are uncorrupted with only one channel, and finally have a sample rate of 22050kHz. A data mining process was carried out by integrating CNN layers, dense layers, and RNNs to predict the transcription of speech input using multiple layers that determine the output of the speech data. The researchers used the JiWER Python library, which was used in parallel when evaluating WER. This is because the trained scripted data set contains at least 500 time recordings totaling 61.78 minutes. Overall, the WER output is at best 99.53% and the percentage of records used is acceptable.

Continuous-Time Analog Filters for Audio Edge Intelligence: Review on Circuit Designs [Feature

Edge audio devices can reduce data bandwidth requirements by pre-processing input speech on the device before transmission to the cloud. As edge devices are required to ensure always-on operation, their stringent power constraints pose several design challenges and force IC designers to look for solutions that use low standby power. One promising bio-inspired approach is to combine the continuous-time analog filter channels with a small memory footprint deep neural network that is trained on edge tasks such as keyword spotting, thereby allowing all blocks to be embedded in an IC. This paper reviews the historical background of the continuous-time analog filter circuits that have been used as feature extractors for current edge audio devices. Starting from the interpretation of a basic biquad filter as a two-integrator-loop topology, we introduce the progression in the design of second-order low-pass and band-pass filters ranging from OTA-based to source-follower-based architectures. We also derive and analyze the small-signal transfer function and discuss their usage in edge audio applications.