Speech Processing Systems Research Articles

Spectrum estimation for voiced speech using average weighted linear prediction

In many different areas, such as neurophysics, geophysics, and communication, it is vital to effectively handle signals when there is unwanted disturbance. Linear prediction (LP) methods provide precise modeling capabilities for various parametric models and are used in a wide range of fields such as predicting future data, compressing speech, analyzing spectra, filling in missing signals, and decreasing noise. LP is especially beneficial in speech processing systems. Speech analysis and synthesis greatly relies on spectral envelopes, which are closely connected to models of speech production and perception. However, the performance of the LP method may decline when limited information is involved in pitch synchronous analysis. In order to overcome this constraint, we developed the implementation of the Average Weighted Linear Prediction (AWLP) technique, which does not require the use of a window. The AWLP utilizes a signal of weighted prediction error acquired from the autocorrelation process of LP. By performing this action, it provides a more precise estimation of the power spectrum and coefficients used for vocal tract parametric models. Importantly, the effectiveness of the AWLP can be seen in both pitch synchronous and asynchronous analyses, which means it is appropriate for situations where segments are chosen randomly. We confirm the effectiveness of the AWLP technique by employing it on both synthetic voice and real speech. Additionally, we assess its effectiveness by comparing it to the direct autocorrelation technique and a modified autocorrelation (MA) approach. To sum up, the AWLP method presents a hopeful solution for reliable spectral analysis that addresses the divide between pitch synchronous and asynchronous situations.

Front-end speech processing system with SVM algorithm in rail passenger flow management

In order to achieve the goal of dynamically adjusting daily passenger flow to effectively control the overall efficiency of the transportation system, this study constructs a real-time monitoring and prediction system for subway passenger flow based on front-end voice processing technology and support vector machine models. The study first conducted a railway passenger flow analysis, and then used a support vector machine model to construct a preliminary prediction system. In order to achieve global optimization, the study also introduced particle swarm optimization algorithm to construct an optimization prediction model based on PSO-SVM. The results show that the proposed PSO-SVM method has undergone 48 iterations of training, and the predicted values closely match the actual passenger flow curve. The maximum RE error is 2%, and the overall prediction error is 98%. The decision coefficient of PSO-SVM is 0.998932. Therefore, this indicates that it has high performance and feasibility in predicting and controlling passenger flow during peak hours of urban rail transit.

Noise Impact on Optimum Parameters for Signal Deconvolution

In many areas, such as speech processing, communication, and measurement systems, where the actual signal is not directly available, Wiener deconvolution is a widely used signal restoration technique. The accuracy of the estimated signal in Wiener deconvolution is determined by the unknown filter parameter. To compute this unknown parameter, complex and time-consuming numerical as well as iterative computation approaches were proposed in the literature, which also often compute an unoptimized unknown filter parameter. To address the computational problem and estimate a more accurate unknown filter parameter, the authors reported an analytical solution in a prior work for a given signal to noise ratio. However, it was revealed that the actual optimum parameter value also varies with noise also. In this paper, the authors propose a new analytical solution for optimum parameter estimation that includes noise as well. The authors compared the modified analytical solution to existing numerical computation methods for different values of Signal to Noise Ratio and demonstrated that the proposed methods’ optimum parameter estimation is always close to the actual optimum value, which resulted in more accurate signal deconvolution. Proposed mathematical solutions are also verified with experimental signals, acquired in a laboratory.

VERBAL COMMUNICATION: NOISE REDUCTION AND CHARACTERISTICS ANALYSIS USING WAVELET AND THRESHOLD DERIVATIVE METHOD

Noise reduction is a very difficult task but it has been a subject of intense research papers in recent years. Audio signal processing system is often suffering from noise troubles. Noise will seriously affect the quality of audio and even lead to audio processing failure. Simulation and results are performed in MATLAB R2012. In order to extract the desired speech signal from its corrupted signal, speech processing and communication systems are applied for effective noise reduction techniques. To avoid this noise troubles and quality degrades, noise reduction and characteristic analysis using discrete wavelet and threshold derivative method are used which improves the signal quality and to achieve high signal to noise ratio (SNR)

Emergence of the cortical encoding of phonetic features in the first year of life

Even prior to producing their first words, infants are developing a sophisticated speech processing system, with robust word recognition present by 4–6 months of age. These emergent linguistic skills, observed with behavioural investigations, are likely to rely on increasingly sophisticated neural underpinnings. The infant brain is known to robustly track the speech envelope, however previous cortical tracking studies were unable to demonstrate the presence of phonetic feature encoding. Here we utilise temporal response functions computed from electrophysiological responses to nursery rhymes to investigate the cortical encoding of phonetic features in a longitudinal cohort of infants when aged 4, 7 and 11 months, as well as adults. The analyses reveal an increasingly detailed and acoustically invariant phonetic encoding emerging over the first year of life, providing neurophysiological evidence that the pre-verbal human cortex learns phonetic categories. By contrast, we found no credible evidence for age-related increases in cortical tracking of the acoustic spectrogram.

Noise estimation based on optimal smoothing and minimum controlled through recursive averaging for speech enhancement

One of the most significant challenges in the real time speech processing applications is the elimination of noise in the corrupted speech data. This noise can significantly impact the efficacy/performance of the applications of speech processing. However, developing a robust noise reduction algorithm is crucial for improving the accuracy of automatic speech recognition (ASR) and other speech processing systems under uncontrolled conditions. In this paper, we propose an algorithm to reduce the background noise in the degraded speech data under highly non-stationary conditions. The proposed optimal smoothing and minima controlled (OSMC) technique uses recursive averaging to enhance degraded speech data. Initially, a smoothed periodogram and local minima of the degraded speech data are computed and determined the time-frequency dependent threshold factor. The ratio of smoothed periodogram to local minima is used to find the active regions of speech in the degraded speech data by adapting the Bayesian minimum cost decision rule. To calculate the estimated noise spectrum for each frequency bin, a time-frequency smoothing factors are used. The perceptual evaluation of speech quality (PESQ) and normalized covariance metric (NCM) are used to evaluate the speech quality and intelligibility of the proposed technique over competing algorithms after speech enhancement. The experimental results demonstrated that the proposed algorithm has given a significant improvement in terms of average values of PESQ by 15.03% and 16.71% and NCM by 3.45% and 5.73% for NOIZEUS and Kannada speech databases at 5 dB and 10 dB respectively, over unprocessed speech under highly non-stationary noisy environments.

A measure of differences in speech signals by the voice timbre

This research relates to the field of speech technologies, where the key problem is the optimization of speech signal processing under conditions of a priori uncertainty of its fine structure. The task of automatic (objective) analysis of voice timbre using a speech signal of finite duration is considered. It is proposed to use a universal information-theoretic approach to solve it. Based on the Kullback-Leibler divergence, an expression is obtained for the asymptotically optimal decision statistic for distinguishing speech signals by voice timbre. Pointed to an acute problem in its practical implementation, namely: synchronization of the sequence of observations with the main tone of speech signals. To overcome the described problem, an objective measure of timbre differences in speech signals is proposed in terms of the acoustic theory of speech production and its model of the speaker’s vocal tract of the “acoustic trumpet” type. The possibilities of practical implementation of a new measure based on an adaptive recursive are considered. A full-scale experiment was set up and carried out. According to its results, two main properties of the proposed measure were confirmed: high sensitivity to differences in speech signals in terms of voice timbre and, at the same time, invariance with respect to the pitch frequency. The results obtained can be used in the design and research of digital speech processing systems tuned to the speaker’s voice, for example, digital speech transmission systems, biometric, biomedical systems, etc.

Optimizing medical personnel speech recognition models using speech synthesis and reinforcement learning

Text-to-Speech synthesis (TTS) can be used to generate training data for building Automatic Speech Recognition models (ASR). Access to medical speech data is because it is sensitive data that is difficult to obtain for privacy reasons. Speech can be synthesized by mimicking different accents, dialects, and speaking styles in a medical language. Reinforcement Learning (RL), in the context of ASR, can be used to optimize a model. A model can be trained to minimize errors in speech-to-text transcription, especially for technical medical terminology. In this case, the “reward” to the RL model can be negatively proportional to the number of transcription errors. The paper presents a method and experimental study from which it is concluded that the combination of TTS and RL can enable the creation of a speech recognition model suited to the specific needs of medical personnel, helping to expand the training data and optimize the model to minimize transcription errors. The learning process used reward functions based on Mean Opinion Score (MOS), a subjective metric for assessing speech quality, and Word Error Rate (WER), which evaluates the quality of speech-to-text transcription. [The Polish National Center for Research and Development (NCBR) supported the project: “ADMEDVOICE- Adaptive intelligent speech processing system of medical personnel with the structuring of test results and support of therapeutic process,” no. INFOSTRATEG4/0003/2022.]

Performance analysis of speech enhancement using spectral gating with U-Net

Abstract Many speech processing systems’ crucial frontends include speech enhancement. Single-channel speech enhancement experiences a number of technological challenges. Due to the advent of cloud-based technology and the use of deep learning systems in big data, deep neural networks in particular have recently been seen as a potent means for complex classification and regression. In this work, spectral gating noise filter is combined with deep neural network U-Net to enhance the performance of speech enhancement network. Further, for performance analysis three distinct objective functions namely, Mean Square Error, Huber Loss and Mean Absolute Error are considered as loss functions. In addition, comparison of three different optimizers Adam, Adagrad and Stochastic Gradient Descent is presented. Proposed system is tested and evaluated on LibriSpeech and NOIZEUS datasets and compared to other state-of-the-art systems. It demonstrates that, in comparison to other state-of-the-art models, the proposed network outperformed them with PESQ scores of 2.737420 for training and 2.67857 for testing, along with better generalization ability.

Synthesizing medical terms—Quality and naturalness of the deep Text-to-Speech algorithm

The main purpose of this study is to develop a deep text-to-speech (TTS) algorithm designated for an embedded system device. First, a critical literature review of state-of-the-art speech synthesis deep models is provided. The algorithm implementation covers both hardware and algorithmic solutions. The algorithm is designed for use with the Raspberry Pi 4 board. 80 synthesized sentences were prepared based on medical and everyday language employing the TTS algorithm developed. For tests, an application is built, containing a questionnaire allowing for evaluating the quality and naturalness of the synthesized speech, for both types of language. It is followed by the algorithm efficiency tests. A presentation of the performed tests, along with the results obtained from 30 respondents, is shown. The discussion consists of a statistical analysis of the obtained results and a comparison with other speech recognition solutions used as a reference. Finally, in the summary section, there is an overall conclusion of this approach and promising directions for future development. [This work is supported by the Polish National Center for Research and Development (NCBR) project: “ADMEDVOICE-Adaptive intelligent speech processing system of medical personnel with the structuring of test results and support of therapeutic process,” no. INFOSTRATEG4/0003/2022.]

NISQE: Non-Intrusive Speech Quality Evaluator Based on Natural Statistics of Mean Subtracted Contrast Normalized Coefficients of Spectrogram

With the evolution in technology, communication based on the voice has gained importance in applications such as online conferencing, online meetings, voice-over internet protocol (VoIP), etc. Limiting factors such as environmental noise, encoding and decoding of the speech signal, and limitations of technology may degrade the quality of the speech signal. Therefore, there is a requirement for continuous quality assessment of the speech signal. Speech quality assessment (SQA) enables the system to automatically tune network parameters to improve speech quality. Furthermore, there are many speech transmitters and receivers that are used for voice processing including mobile devices and high-performance computers that can benefit from SQA. SQA plays a significant role in the evaluation of speech-processing systems. Non-intrusive speech quality assessment (NI-SQA) is a challenging task due to the unavailability of pristine speech signals in real-world scenarios. The success of NI-SQA techniques highly relies on the features used to assess speech quality. Various NI-SQA methods are available that extract features from speech signals in different domains, but they do not take into account the natural structure of the speech signals for assessment of speech quality. This work proposes a method for NI-SQA based on the natural structure of the speech signals that are approximated using the natural spectrogram statistical (NSS) properties derived from the speech signal spectrogram. The pristine version of the speech signal follows a structured natural pattern that is disrupted when distortion is introduced in the speech signal. The deviation of NSS properties between the pristine and distorted speech signals is utilized to predict speech quality. The proposed methodology shows better performance in comparison to state-of-the-art NI-SQA methods on the Centre for Speech Technology Voice Cloning Toolkit corpus (VCTK-Corpus) with a Spearman’s rank-ordered correlation constant (SRC) of 0.902, Pearson correlation constant (PCC) of 0.960, and root mean squared error (RMSE) of 0.206. Conversely, on the NOIZEUS-960 database, the proposed methodology shows an SRC of 0.958, PCC of 0.960, and RMSE of 0.114.

Ensemble deep learning in speech signal tasks: A review

Machine learning methods are extensively used for processing and analysing speech signals by virtue of their performance gains over multiple domains. Deep learning and ensemble learning are the two most commonly used techniques, which results in benchmark performance across different downstream tasks. Ensemble deep learning is a recent development which combines these two techniques to result in a robust architecture having substantial performance gains, as well as better generalization performance over the individual techniques. In this paper, we extensively review the use of ensemble deep learning methods for different speech signal related tasks, ranging from general objectives such as automatic speech recognition and voice activity detection, to more specific areas such as biomedical applications involving the detection of pathological speech or music genre detection. We provide a discussion on the use of different ensemble strategies such as bagging, boosting and stacking in the context of speech signals, and identify the various salient features and advantages from a broader perspective when coupled with deep learning architectures. The main objective of this study is to comprehensively evaluate existing works in the area of ensemble deep learning, and highlight the future directions that may be explored to further develop it as a tool for several speech related tasks. To the best of our knowledge, this is the first review study which primarily focuses on ensemble deep learning for speech applications. This study aims to serve as a valuable resource for researchers in academia and in industry working with speech signals, supporting advanced novel applications of ensemble deep learning models towards solving challenges in existing speech processing systems.

Metamodal Coupling of Vibrotactile and Auditory Speech Processing Systems through Matched Stimulus Representations.

It has been postulated that the brain is organized by "metamodal," sensory-independent cortical modules capable of performing tasks (e.g., word recognition) in both "standard" and novel sensory modalities. Still, this theory has primarily been tested in sensory-deprived individuals, with mixed evidence in neurotypical subjects, thereby limiting its support as a general principle of brain organization. Critically, current theories of metamodal processing do not specify requirements for successful metamodal processing at the level of neural representations. Specification at this level may be particularly important in neurotypical individuals, where novel sensory modalities must interface with existing representations for the standard sense. Here we hypothesized that effective metamodal engagement of a cortical area requires congruence between stimulus representations in the standard and novel sensory modalities in that region. To test this, we first used fMRI to identify bilateral auditory speech representations. We then trained 20 human participants (12 female) to recognize vibrotactile versions of auditory words using one of two auditory-to-vibrotactile algorithms. The vocoded algorithm attempted to match the encoding scheme of auditory speech while the token-based algorithm did not. Crucially, using fMRI, we found that only in the vocoded group did trained-vibrotactile stimuli recruit speech representations in the superior temporal gyrus and lead to increased coupling between them and somatosensory areas. Our results advance our understanding of brain organization by providing new insight into unlocking the metamodal potential of the brain, thereby benefitting the design of novel sensory substitution devices that aim to tap into existing processing streams in the brain.SIGNIFICANCE STATEMENT It has been proposed that the brain is organized by "metamodal," sensory-independent modules specialized for performing certain tasks. This idea has inspired therapeutic applications, such as sensory substitution devices, for example, enabling blind individuals "to see" by transforming visual input into soundscapes. Yet, other studies have failed to demonstrate metamodal engagement. Here, we tested the hypothesis that metamodal engagement in neurotypical individuals requires matching the encoding schemes between stimuli from the novel and standard sensory modalities. We trained two groups of subjects to recognize words generated by one of two auditory-to-vibrotactile transformations. Critically, only vibrotactile stimuli that were matched to the neural encoding of auditory speech engaged auditory speech areas after training. This suggests that matching encoding schemes is critical to unlocking the brain's metamodal potential.

Neuromorphic acoustic sensing using an adaptive microelectromechanical cochlea with integrated feedback

Many speech processing systems struggle in conditions with low signal-to-noise ratios and in changing acoustic environments. Adaptation at the transduction level with integrated signal processing could help to address this; in human hearing, transduction and signal processing are integrated and can be adaptively tuned for noisy conditions. Here we report a microelectromechanical cochlea as a bio-inspired acoustic sensor with integrated signal processing functionality. Real-time feedback is used to tune the sensing and processing properties, and dynamic switching between linear and nonlinear characteristics improves the detection of signals in noisy conditions, increases the sensor dynamic range and enables adaptation to changing acoustic environments. The transition to nonlinear behaviour is attributed to a Hopf bifurcation and we experimentally validate its dependence on sensor and feedback parameters. We also show that output-signal coupling between two coupled sensors can increase the frequency coverage.

Acoustic feedback control in hearing aids with frequency warping

Acoustic Feedback Control continues to be a challenging problem due to the emerging form factors in advanced hearing aids and hearables, especially for severe and profound losses. In order to mitigate the effects of feedback in hearing aids, we present a novel use of frequency warping called “Freping,” for manipulating signals in the frequency domain. Freping is a non-linear transformation which breaks the similarity between the input and acoustic feedback signals of a hearing aid to prevent feedback. In this work, we implemented Freping in realtime for feedback management, as well as novel techniques to preserve the naturalness of sound. Our prior work has shown that Freping improves Added Stable Gain (ASG) in the presence of feedback while maintaining the same Hearing Aid Speech Quality Index (HASQI). We present new results for feedback management with Freping by performing “on-ear” measurements on an acoustic KEMAR manikin using standard hearing aid verification systems. We conducted feedback tests to measure the ASG with various configurations of Freping for various hearing loss levels using the NAL-NL2 prescriptions on a realtime open-source speech processing system for hearing aids. We also compare Freping feedback performance with that of commercial hearing aids.

Mechanized acoustic panels for variable reverb control

Reverberation poses a challenge for speech processing systems and is unavoidable in real environments. As such, acoustic signal processing researchers are interested in robust algorithms that can perform well regardless of reverb severity. Measuring reverberant speech requires access to rooms with the desired dimensions, which may not be readily available. Furthermore, for heavily instrumented recording setups such as our Mechatronic Acoustic Research System (MARS), moving equipment between locations is not practical. We propose a system of actuated multi-textured panels which can significantly alter the reverb properties of a static room. By changing the angle of these panels, we can smoothly transition from high to low reverberation times. We show this empirically and develop a system that provides a requested reverb level automatically, once configured for a fixed size installation. This tool can allow researchers to use a single room to take measurements applicable to a more diverse range of environments. We integrate our system into the MARS project, an automated tool for generating spatial audio datasets.

Individual theta-band cortical entrainment to speech in quiet predicts word-in-noise comprehension.

Speech elicits brain activity time-locked to its amplitude envelope. The resulting speech-brain synchrony (SBS) is thought to be crucial to speech parsing and comprehension. It has been shown that higher speech-brain coherence is associated with increased speech intelligibility. However, studies depending on the experimental manipulation of speech stimuli do not allow conclusion about the causality of the observed tracking. Here, we investigate whether individual differences in the intrinsic propensity to track the speech envelope when listening to speech-in-quiet is predictive of individual differences in speech-recognition-in-noise, in an independent task. We evaluated the cerebral tracking of speech in source-localized magnetoencephalography, at timescales corresponding to the phrases, words, syllables and phonemes. We found that individual differences in syllabic tracking in right superior temporal gyrus and in left middle temporal gyrus (MTG) were positively associated with recognition accuracy in an independent words-in-noise task. Furthermore, directed connectivity analysis showed that this relationship is partially mediated by top-down connectivity from premotor cortex-associated with speech processing and active sensing in the auditory domain-to left MTG. Thus, the extent of SBS-even during clear speech-reflects an active mechanism of the speech processing system that may confer resilience to noise.

ODIN112–AI-Assisted Emergency Services in Romania

The evolution of Natural Language Processing technologies transformed them into viable choices for various accessibility features and for facilitating interactions between humans and computers. A subset of them consists of speech processing systems, such as Automatic Speech Recognition, which became more accurate and more popular as a result. In this article, we introduce an architecture built around various speech processing systems to enhance Romanian emergency services. Our system is designed to help the operator evaluate various situations with the end goal of reducing the response times of emergency services. We also release the largest high-quality speech dataset of more than 150 h for Romanian. Our architecture includes an Automatic Speech Recognition model to transcribe calls automatically and augment the operator’s notes, as well as a Speech Recognition model to classify the caller’s emotions. We achieve state-of-the-art results on both tasks, while our demonstrator is designed to be integrated with the Romanian emergency system.

Self-Supervised Language Learning From Raw Audio: Lessons From the Zero Resource Speech Challenge

Recent progress in self-supervised or unsupervised machine learning has opened the possibility of building a full speech processing system from raw audio without using any textual representations or expert labels such as phonemes, dictionaries or parse trees. The contribution of the Zero Resource Speech Challenge series since 2015 has been to break down this long-term objective into four well-defined tasks -- Acoustic Unit Discovery, Spoken Term Discovery, Discrete Resynthesis, and Spoken Language Modeling -- and introduce associated metrics and benchmarks enabling model comparison and cumulative progress. We present an overview of the six editions of this challenge series since 2015, discuss the lessons learned, and outline the areas which need more work or give puzzling results.

DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing From Decentralized Data.

Deep neural speech and audio processing systems have a large number of trainable parameters, a relatively complex architecture, and require a vast amount of training data and computational power. These constraints make it more challenging to integrate such systems into embedded devices and utilize them for real-time, real-world applications. We tackle these limitations by introducing DeepSpectrumLite, an open-source, lightweight transfer learning framework for on-device speech and audio recognition using pre-trained image Convolutional Neural Networks (CNNs). The framework creates and augments Mel spectrogram plots on the fly from raw audio signals which are then used to finetune specific pre-trained CNNs for the target classification task. Subsequently, the whole pipeline can be run in real-time with a mean inference lag of 242.0 ms when a DenseNet121 model is used on a consumer-grade Motorola moto e7 plus smartphone. DeepSpectrumLite operates decentralized, eliminating the need for data upload for further processing. We demonstrate the suitability of the proposed transfer learning approach for embedded audio signal processing by obtaining state-of-the-art results on a set of paralinguistic and general audio tasks, including speech and music emotion recognition, social signal processing, COVID-19 cough and COVID-19 speech analysis, and snore sound classification. We provide an extensive command-line interface for users and developers which is comprehensively documented and publicly available at https://github.com/DeepSpectrum/DeepSpectrumLite.