Speech Extraction Research Articles

Multi-Channel Speech Enhancement using a Minimum Variance Distortionless Response Beamformer based on Graph Convolutional Network

The Minimum Variance Distortionless Response (MVDR) beamforming algorithm is frequently utilized to extract speech and noise from noisy signals captured from multiple microphones. A frequency-time mask should be employed to compute the Power Spectral Density (PSD) matrices of the noise and the speech signal of interest to obtain the optimal weights for the beamformer. Deep Neural Networks (DNNs) are widely used for estimating time-frequency masks. This paper adopts a novel method using Graph Convolutional Networks (GCNs) to learn spatial correlations among the different channels. GCNs are integrated into the embedding space of a U-Net architecture to estimate a Complex Ideal Ratio Mask (cIRM). We use the cIRM in an MVDR beamformer to further improve the enhancement system. We simulate room acoustics data to experiment extensively with our approach using different types of the microphone array. Results indicate the superiority of our approach when compared to current state-of-the-art methods. The metrics obtained by the proposed method are significantly improved, except the Scale-Invariant Source-to-Distortion Ratio (SI-SDR) score. The Perceptual Evaluation of Speech Quality (PESQ) score shows a noticeable improvement over the baseline models (i.e., 2.207 vs. 2.104 and 2.076). Our implementation of the proposed method can be found in the following link: https://github.com/3i-hust-asr/gnn-mvdr-final.

An investigation of neural uncertainty estimation for target speaker extraction equipped RNN transducer

Target-speaker speech recognition aims to recognize the speech of an enrolled speaker from an environment with background noise and interfering speakers. This study presents a joint framework that combines time-domain target speaker extraction and recurrent neural network transducer (RNN-T) for speech recognition. To alleviate the adverse effects of residual noise and artifacts introduced by the target speaker extraction module to the speech recognition back-end, we explore to training the target speaker extraction and RNN-T jointly. We find a multi-stage training strategy that pre-trains and fine-tunes each module before joint training is crucial in stabilizing the training process. In addition, we propose a novel neural uncertainty estimation that leverages useful information from the target speaker extraction module to further improve the back-end speech recognizer (i.e., speaker identity uncertainty and speech enhancement uncertainty). Compared to a recognizer with target speech extraction front-end, our experiments show that joint-training and the neural uncertainty module reduce 7% and 17% relative character error rate (CER) on multi-talker simulation data, respectively. The multi-condition experiments indicate that our method can reduce 9% relative CER in the noisy condition without losing performance in the clean condition. We also observe consistent improvements in further evaluation of real-world data based on vehicular speech.

Comparative Analysis of Methods Used to Extract Speech Signal Features

The stage of extracting the features of the speech file is one of the most important stages of building a system for identifying a person through the use of his voice. Accordingly, the choice of the method of extracting speech features is an important process because of its subsequent negative or positive effects on the speech recognition system. In this paper research we will analyze the most popular methods of speech signal features extraction: LPC, Kmeans clustering, WPT decomposition and MLBP methods. These methods will be implemented and tested using various speech files. The amplitude and sampling frequency will be changed to see the affects of changing on the extracted features. Depending on the results of analysis some recommendations will be given.

Speech extraction with RGB-intensity gradient on rolling-shutter video

Recent studies have been proposed to extract speech from the captured video of objects vibrating by sound waves. Among them, from the viewpoint of equipment cost, the method of extracting speech from the video captured by rolling-shutter cameras, which are widely used in consumer digital single-lens reflex cameras, has been attracting attention. The conventional method with the rolling-shutter video uses a grayscale video for processing based on phase images. However, a grayscale video has a smaller dynamic range than an RGB video, and thus the speech extraction accuracy of the conventional method degrades. Therefore, this paper proposes a speech extraction method based on RGB-intensity gradients on an RGB video to improve speech extraction accuracy. The proposed method extracts the speech by calculating the similarity of R, G, and B intensity gradients, and using these three intensity gradients expands the dynamic range. The experimental results on the quality and intelligibility of the extracted speech show our proposed method outperforms the conventional method.

HiLAM-aligned kernel discriminant analysis for text-dependent speaker verification

Probabilistic Linear Discriminant Analysis (PLDA) has been a commonly used backend classifier for many text-dependent speaker verification (TDSV) systems. Lately, PLDA projections have been integrated with the traditional Dynamic Time Warping (DTW) template matching framework, resulting in the DTW Online i-vector/PLDA system. The system is shown to achieve state-of-the-art performance for TDSV task. PLDA model serves to train a subspace that compensates for channel and session variabilities. It assumes linear separability between speaker-phrase information and other components. However, this relationship is known to be non-linear. The non-linearity is more prominent in case of short speech extracts, as in the case of the online i-vectors. This results in loss of vital speaker-phrase information at PLDA modeling. To this end, this work explores Kernel Discriminant Analysis (KDA) for TDSV task. It further proposes to use Hierarchical Multi-Layer Acoustic Model (HiLAM) to complement KDA with a more effective speaker-text class definition. The proposed system is hypothesized to benefit on three counts — non-linear modeling ability of KDA, speaker idiosyncrasy information associated with HiLAM-defined speaker-text units and modeling of the exact context of the pass-phrase, as offered by HiLAM. It shows a relative Equal Error Rate (EER) reduction of up to 50.63% on Part 1 of the RSR2015 database when compared to the baseline DTW Online i-vector/PLDA system.

Pattern and content controlled response generation

Controllable response generation is an attractive and valuable task to the success of conversational systems. However, controlling both pattern and content of the response has not been well studied in existing models since they are mainly based on matching mechanisms. To tackle the problem, we first design a pattern model to automatically learn and extract speech patterns from words. The pattern is then integrated into the encoder–decoder model to control the response pattern. Second, a sentence sampling algorithm is built to directly insert or delete words in the generated response, so that the content is controlled. In this two-stage framework, the response could be explicitly controlled by the pattern and content, without any human annotation of the post-response dataset. Experiments show the proposed framework achieves better performance in response controllability than the state-of-the-art.

Real-time pre-processing for improved feature extraction of noisy speech

Several improvements of algorithms for the front-end feature extraction of real-time speech decoding in noisy ambiance have been proposed with their demonstration on the TIMIT speech corpus. Real-Time Voice Activity Detection (RT-VAD) is used to separate the voiced–unvoiced part of input from silence in the streaming speech input. Novel techniques for RT-Zero Crossing Detection and RT-Pitch Detection are presented as part of RT-VAD. Real-Time approximate Kalman filter is then applied to de-noise the incoming signal. All these are applied across a collection of frames of speech called context. Frame-based Linear Discriminant Analysis (LDA)-feature extraction is done by RT-Cepstral Mean and Variance Normalization (RT-CMVN) and RT-Splicing. The algorithms are tested on the TIMIT database for various noise levels. It is observed that we obtain a word-error rate (WER) improvement of 5% for 30 dB and 7% for 10 dB SNR, thus validating the proposed algorithms. Also, the comparison with other works shows a superior Speech Hit Rate (SHR) of 90.6% and Noise Hit Rate (NHR) of 86.2%.

Speaker Separation Using Speaker Inventories and Estimated Speech

We propose speaker separation using speaker inventories and estimated speech (SSUSIES), a framework leveraging speaker profiles and estimated speech for speaker separation. SSUSIES contains two methods, speaker separation using speaker inventories (SSUSI) and speaker separation using estimated speech (SSUES). SSUSI performs speaker separation with the help of speaker inventory. By combining the advantages of permutation invariant training (PIT) and speech extraction, SSUSI significantly outperforms conventional approaches. SSUES is a widely applicable technique that can substantially improve speaker separation performance using the output of first-pass separation. We evaluate the models on both speaker separation and speech recognition metrics.

Speaker Identification for Business-Card-Type Sensors

Human collaboration has a great impact on the performance of multi-person activities. The analysis of speaker information and speech timing can be used to extract human collaboration data in detail. Some studies have extracted human collaboration data by identifying a speaker with business-card-type sensors. However, it is difficult to realize speaker identification for business-card-type sensors at low cost and high accuracy because of spikes in the measured sound pressure data, ambient noise in the non-speaker sensor, and synchronization errors across each sensor. This study proposes a novel sound pressure sensor and speaker identification algorithm to realize speaker identification for business-card-type sensors. The sensor extracts the user's speech at low cost and high accuracy by employing a peak hold circuit and time synchronization module for spike mitigation and precise time synchronization. The algorithm identifies a speaker with high accuracy by removing ambient noise. The evaluations show that the algorithm accurately identifies a speaker in a multi-person activity considering varying numbers of users, environmental noises, and reverberation conditions as well as long or short utterances. In addition, the peak hold circuit enables accurate extraction of speech and the synchronization error between the sensors is always within $\pm$ 30 $\boldsymbol\mu$ s, that is, negligible error.

SEMANTIC SURVEY ON AUTOMATIC SPEECH RECOGNITION SYSTEM FOR REGIONAL LANGUAGES

Speech recognition in regional languages has become one of the most important researchable area which employs as a helping mechanism to talk and understand in our own language. The Automatic speech recognition system has played an important role for the people who have hearing and linguistic problem. Speech recognition has become a realistic technology that is now being applied in numerous languages around the world and is being used in applications of real-life human language, such as knowledge recovery. In this paper various parameters including development of Automatic Speech Recognition systems, tools and techniques used to construct an ASR system with their advantages and limitations and research on system of regional languages with ASR have been discussed. The main motive of this work is to present various techniques used for speech extraction, recognition and subtitle generation.

Application of an Isolated Word Speech Recognition System in the Field of Mental Health Consultation: Development and Usability Study.

BackgroundSpeech recognition is a technology that enables machines to understand human language.ObjectiveIn this study, speech recognition of isolated words from a small vocabulary was applied to the field of mental health counseling.MethodsA software platform was used to establish a human-machine chat for psychological counselling. The software uses voice recognition technology to decode the user's voice information. The software system analyzes and processes the user's voice information according to many internal related databases, and then gives the user accurate feedback. For users who need psychological treatment, the system provides them with psychological education.ResultsThe speech recognition system included features such as speech extraction, endpoint detection, feature value extraction, training data, and speech recognition.ConclusionsThe Hidden Markov Model was adopted, based on multithread programming under a VC2005 compilation environment, to realize the parallel operation of the algorithm and improve the efficiency of speech recognition. After the design was completed, simulation debugging was performed in the laboratory. The experimental results showed that the designed program met the basic requirements of a speech recognition system.

REVIEW, STUDY & MODELLING OF NEURO-FUZZY SYSTEM, IT'S PERFORMANCE COMPARISION USING VARIOUS SPEECH DATABASES FOR RECOGNITION OF EMOTIONAL STATES

Objective Study and modelling of various Neural Network & Neuro Fuzzy System to determine best design solution for recognition of emotional states through human speech. Methods By performing test experiment on 3 different models using MFCC and other speech extraction technique on multiple databases. Database must include native language speech samples as well. Results As per above mentioned objective and method 76% (approx.) overall average recognition rate was achieved. Some significant experimental finding and observations were also recorded for further and future research.

S110. INTERESTING HAPPY THOUGHTS: STRUCTURAL, SEMANTIC AND EMOTIONAL ANALYSIS OF PSYCHOTIC SPEECH USING TIME-LIMITED POSITIVE IMAGE NARRATIVES

BackgroundSpeech and language analysis from free speech protocols has recently provided a discriminative signal, useful for early diagnosis of schizophrenia. Although different aspects of language (such as structural and semantic coherence) have been applied to different contexts using different data collection protocols, we need to standardize a safe and minimum-effort protocol that can reveal discriminative data, enabling large and remote dataset collection. Also, it is important to understand the correlations between semantic, structural and emotional analysis from the same dataset. In the past decade, we have developed a non-semantic structural analysis based on graph theory that was able to automatically discriminate speech samples from patients with schizophrenia diagnosis with more than 90% accuracy in chronic, first-episode patients, and in different languages. Moreover, we could verify correlations of structural attributes with negative symptoms, as well as with cognitive performances in patients and in typical children at regular school time. But the most predictive contents were a dream report (sometimes absent) or a negative image report (which could cause a psychological burden for some subjects). The current project aims to verify the accuracy to discriminate schizophrenia reports from 3 different positive image prompts, using a minimum of 30 seconds reports. Furthermore, we want to verify correlates between semantic, structural and emotional analysis.MethodsWe analyzed reports of 3 positive images from 31 subjects (10 matched controls and 21 at the first episode of psychosis - 11 with schizophrenia and 10 with bipolar disorder as a final diagnosis after 6 months of follow-up). We performed speech graph analysis to extract speech connectedness attributes. Next, we combined connectedness measures from the 3 prompts (after extracting collinear measures) to create a disorganization index (performing multilinear correlation with the PANSS negative subscale). We used this index as an input to a machine learning classifier to verify the accuracy to discriminate reports from the schizophrenia group. Finally, we studied the correlations between the connectedness-based disorganization index and minimum semantic coherence between consecutive sentences, and the emotional intensity measured by the proportion of emotional words.ResultsSpeech connectedness of positive image reports was correlated with negative symptomatology severity measured by PANSS negative subscale (R2 = 0.73, p = 0.0160), and the disorganization index was able to discriminate the subjects diagnosed with schizophrenia disorder six months later with AUC = 0.82. Moreover, the disorganization index was negatively correlated with positive emotional intensity (Rho = -0.48, p = 0.0061), but not correlated with minimum semantic coherence (Rho = -0.06, p = 0.7442). Emotional intensity was not correlated with minimum semantic coherence (Rho = 0.17, p = 0.3458).DiscussionThis safe, short and standardized data collection protocol seems to be informative and reveals an interdependent relationship between different aspects of computational language analysis. With less than two minutes of oral speech data, we can accurately discriminate reports from the schizophrenia group at the first interview, and verify that the less connected the report, the fewer positive emotional words are used. Future directions point to the feasibility of automatic and remote access of a large and diverse population, allowing the upscaling of this type of assessment to big data.

Automatic and manual extraction of speech and language characteristics associated with depression

Early and accurate diagnosis and intervention of depression is important to facilitate timely, direct, and appropriate interventions with potential for improved clinical outcomes. Delays in the diagnosis of patients with depression may be reduced if simple tools were available to indicate probability of diagnosis. Clinicians use speech and language characteristics to establish current mental state and diagnosis. The use of automatic acoustic feature extraction allows leveraging pitch, power, and variability and can provide an unbiased evaluation of speech.
 This study examined speech samples from a youth at-risk cohort, aged 9-25, and developed a manual rating system of speech and language characteristics, which involved rating short segments of audio and transcript on emotion, sentiment, affect, and richness. This competed against an automated model of extracting zero-crossing rate, energy, the entropy of energy, and Mel-frequency cepstral coefficients to identify speech characteristics associated with major depressive disorder.
 The results showed that the automatic feature extraction outcompeted the manual rating system in explaining the difference in speech between participants with and without major depressive disorder through speech and language characteristics. While the extraction of audio features is not a substitute for the clinical interview, the ability to provide an unbiased prediction of vulnerability to depression from speech may assist clinicians in early diagnosis.

Efficient Feature Extraction Algorithms to Develop an Arabic Speech Recognition System

This paper studies three feature extraction methods, Mel-Frequency Cepstral Coefficients (MFCC), Power-Normalized Cepstral Coefficients (PNCC), and Modified Group Delay Function (ModGDF) for the development of an Automated Speech Recognition System (ASR) in Arabic. The Support Vector Machine (SVM) algorithm processed the obtained features. These feature extraction algorithms extract speech or voice characteristics and process the group delay functionality calculated straight from the voice signal. These algorithms were deployed to extract audio forms from Arabic speakers. PNCC provided the best recognition results in Arabic speech in comparison with the other methods. Simulation results showed that PNCC and ModGDF were more accurate than MFCC in Arabic speech recognition.

Blind Speech Extraction Based on Rank-Constrained Spatial Covariance Matrix Estimation With Multivariate Generalized Gaussian Distribution

In this article, we propose a new blind speech extraction (BSE) method that robustly extracts a directional speech from background diffuse noise by combining independent low-rank matrix analysis (ILRMA) and efficient rank-constrained spatial covariance matrix (SCM) estimation. To achieve more accurate BSE than ILRMA, which assumes each source to be a point source (rank-1 spatial model), the proposed method restores the lost spatial basis for the full-rank SCM of diffuse noise. We adopt the multivariate complex generalized Gaussian distribution (GGD) as the statistical generative model to express various types of observed signal. To estimate the model parameters for an arbitrary shape parameter of the multivariate GGD, we derive a new inequality for rank-constrained SCMs. Also, we propose new acceleration methods to accomplish much faster extraction than conventional blind source separation methods. In BSE experiments using simulated and real recorded data, we confirm that the proposed method achieves more accurate and faster speech extraction than conventional methods.

Successive Relative Transfer Function Identification Using Blind Oblique Projection

Distortionless speech extraction in a reverberant environment can be achieved by applying a beamforming algorithm, provided that the relative transfer functions (RTFs) of the sources and the covariance matrix of the noise are known. In this paper, the challenge of RTF identification in a multi-speaker scenario is addressed. We propose a successive RTF identification (SRI) technique, based on the sole assumption that sources do not become simultaneously active. That is, we address the challenge of estimating the RTF of a specific speech source while assuming that the RTFs of all other active sources in the environment were previously estimated in an earlier stage. The RTF of interest is identified by applying the blind oblique projection (BOP)-SRI technique. When a new speech source is identified, the BOP algorithm is applied. BOP results in a null steering toward the RTF of interest, by means of applying an oblique projection to the microphone measurements. We prove that by artificially increasing the rank of the range of the projection matrix, the RTF of interest can be identified. An experimental study is carried out to evaluate the performance of the BOP-SRI algorithm in various signal to noise ratio (SNR) and signal to interference ratio (SIR) conditions and to demonstrate its effectiveness in speech extraction tasks.

一种稳定、精准、实时的语音信号基频的检测与提取算法

一种稳定、精准、实时的语音信号基频的检测与提取算法

Comprehensive Analysis of Discussion Forum Participation: From Speech Acts to Discussion Dynamics and Course Outcomes

Learning in computer-mediated setting represents a complex, multidimensional process. This complexity calls for a comprehensive analytical approach that would allow for understanding of various dimensions of learner generated discourse and the structure of the underlying social interactions. Current research, however, primarily focuses on manual or, more recently, supervised methods for discourse analysis. Moreover, discourse and social structures are typically analyzed separately without the use of computational methods that can offer a holistic perspective. This paper proposes an approach that addresses these two challenges, first, by using an unsupervised machine learning approach to extract speech acts as representations of knowledge construction processes and finds transition probabilities between speech acts across different messages, and second, by integrating the use of discovered speech acts to explain the formation of social ties and predicting course outcomes. We extracted six categories of speech acts from messages exchanged in discussion forums of two MOOCs and each category corresponded to knowledge construction processes from well-established theoretical models. We further showed how measures derived from discourse analysis explained the ways how social ties were created that framed emerging social networks. Multiple regression models showed that the combined use of measures derived from discourse analysis and social ties predicted learning outcomes.

The discourse and coordination among advocacy coalitions: the case of Belo Monte

Purpose This study aims to analyze the use of discourse to solve issues related to coordination between advocacy coalitions in processes of gradual and transformative institutional change related to public policies. Design/methodology/approach Theoretical background is based on the advocacy coalition framework (ACF), new discursive institutionalism and critical discourse analysis theories. The research examines shorthand notes of public hearings held in the Brazilian Chamber of Deputies and the Federal Senate between 1999 and 2012, carrying out a case study on Belo Monte hydroelectric power plant. The speech extracts were categorized according to the modes of operation of ideology and typical strategies of symbolic construction proposed by Thompson (1995). Findings The results suggest that the discourse can be an instrument of internal coordination and between coalitions that share beliefs about a policy, as in the case of Belo Monte. Potentially existing coalitions define their identities and set positions on controversial issues, aligning interests and expectations. In the case studied, the modes of operation of ideology verified as instruments of the coalitions were dissimulation, reification, fragmentation, unification and legitimation. Research limitations/implications The paper represents a unique analysis of the modes of operation of ideology (Thompson, 1999) in the case of Belo Monte. In addition, the paper aims to contribute to the New Discursive Institutionalism and to the ACF when it uses the critical discourse analysis to articulate a method to analyze the use of the Discourse by the coalitions. In fact, such an approach integrating the ACF, the New Discursive Institutionalism and the critical discourse analysis is something original. Finally, it also addresses a gap in ACF: issues related to advocacy coalition coordination. Practical implications Attentive readers linked to organizations working on infrastructure and environmental policies can benefit from the results by envisaging the deliberate manipulation of typical symbolic construction strategies and general modes of operation of ideology. Social implications The study sheds light on the daily and behind-the-scenes disputes among stakeholders who are interested in a certain public policy. It may draw attention to the access and professional use of the shorthand notes of the hearings held at the National Congress. Originality/value This paper aims to fill a gap pointed out by Jenkins-Smith et al. (2014) regarding problems of coordination of advocacy coalitions. In addition, it innovates by using critical discourse analysis as a methodological reference in ACF empirical studies. In addition, this work continues a trajectory of two other previously published studies dealing with the same phenomenon: a theoretical essay and a case study.