TIMIT Speech Research Articles

Speech preprocessing and enhancement based on joint time domain and time-frequency domain analysis.

Speech enhancement aims to make noisy speech signals clearer. Traditional time-frequency domain methods struggle to differentiate between speech and noise, leading to a risk of speech distortion. This paper introduces an approach that combines the time domain and time-frequency domain using the W-net module to suppress noise at the front end. The module is an improved version of Wave-U-Net, called TTF-W-Net. We conducted experiments using the TIMIT speech and NOISEX-92 noise datasets to evaluate the enhancement performance achieved by integrating preprocessing networks, specifically Wave-U-Net and our TTF-W-Net, into the baseline methods: Phase, FullSubNet+, and DB-AIAT. Experimental results show that TTF-W-Net outperforms the baseline Wave-U-Net by 15.7% on the PESQ metric and the effect of the network by using our preprocessing method is improved. Consequently, the TTF-W-Net preprocessing Net offers effective speech enhancement.

Recurrence plot embeddings as short segment nonlinear features for multimodal speaker identification using air, bone and throat microphones

Speech is produced by a nonlinear, dynamical Vocal Tract (VT) system, and is transmitted through multiple (air, bone and skin conduction) modes, as captured by the air, bone and throat microphones respectively. Speaker specific characteristics that capture this nonlinearity are rarely used as stand-alone features for speaker modeling, and at best have been used in tandem with well known linear spectral features to produce tangible results. This paper proposes Recurrent Plot (RP) embeddings as stand-alone, non-linear speaker-discriminating features. Two datasets, the continuous multimodal TIMIT speech corpus and the consonant-vowel unimodal syllable dataset, are used in this study for conducting closed-set speaker identification experiments. Experiments with unimodal speaker recognition systems show that RP embeddings capture the nonlinear dynamics of the VT system which are unique to every speaker, in all the modes of speech. The Air (A), Bone (B) and Throat (T) microphone systems, trained purely on RP embeddings perform with an accuracy of 95.81%, 98.18% and 99.74%, respectively. Experiments using the joint feature space of combined RP embeddings for bimodal (A–T, A–B, B–T) and trimodal (A–B–T) systems show that the best trimodal system (99.84% accuracy) performs on par with trimodal systems using spectrogram (99.45%) and MFCC (99.98%). The 98.84% performance of the B–T bimodal system shows the efficacy of a speaker recognition system based entirely on alternate (bone and throat) speech, in the absence of the standard (air) speech. The results underscore the significance of the RP embedding, as a nonlinear feature representation of the dynamical VT system that can act independently for speaker recognition. It is envisaged that speech recognition too will benefit from this nonlinear feature.

FRMDN: Flow-based Recurrent Mixture Density Network

The class of recurrent mixture density networks is an important class of probabilistic models used extensively in sequence modeling and sequence-to-sequence mapping applications. In this class of models, the density of a target sequence in each time-step is modeled by a Gaussian mixture model with the parameters given by a recurrent neural network. In this paper, we generalize recurrent mixture density networks by using a normalizing flow to non-linearly transform the target space. Furthermore to improve the modeling power, we adopting a suitable covariance matrix decomposition involving a summation of a low-rank and a diagonal matrix. Using these two techniques, we still have a tractable log-likelihood. We also applied the proposed model on some speech and image data, and observed that the model has significant modeling power outperforming other state-of-the-art methods in terms of the log-likelihood on some data. The log-likelihood improvement over other methods is 3523 units for TIMIT speech dataset, and is 1118 and 176 units for MNIST and CIFAR10 image datasets. We were only underperformed on one of the speech datasets by 1209 units.

Spectral degradations in the TIMIT, QuickSIN, NU-6, and other popular bandlimited speech materials

Many popular speech corpora were recorded decades ago using transducers, recording techniques, or post-processing techniques that resulted in spectral degradations. The use of such materials very likely has consequences for measures of speech perception, but this is not often considered in studies that use these materials. We analyzed several of these popular corpora to ascertain spectral degradations that may be present. Spectral degradations we frequently observed included limited extended high-frequency (EHF) (>8 kHz) content, and, for a few corpora, substantial low-frequency roll-off <500 Hz. Two corpora analyzed here, the TIMIT/PRESTO and NU-6, exhibited both high- and low-frequency degradations. Such spectral degradations will likely have an effect on behavioral measures because they do not match the high-fidelity speech signals listeners encounter in their everyday lives. This should be considered when selecting stimuli for speech perception experiments. [Work supported by NIH Grant R01-DC019745]

Mel spectrogram-based audio forgery detection using CNN

In this time of technology, digital speech can be created and falsified by a very diverse of hardware and software technologies. Audio copy-move forgery is an audio forgery technique that goals to create forged audio by hiding undesirable words or repeating wanted words in identical speech. Therefore, audio authentication has been a necessary requisition. In this study, an effective approach to spectral images based on audio copy-move forgery detection using convolutional neural networks (CNN) with data augmentation is proposed. There are only a few handcrafted methods conducted for the detection of audio copy-move forgery. None of the existing works on audio copy-move forgery detection has proposed deep feature learning from speech recording with Mel spectrogram. This is the first method to employ deep learning with Mel spectrogram of audio for the detection of audio copy-move forgery. The proposed CNN architecture classifies the suspicious Mel spectrogram images into two classes: original and forged. The proposed CNN system is successfully trained on these Mel spectrogram image feature extraction. The proposed algorithm has been tested on our datasets generated from Arabic Speech Corpus and TIMIT speech database. The results show the effectiveness, robustness of post-processing operations, and high accuracy of the proposed approach compared to other studies.

Detection of audio copy-move-forgery with novel feature matching on Mel spectrogram

Audio copy-move-forgery created by copying one or more segments of an audio file and pasting it in a different position within the same audio is one of the most widely used methods in the field of audio forensics. This type of forgery is easy to apply but difficult to detect in the case of post-processing operations applied to forged speech to hide traces of forgeries. This paper proposes a robust method for the detection and localization of the audio copy-move forgery using a keypoint-based approach to the Mel spectrogram representation of audio. In the proposed method, first, the Mel spectrogram image is created from the input audio. Then, SIFT keypoints are obtained from each RGB color channel of this image. The obtained keypoints from each channel are matched via feature vectors to reveal the clues of the forgery regions, and the image sub-blocks whose keypoints are determined to be the center are labeled as forged blocks. Then the blocks in the neighborhood of the forged blocks are investigated whether forged or not. The proposed post-processing stage completes the determination of the forged regions. This stage eliminates the possible false positives and marks the forged areas in the spectrogram image. The forged segments are marked in the audio file by utilizing the positions of the forged regions in the spectrogram image. Experimental studies are carried out on two pitch-based datasets, using TIMIT and Arabic Speech Corpus. The paper presents the detailed performance results of popular referenced studies on these datasets. The performance results prove that the proposed method is more robust against common post-processing operations such as noise addition, filtering operation, and especially compression operation.

Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning

Spiking neural networks (SNNs) with leaky integrate and fire (LIF) neurons, can be operated in an event-driven manner and have internal states to retain information over time, providing opportunities for energy-efficient neuromorphic computing, especially on edge devices. Note, however, many representative works on SNNs do not fully demonstrate the usefulness of their inherent recurrence (membrane potential retaining information about the past) for sequential learning. Most of the works train SNNs to recognize static images by artificially expanded input representation in time through rate coding. We show that SNNs can be trained for practical sequential tasks by proposing modifications to a network of LIF neurons that enable internal states to learn long sequences and make their inherent recurrence resilient to the vanishing gradient problem. We then develop a training scheme to train the proposed SNNs with improved inherent recurrence dynamics. Our training scheme allows spiking neurons to produce multi-bit outputs (as opposed to binary spikes) which help mitigate the mismatch between a derivative of spiking neurons' activation function and a surrogate derivative used to overcome spiking neurons' non-differentiability. Our experimental results indicate that the proposed SNN architecture on TIMIT and LibriSpeech 100h speech recognition dataset yields accuracy comparable to that of LSTMs (within 1.10% and 0.36%, respectively), but with 2x fewer parameters than LSTMs. The sparse SNN outputs also lead to 10.13x and 11.14x savings in multiplication operations compared to GRUs, which are generally considered as a lightweight alternative to LSTMs, on TIMIT and LibriSpeech 100h datasets, respectively.

APhL aligner: A neural network forced-alignment system

Forced alignment is increasingly used in phonetics to automatically produce boundaries between words and phones. These boundaries can have significant errors and are often only placed at some predetermined time interval, like every 10 ms. We discuss some potential remedies to these difficulties and test them in a new neural network-based forced alignment system called the APhL Aligner, trained on the TIMIT and Buckeye speech corpora. In part, errors incurred during forced alignment can be attributed to the acoustic models that attempt to separate phones from each other. Even state-of-the-art neural network models struggle to acoustically separate phones. We examine the effect of relaxing the requirement to separate phones by instead training separate detectors for each phone class. Resolving the 10 ms interval difficulty requires a different approach. As with most aligners, we perform a Viterbi-style alignment to align windows of audio spaced at 10 ms to the phone string given by a pronunciation dictionary. We add an additional step, however, and use linear interpolation to determine an intermediate point after the 10 ms interval to place the boundary. We compare the results of these manipulations to the results of the Montreal Forced Aligner, custom-trained on the same data.

Real-time pre-processing for improved feature extraction of noisy speech

Several improvements of algorithms for the front-end feature extraction of real-time speech decoding in noisy ambiance have been proposed with their demonstration on the TIMIT speech corpus. Real-Time Voice Activity Detection (RT-VAD) is used to separate the voiced–unvoiced part of input from silence in the streaming speech input. Novel techniques for RT-Zero Crossing Detection and RT-Pitch Detection are presented as part of RT-VAD. Real-Time approximate Kalman filter is then applied to de-noise the incoming signal. All these are applied across a collection of frames of speech called context. Frame-based Linear Discriminant Analysis (LDA)-feature extraction is done by RT-Cepstral Mean and Variance Normalization (RT-CMVN) and RT-Splicing. The algorithms are tested on the TIMIT database for various noise levels. It is observed that we obtain a word-error rate (WER) improvement of 5% for 30 dB and 7% for 10 dB SNR, thus validating the proposed algorithms. Also, the comparison with other works shows a superior Speech Hit Rate (SHR) of 90.6% and Noise Hit Rate (NHR) of 86.2%.

VOP detection for read and conversation speech using CWT coefficients and phone boundaries

In this paper, we propose a novel approach for accurate detection of vowel onset points (VOPs). A VOP is the instant at which a vowel begins in a speech signal. Precise identification of VOPs is important for various speech applications such as speech segmentation and speech rate modification. Existing methods detect the majority of VOPs to an accuracy of 40 ms deviation, which may not be appropriate for the above speech applications. To address this issue, we proposed a two-stage approach for accurate detection of VOPs. At the first stage, VOPs are detected using continuous wavelet transform coefficients, and the position of the detected VOPs are corrected using phone boundaries in the second stage. The phone boundaries are detected by the spectral transition measure method. Experiments are done using TIMIT and Bengali speech corpora. Performance of the proposed approach is compared with two standard signal processing based methods as well as with a recent VOP detection technique. The evaluation results show that the proposed method performs better than the existing methods.

Detection of vowel segments in noise with ImageNet neural network architectures

In this article we report on experiments on detection of vowel segments in speech with additive noise. Deep neural networks have become the key algorithm in the majority of modern machine learning solutions. We investigate the performance of four ImageNet convolutional neural network (CNN) architectures. Usage of image processing CNNs is enabled by transforming the speech segments into spectrograms before the classification takes place. We perform experiments on TIMIT speech dataset and noise from datasets MAVD and ESC-50. The accuracy of individual architectures did not vary significantly among architectures on the dataset with added noise. However, accuracy of various architectures did differ significantly when applied to noise with absent speech.

An Improved Unsupervised Single‐Channel Speech Separation Algorithm for Processing Speech Sensor Signals

As network supporting devices and sensors in the Internet of Things are leaping forward, countless real‐world data will be generated for human intelligent applications. Speech sensor networks, an important part of the Internet of Things, have numerous application needs. Indeed, the sensor data can further help intelligent applications to provide higher quality services, whereas this data may involve considerable noise data. Accordingly, speech signal processing method should be urgently implemented to acquire low‐noise and effective speech data. Blind source separation and enhancement technique refer to one of the representative methods. However, in the unsupervised complex environment, in the only presence of a single‐channel signal, many technical challenges are imposed on achieving single‐channel and multiperson mixed speech separation. For this reason, this study develops an unsupervised speech separation method CNMF+JADE, i.e., a hybrid method combined with Convolutional Non‐Negative Matrix Factorization and Joint Approximative Diagonalization of Eigenmatrix. Moreover, an adaptive wavelet transform‐based speech enhancement technique is proposed, capable of adaptively and effectively enhancing the separated speech signal. The proposed method is aimed at yielding a general and efficient speech processing algorithm for the data acquired by speech sensors. As revealed from the experimental results, in the TIMIT speech sources, the proposed method can effectively extract the target speaker from the mixed speech with a tiny training sample. The algorithm is highly general and robust, capable of technically supporting the processing of speech signal acquired by most speech sensors.

The LSTM Neural Network Based on Memristor

The recurrent neural network adds the concept of time series on the basis of the traditional multi-layer feedforward neural networks, provides the memory function, and makes the network show good modeling ability on time-series data. Therefore, this paper proposes a LSTM (Long Short-Term Memory) neural network based on memristor. It establishes a discrete weighted LSTM network model by simplifying the traditional recurrent neural network, and uses memristor arrays on the premise of ensuring recognition performance. We realize the function of weight matrix to improve the structure of LSTM neural network, and finally carry out simulation research on the proposed neural network. And due to the volatility and yield of memristors, this paper also demonstrates and analyzes the impact of these two characteristics on network performance, and the performance level of the LSTM neural network based on memristor is verified under the existing preparation level. Experiments on the TIMIT speech database show that the proposed neural network in this paper has good accuracy and its speech recognition performance is superior.

Automatic text-independent speaker verification using convolutional deep belief network

This paper is devoted to the use of the convolutional deep belief network as a speech feature extractor for automatic text-independent speaker verification. The paper describes the scope and problems of automatic speaker verification systems. Types of modern speaker verification systems and types of speech features used in speaker verification systems are considered. The structure and learning algorithm of convolutional deep belief networks is described. The use of speech features extracted from three layers of a trained convolution deep belief network is proposed. Experimental studies of the proposed features were performed on two speech corpora: own speech corpus including audio recordings of 50 speakers and TIMIT speech corpus including audio recordings of 630 speakers. The accuracy of the proposed features was assessed using different types of classifiers. Direct use of these features did not increase the accuracy compared to the use of traditional spectral speech features, such as mel-frequency cepstral coefficients. However, the use of these features in the classifiers ensemble made it possible to achieve a reduction of the equal error rate to 0.21% on 50-speaker speech corpus and to 0.23% on the TIMIT speech corpus.

High level speaker specific features modeling in automatic speaker recognition system

Spoken words convey several levels of information. At the primary level, the speech conveys words or spoken messages, but at the secondary level, the speech also reveals information about the speakers. This work is based on the high-level speaker-specific features on statistical speaker modeling techniques that express the characteristic sound of the human voice. Using Hidden Markov model (HMM), Gaussian mixture model (GMM), and Linear Discriminant Analysis (LDA) models build Automatic Speaker Recognition (ASR) system that are computational inexpensive can recognize speakers regardless of what is said. The performance of the ASR system is evaluated for clear speech to a wide range of speech quality using a standard TIMIT speech corpus. The ASR efficiency of HMM, GMM, and LDA based modeling technique are 98.8%, 99.1%, and 98.6% and Equal Error Rate (EER) is 4.5%, 4.4% and 4.55% respectively. The EER improvement of GMM modeling technique based ASR systemcompared with HMM and LDA is 4.25% and 8.51% respectively.

VEP Detection for Read, Extempore and Conversation Speech

ABSTRACT In this paper, we propose a novel approach for accurate detection of the vowel end points (VEPs) in any mode of speech. VEP is the instant at which the vowel ends in the speech signal. In this study, we have considered three broad modes of speech, namely; conversation, extempore, and read. The existing methods were explored the VEP detection for read mode of speech, and it may not be appropriate for the VEP detection in extempore and conversation modes. This is due to the acoustic characteristic of read mode is very different from the modes as mentioned earlier. To handle this problem, we proposed a two-stage method for accurately detecting the VEPs, irrespective of modes. At the first stage, vowel onset points (VOPs) are detected in a speech signal using our recent method based on continuous wavelet transform and phone boundary. VOP represents the start of the vowel in the speech signal. At the second stage, phone boundaries are detected using spectral transition measure approach, and then the closest succeeding phone boundary for each detected VOP is considered as detected VEP. Experiments involve TIMIT and Bengali speech corpora. Performance of the proposed VEP detection method is compared with two state-of-the-art signal processing methods. The significance of the proposed method is shown by automatically detecting vowel regions from the TIMIT and Bengali speech corpora. The evaluation results report that the performance of the proposed method is significantly better than the existing methods.

Maximum Feasible Subsystem Algorithms for Recovery of Compressively Sensed Speech

The goal in signal compression is to reduce the size of the input signal without a significant loss in the quality of the recovered signal. One way to achieve this goal is to apply the principles of compressive sensing, but this has not been particularly successful for real-world signals that are insufficiently sparse, such as speech. We present three new algorithms based on solutions for the maximum feasible subsystem problem (MAX FS) that improve on the state of the art in recovery of compressed speech signals: more highly compressed signals can be successfully recovered with greater quality. The new recovery algorithms deliver sparser solutions when compared with those obtained using traditional compressive sensing recovery algorithms. When tested by recovering compressively sensed speech signals in the TIMIT speech database, the recovered speech has better perceptual quality than speech recovered using traditional compressive sensing recovery algorithms.

Maximum entropy PLDA for robust speaker recognition under speech coding distortion

The system combining i-vector and probabilistic linear discriminant analysis (PLDA) has been applied with great success in the speaker recognition task. The i-vector space gives a low-dimensional representation of a speech segment and training data of a PLDA model, which offers greater robustness under different conditions. In this paper, we propose a new framework based on i-vector/PLDA and Maximum Entropy (ME) to improve the performance of speaker identification system in the presence of speech coding distortion. The results are reported on TIMIT database and speech coding obtained by passing the speech test from TIMIT database through the AMR encoder/decoder. Our results show that the proposed methode achieves improved performance when compared with the i-vector/PLDA and MEGMM.

Nonlinear waveform distortion: Assessment and detection of clipping on speech data and systems

Speech, speaker, and language systems have traditionally relied on carefully collected speech material for training acoustic models. There is an overwhelming abundance of publicly accessible audio material available for training. A major challenge, however, is that such found data is not professionally recorded, and therefore may contain a wide diversity of background noise, nonlinear distortions, or other unknown environmental or technology-based contamination or mismatch. There is a critical need for automatic analysis to screen such unknown data sets before acoustic model development training, or to perform input audio purity screening prior to classification. In this study, we propose a waveform based clipping detection algorithm for naturalistic audio streams and analyze the impact of clipping at different severities on speech quality measures and automatic speaker identification systems. We use the TIMIT and NIST-SRE08 speech corpora as case studies. The results show, as expected, that clipping introduces a nonlinear distortion into clean speech data, which reduces speech quality and performance for speaker recognition. We also investigate what degree of clipping can be present to sustain effective speech system performance. The proposed detection system, which will be released, could potentially contribute to new audio collections for speech and language technology development.

Identity Vector Extraction by Perceptual Wavelet Packet Entropy and Convolutional Neural Network for Voice Authentication.

Recently, the accuracy of voice authentication system has increased significantly due to the successful application of the identity vector (i-vector) model. This paper proposes a new method for i-vector extraction. In the method, a perceptual wavelet packet transform (PWPT) is designed to convert speech utterances into wavelet entropy feature vectors, and a Convolutional Neural Network (CNN) is designed to estimate the frame posteriors of the wavelet entropy feature vectors. In the end, i-vector is extracted based on those frame posteriors. TIMIT and VoxCeleb speech corpus are used for experiments and the experimental results show that the proposed method can extract appropriate i-vector which reduces the equal error rate (EER) and improve the accuracy of voice authentication system in clean and noisy environment.