Speech Waveform Research Articles

Speech Emotion Classification Using Attention-Based LSTM

Automatic speech emotion recognition has been a research hotspot in the field of human–computer interaction over the past decade. However, due to the lack of research on the inherent temporal relationship of the speech waveform, the current recognition accuracy needs improvement. To make full use of the difference of emotional saturation between time frames, a novel method is proposed for speech recognition using frame-level speech features combined with attention-based long short-term memory (LSTM) recurrent neural networks. Frame-level speech features were extracted from waveform to replace traditional statistical features, which could preserve the timing relations in the original speech through the sequence of frames. To distinguish emotional saturation in different frames, two improvement strategies are proposed for LSTM based on the attention mechanism: first, the algorithm reduces the computational complexity by modifying the forgetting gate of traditional LSTM without sacrificing performance and second, in the final output of the LSTM, an attention mechanism is applied to both the time and the feature dimension to obtain the information related to the task, rather than using the output from the last iteration of the traditional algorithm. Extensive experiments on the CASIA, eNTERFACE, and GEMEP emotion corpora demonstrate that the performance of the proposed approach is able to outperform the state-of-the-art algorithms reported to date.

ConflictNET: End-to-End Learning for Speech-Based Conflict Intensity Estimation

Computational paralinguistics aims to infer human emotions, personality traits and behavioural patterns from speech signals. In particular, verbal conflict is an important example of human-interaction behaviour, whose detection would enable monitoring and feedback in a variety of applications. The majority of methods for detection and intensity estimation of verbal conflict apply off-the-shelf classifiers/regressors to generic hand-crafted acoustic features. Generating conflict-specific features requires refinement steps and the availability of metadata, such as the number of speakers and their speech overlap duration. Moreover, most techniques treat feature extraction and regression as independent modules, which require separate training and parameter tuning. To address these limitations, we propose the first end-to-end convolutional-recurrent neural network architecture that learns conflict-specific features directly from raw speech waveforms, without using explicit domain knowledge or metadata. Additionally, to selectively focus the model on portions of speech containing verbal conflict instances, we include a global attention interface that learns the alignment between layers of the recurrent network. Experimental results on the SSPNet Conflict Corpus show that our end-to-end architecture achieves state-of-the-art performance in terms of Pearson Correlation Coefficient.

High Throughput CORDIC Architecture Based 3D-DCT/IDCT Processor

In this paper, 8x8x8 3D-DCT/IDCT processor based on CORDIC architecture for high data rate of image processing and video coding with reduced hardware has been presented. It uses two stages of CORDIC DCT processor i.e. one is 1D-DCT and another one is 2D-DCT processor based on fully pipelined unfolded CORDIC architecture with RAM buffer. The one-dimensional DCT is useful in processing of speech waveforms. For images and video signal processing, we need a 3D version of the DCT/IDCT data, especially in coding for compression and decompression, for its best performance. This processor performs both the DCT and IDCT simultaneously with the help of CORDIC algorithm of both 3D-DCT and IDCT which has boost the speed of the processor. Ease the use of less computation based unfolded CORDIC architecture in the processor reduces complexity and power consumption. With the working frequency approximately 414 MHz less power dissipation, low latency and high throughput can be obtained. The processor has been implemented on Xilinx ISE 14.7 and design is simulated in DE-2 board. This processor has not only reduces the complication but also increases the speed which lead to broad use in image and video processing.

Calibrating rhythms in L1 Japanese and Japanese accented English

Japanese and Japanese-accented English utterances produced by native Japanese speakers were classified using rhythm spectra automatically derived from the speech signals. Results of a stepwise discriminant function analysis revealed an overall classification rate of 81.9% (chance = 50%) for Japanese and 85.9% for English utterances. Two parameters extracted from the spectra emerged as the strongest predictors: normalized amplitude modulation rate above 4 Hz in the 1,000 Hz octave band frequency and normalized amplitude modulation rate between 3-6 Hz in the 4,000 Hz octave band. These results suggested that native Japanese speakers produce Japanese and English with different patterns of temporal regularities in the amplitude envelope of the speech waveform.

Speech enhancement based on simple recurrent unit network

Speech enhancement is a crucial and challenging task in many applications. A novel speech enhancement method based on the simple recurrent unit (SRU) is proposed in this paper. First, the log-power spectra of noisy and clean speeches are extracted. Then, the mapping relationship between noisy and clean speech spectra is learned by a multiple-layer stacked SRU network. Finally, the well-trained model is used to predict the corresponding clean speech spectra from the noisy speech spectra and the whole clean speech waveform can be recovered. Compared with the existing algorithms, DNN, LSTM and GRU, the proposed method achieves significant improvements at training speed and has capability to balance the performance and the training time. Experimental results demonstrate the validity and robustness of the proposed method.

Towards emotion recognition from contextual information using machine learning

Emotions influence cognitive processes that underlie human behavior. Whereas experiencing negative emotions may lead to develop psychological disorders, experiencing positive emotions may improve creative thinking and promote cooperative behavior. The importance of human emotions has led to the development of automatic emotion recognition systems based on analysis of speech waveforms, facial expressions, and physiological signals as well as text data mining. However, emotions are associated with a context (in which emotions are actually experienced), hence, this work focuses on emotion recognition from contextual information. In this paper, we present a study aimed to assess the feasibility of automatically recognizing emotions from individuals’ contexts. In this study, 32 participants provided information using a mobile application about their emotions and the context (e.g., companions, activities, and locations) in which these emotions were experienced. We used machine learning techniques to build individual models, general models, and gender-specific models to automatically recognize emotions of participants. The empirical results show that individuals’ emotions are highly related to their context and that automatic recognition of emotions in real-world situations is feasible by using contextual data.

Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input

In this paper, we explore neural network models that learn to associate segments of spoken audio captions with the semantically relevant portions of natural images that they refer to. We demonstrate that these audio-visual associative localizations emerge from network-internal representations learned as a by-product of training to perform an image-audio retrieval task. Our models operate directly on the image pixels and speech waveform, and do not rely on any conventional supervision in the form of labels, segmentations, or alignments between the modalities during training. We perform analysis using the Places 205 and ADE20k datasets demonstrating that our models implicitly learn semantically coupled object and word detectors.

GlotNet—A Raw Waveform Model for the Glottal Excitation in Statistical Parametric Speech Synthesis

Recently, generative neural network models which operate directly on raw audio, such as WaveNet, have improved the state of the art in text-to-speech synthesis TTS. Moreover, there is increasing interest in using these models as statistical vocoders for generating speech waveforms from various acoustic features. However, there is also a need to reduce the model complexity, without compromising the synthesis quality. Previously, glottal pulseforms i.e., time-domain waveforms corresponding to the source of human voice production mechanism have been successfully synthesized in TTS by glottal vocoders using straightforward deep feedforward neural networks. Therefore, it is natural to extend the glottal waveform modeling domain to use the more powerful WaveNet-like architecture. Furthermore, due to their inherent simplicity, glottal excitation waveforms permit scaling down the waveform generator architecture. In this study, we present a raw waveform glottal excitation model, called GlotNet, and compare its performance with the corresponding direct speech waveform model, WaveNet, using equivalent architectures. The models are evaluated as part of a statistical parametric TTS system. Listening test results show that both approaches are rated highly in voice similarity to the target speaker, and obtain similar quality ratings with large models. Furthermore, when the model size is reduced, the quality degradation is less severe for GlotNet.

Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation.

Single-channel, speaker-independent speech separation methods have recently seen great progress. However, the accuracy, latency, and computational cost of such methods remain insufficient. The majority of the previous methods have formulated the separation problem through the time-frequency representation of the mixed signal, which has several drawbacks, including the decoupling of the phase and magnitude of the signal, the suboptimality of time-frequency representation for speech separation, and the long latency of the entire system. To address these shortcomings, we propose a fully-convolutional time-domain audio separation network (Conv-TasNet), a deep learning framework for end-to-end time-domain speech separation. Conv-TasNet uses a linear encoder to generate a representation of the speech waveform optimized for separating individual speakers. Speaker separation is achieved by applying a set of weighting functions (masks) to the encoder output. The modified encoder representations are then inverted back to the waveforms using a linear decoder. The masks are found using a temporal convolutional network (TCN) consisting of stacked 1-D dilated convolutional blocks, which allows the network to model the long-term dependencies of the speech signal while maintaining a small model size. The proposed Conv-TasNet system significantly outperforms previous time-frequency masking methods in separating two- and three-speaker mixtures. Additionally, Conv-TasNet surpasses several ideal time-frequency magnitude masks in two-speaker speech separation as evaluated by both objective distortion measures and subjective quality assessment by human listeners. Finally, Conv-TasNet has a significantly smaller model size and a much shorter minimum latency, making it a suitable solution for both offline and real-time speech separation applications. This study therefore represents a major step toward the realization of speech separation systems for real-world speech processing technologies.

Modification of Velvet Noise for Speech Waveform Generation by Using Vocoder-Based Speech Synthesizer

Modification of Velvet Noise for Speech Waveform Generation by Using Vocoder-Based Speech Synthesizer

Phoneme boundary detection from speech: A rule based approach

In this paper, a novel approach has been proposed for the automatic segmentation of speech signal into phonemes. In a well spoken word, phonemes can be characterized by the changes observed in speech waveform. To get phoneme boundaries, the signal level properties of speech waveform i.e. changes in the waveform during transformation from one phoneme to the other are explored. The problem of phoneme level segmentation has been addressed in this work from two aspects 1. Segmentation of phonemes between voiced and unvoiced portions and 2. Segmentation of phonemes within voiced and unvoiced regions. Pitch and zero-frequency filter signal are used to get the region of change from voiced to unvoiced and vice versa. The segmentation of phoneme boundaries within voiced and unvoiced regions are approximated using the properties of power spectrum of correlation of adjacent frames of the signal. A finite set of rules is proposed on the variations observed in the power spectrum during phoneme transitions. The segmentation results of both approaches are combined to get the final phoneme boundaries. Three databases namely TIMIT Corpus, IIIT Hyderabad Marathi database & IIIT Hyderabad Hindi database (IIIT-H Indic Speech Databases) are used to test the proposed approach; an accuracy of 95.40%, 96.87% and 96.12% is achieved within the tolerance range of 10 ms respectively. The results of the proposed approach are observed to give precise phoneme boundaries.

Voice Conversion With CycleRNN-Based Spectral Mapping and Finely Tuned WaveNet Vocoder

In this paper, we present a novel framework for a voice conversion (VC) system based on a cyclic recurrent neural network (CycleRNN) and a finely tuned WaveNet vocoder. Even though WaveNet is capable of producing natural speech waveforms when fed with natural speech features, it still suffers from speech quality degradation when fed with oversmoothed features, such as spectral parameters estimated from a statistical model. One way to address this problem is to introduce oversmoothed features while developing a WaveNet model. However, in a VC framework, providing oversmoothed spectral features of a target speaker for WaveNet modeling is not straightforward owing to the difference in the time-sequence alignment from that of a source speaker. To overcome this problem, we propose the use of a cyclic spectral conversion network, i.e., CycleRNN, capable of performing a conversion flow, i.e., source-to-target, and a cyclic flow, i.e., to generate self-predicted target spectra. The CycleRNN spectral model is trained using both conversion and weighted cyclic losses. To finely tune WaveNet, a pretrained multispeaker WaveNet model is optimized using the self-predicted features of the corresponding target speaker of a speaker conversion pair. The experimental results demonstrate that 1) the proposed CycleRNN-based spectral model for WaveNet fine-tuning significantly improves the naturalness of the converted speech waveforms, giving an overall mean opinion score of 3.50; and 2) the proposed model yields the highest speaker conversion accuracy, giving an overall speaker similarity score of 78.33%, which is a significant improvement compared with conventional WaveNet fine-tuning using natural target features.

Vocal Effort Based Speaking Style Conversion Using Vocoder Features and Parallel Learning

Speaking style conversion (SSC) is the technology of converting natural speech signals from one style to another. In this study, we aim to provide a general SSC system for converting styles with varying vocal effort and focus on normal-to-Lombard conversion as a case study of this problem. We propose a parametric approach that uses a vocoder to extract speech features. These features are mapped using parallel machine learning models from utterances spoken in normal style to the corresponding features of Lombard speech. Finally, the mapped features are converted to a Lombard speech waveform with the vocoder. A total of three vocoders (GlottDNN, STRAIGHT, and Pulse model in log domain (PML)) and three machine learning mapping methods (standard GMM, Bayesian GMM, and feed-forward DNN) were compared in the proposed normal-to-Lombard style conversion system. The conversion was evaluated using two subjective listening tests measuring perceived Lombardness and quality of the converted speech signals, and by using an instrumental measure called Speech Intelligibility in Bits (SIIB) for speech intelligibility evaluation under various noise levels. The results of the subjective tests show that the system is able to convert normal speech into Lombard speech and that there is a trade-off between quality and Lombardness of the mapped utterances. The GlottDNN and PML stand out as the best vocoders in terms of quality and Lombardness, respectively, whereas the DNN is the best mapping method in terms of Lombardness. PML with the standard GMM seems to give a good compromise between the two attributes. The SIIB experiments indicate that intelligibility of converted speech compared to that of normal speech improved in noisy conditions most effectively when DNN mapping was used with STRAIGHT and PML.

Hidden-Markov-model based statistical parametric speech synthesis for Marathi with optimal number of hidden states

Hidden Markov Model and Deep Neural Networks based Statistical Parametric Speech Synthesis systems, gain a significant attention from researchers because of their flexibility in generating speech waveforms in diverse voice qualities as well as in styles. This paper describes HMM-based speech synthesis system (SPSS) for the Marathi language. In proposed synthesis method, speech parameter trajectories used for synthesis are generated from the trained hidden Markov models (HMM). We have recorded our database of 5300 phonetically balanced Marathi sentences to train the context-dependent HMM with five, seven and nine hidden states. The subjective quality measures (MOS and PWP) shows that the HMMs with seven hidden states are capable of giving an adequate quality of synthesized speech as compared to five state and with less time complexity than seven state HMMs. The contextual features used for experimentation are inclusive of a position of an observed phoneme in a respective syllable, word, and sentence.

HMM speech synthesis based on MDCT representation

Hidden Markov model (HMM) based text-to-speech (TTS) has become one of the most promising approaches, as it has proven to be a particularly flexible and robust framework to generate synthetic speech. However, several factors such as mel-cepstral vocoder and over-smoothing are responsible for causing quality degradation of synthetic speech. This paper presents an HMM speech synthesis technique based on the modified discrete cosine transform (MDCT) representation to cope with these two issues. To this end, we use an analysis/synthesis technique based on MDCT that guarantees a perfect reconstruction of the signal frame from feature vectors and allows for a 50% overlap between frames without increasing the data vector, in contrast to the conventional mel-cepstral spectral parameters that do not ensure direct speech waveform reconstruction. Experimental results show that a sound of good quality, conveniently evaluated using both objective and subjective tests, is obtained.

A Comparison Between STRAIGHT, Glottal, and Sinusoidal Vocoding in Statistical Parametric Speech Synthesis

A vocoder is used to express a speech waveform with a controllable parametric representation that can be converted back into a speech waveform. Vocoders representing their main categories (mixed excitation, glottal, and sinusoidal vocoders) were compared in this study with formal and crowd-sourced listening tests. The vocoder quality was measured within the context of analysis–synthesis as well as text-to-speech (TTS) synthesis in a modern statistical parametric speech synthesis framework. Furthermore, the TTS experiments were divided into synthesis with vocoder-specific features and synthesis with a shared envelope model, where the waveform generation method of the vocoders is mainly responsible for the quality differences. Finally, all of the tests included four distinct voices as a way to investigate the effect of different speakers on the synthesized speech quality. The obtained results suggest that the choice of the voice has a profound impact on the overall quality of the vocoder-generated speech, and the best vocoder for each voice can vary case by case. The single best-rated TTS system was obtained with the glottal vocoder GlottDNN using a male voice with low expressiveness. However, the results indicate that the sinusoidal vocoder PML (pulse model in log-domain) has the best overall performance across the performed tests. Finally, when controlling for the spectral models of the vocoders, the observed differences are similar to the baseline results. This indicates that the waveform generation method of a vocoder is essential for quality improvements.

Speech Enhancement Using Heterogeneous Information

This article describes how to use heterogeneous information in speech enhancement. In most of the current speech enhancement systems, clean speeches are recovered only from the signals collected by acoustic microphones, which will be greatly affected by the acoustic noises. However, heterogeneous information from different kinds of sensors, which is usually called the “multi-stream,” are seldom used in speech enhancement because the speech waveforms cannot be recovered from the signals provided by many kinds of sensors. In this article, the authors propose a new model-based multi-stream speech enhancement framework that can make use of the heterogeneous information provided by the signals from different kinds of sensors even when some of them are not directly related to the speech waveform. Then a new speech enhancement scheme using the acoustic and throat microphone recordings is also proposed based on the new speech enhancement framework. Experimental results show that the proposed scheme outperforms several single-stream speech enhancement methods in different noisy environments.

Mel-Cepstrum-Based Quantization Noise Shaping Applied to Neural-Network-Based Speech Waveform Synthesis

This paper presents a mel-cepstrum-based quantization noise shaping method for improving the quality of synthetic speech generated by neural-network-based speech waveform synthesis systems. Since mel-cepstral coefficients closely match the characteristics of human auditory perception, the proposed method effectively masks the white noise introduced by the quantization typically used in neural-network-based speech waveform synthesis systems. The paper also describes a computationally efficient implementation of the proposed method using the structure of the mel-log spectrum approximation filter. Experiments using the WaveNet generative model, which is a state-of-the-art model for neural-network-based speech waveform synthesis, showed that speech quality is significantly improved by the proposed method.

Patterns of Aided Loudness Growth in Experienced Adult Listeners with Early-Onset Severe-Profound Hearing Loss.

Individuals with early-onset severe-profound bilateral hearing loss (S/PHL) manifest diverse levels of benefit and satisfaction with hearing aids (HAs), even with prescriptive HA fitting. Such fittings incorporate normal loudness values, but little is known about aided loudness outcomes in this population and how those outcomes affect benefit or satisfaction. To describe aided loudness growth and satisfaction with aided listening in experienced adult HA users with S/PHL. The Contour Test of loudness perception was administered to listeners with S/PHL in the aided sound field using broadband speech, band-limited speech, and warble tones. Patterns and slopes of resultant loudness growth functions were referenced to sound field results from listeners with normal hearing (NH). S/PHL listeners also rated their aided listening satisfaction. It was expected that (1) most S/PHL listeners would demonstrate steeper than normal aided loudness growth, (2) loudness normalization would be associated with better high-frequency detection thresholds and speech recognition, and (3) closer approximation to normal would yield greater satisfaction. Participants were paid college-student volunteers: 23 with S/PHL, long-term aided listening experience, and new HAs; 15 with NH. Participants rated loudness on four ascending runs per stimulus (5-dB increments) using categories defined in 1997 by Cox and colleagues. The region between the 10th and 90th percentiles of the NH distribution constituted local norms against which location and slope of the S/PHL functions were examined over the range from Quiet to Loud-but-OK. S/PHL functions were categorized on the basis of their configurations (locations/slopes) relative to the norms. Pattern of aided loudness was normalized or within 5 dB of the normal region on 37% of trials with sufficient data for analysis. Only one of the 23 S/PHL listeners did not demonstrate Normal/Near-normal loudness on any trials. Four nonnormal patterns were identified: Steep (recruitment-like; 38% of trials); Shifted right, with normal growth rate (10%); Hypersensitive, with most intensities louder than normal (10%); and Shallow, with decreasing growth rate (7%). Listeners with high-frequency average thresholds above 100 dB hearing loss or no phonemic-based speech-discrimination skill were less likely to display normalized loudness. Slope was within norms for 52% of S/PHL trials, most also having a Normal/Near-normal growth pattern. Regardless of measured loudness results, all but four listeners with S/PHL reported satisfactory hearing almost always or most of the time with their HAs in designated priority need areas. The variety of aided loudness growth patterns identified reflects the diversity known to characterize individuals with early-onset S/PHL. Loudness rating at the validation stage of HA fit with these listeners is likely to reveal nonnormal loudness, signaling need for further HA adjustment. High satisfaction, however, despite nonnormal loudness growth, suggests that listeners with poor auditory speech recognition may benefit more from aided loudness that supports pattern perception (via the time-intensity waveform of speech), different from most current-day prescription fits.

Automated depression analysis using convolutional neural networks from speech

To help clinicians to efficiently diagnose the severity of a person’s depression, the affective computing community and the artificial intelligence field have shown a growing interest in designing automated systems. The speech features have useful information for the diagnosis of depression. However, manually designing and domain knowledge are still important for the selection of the feature, which makes the process labor consuming and subjective. In recent years, deep-learned features based on neural networks have shown superior performance to hand-crafted features in various areas. In this paper, to overcome the difficulties mentioned above, we propose a combination of hand-crafted and deep-learned features which can effectively measure the severity of depression from speech. In the proposed method, Deep Convolutional Neural Networks (DCNN) are firstly built to learn deep-learned features from spectrograms and raw speech waveforms. Then we manually extract the state-of-the-art texture descriptors named median robust extended local binary patterns (MRELBP) from spectrograms. To capture the complementary information within the hand-crafted features and deep-learned features, we propose joint fine-tuning layers to combine the raw and spectrogram DCNN to boost the depression recognition performance. Moreover, to address the problems with small samples, a data augmentation method was proposed. Experiments conducted on AVEC2013 and AVEC2014 depression databases show that our approach is robust and effective for the diagnosis of depression when compared to state-of-the-art audio-based methods.