Extracting structural spectral features using what-where auto-encoders for statistical parametric speech synthesis

Abstract

This paper presents a method to extract structural spectral features from spectral envelopes using what-where autoencoders (WWAE) for statistical parametric speech synthesis (SPSS). A WWAE is constructed by concatenating a convolutional net for input encoding and a deconvolutional net for reconstruction. The output values of the max-pooling layer in the encoder and the positions of the max-pooling switches are utilized as the what and where features respectively. Considering the intrinsic formant structures in the spectral envelopes of voiced speech frames, the WWAE model is adopted in this paper to detect, locate, and reconstruct the formants and other local structures in spectral envelopes. Here, the what and where features describe the prominences and positions of specific local spectral structures within a pooling frequency window. Then, the extracted what and where features are modeled as separate streams under the hidden Markov model (HMM)-based SPSS framework. Experimental results show that the speech synthesis system built using our proposed spectral features can produce synthetic speech with sharper formant structures and better naturalness than the systems using mel-cepstra and conventional auto-encoder-based spectral features.