Acoustic Modeling Techniques Research Articles

Improved Recognition of Spontaneous Hungarian Speech—Morphological and Acoustic Modeling Techniques for a Less Resourced Task

Various morphological and acoustic modeling techniques are evaluated on a less resourced, spontaneous Hungarian large-vocabulary continuous speech recognition (LVCSR) task. Among morphologically rich languages, Hungarian is known for its agglutinative, inflective nature that increases the data sparseness caused by a relatively small training database. Although Hungarian spelling is considered as simple phonological, a large part of the corpus is covered by words pronounced in multiple, phonemically different ways. Data-driven and language specific knowledge supported vocabulary decomposition methods are investigated in combination with phoneme- and grapheme-based acoustic modeling techniques on the given task. Word baseline and morph-based advanced baseline results are significantly outperformed by using both statistical and grammatical vocabulary decomposition methods. Although the discussed morph-based techniques recognize a significant amount of out of vocabulary words, the improvements are due not to this fact but to the reduction of insertion errors. Applying grapheme-based acoustic models instead of phoneme-based models causes no severe recognition performance deteriorations. Moreover, a fully data-driven acoustic modeling technique along with a statistical morphological modeling approach provides the best performance on the most difficult test set. The overall best speech recognition performance is obtained by using a novel word to morph decomposition technique that combines grammatical and unsupervised statistical segmentation algorithms. The improvement achieved by the proposed technique is stable across acoustic modeling approaches and larger with speaker adaptation.

Modeling the acoustic channel for mixed pressure and particle velocity sensors.

Looking at our repertoire of acoustic modeling techniques, ray tracing and Gaussian beams provide a good match to modeling the underwater channel for acoustic communications, given their fundamentally broadband nature. We will discuss how we have augmented existing acoustic models, including Gaussian beam and normal mode codes, to produce particle velocity predictions. In both of these cases, the modifications consisted of manipulating the components used to represent the pressure field, as opposed to relying upon a finite difference approximation of the predicted field itself. We will discuss further modifications to these codes that enable the dynamic features of the channel to be modeled, including source and receiver motion, and motion of the ocean surface due to ocean surface waves. It is important to model realizations of the channel, rather than averaged representations, because many of the high-speed, coherent modulation schemes that are the focus of continuing research use tracking loops in their design. These tracking loops must be stressed with realistically time-evolving channels. Acoustic channel predictions will be compared with experiment data from acoustic communications experiments off the coast of Kauai in Hawaii.

Perceptually relevant parameters for virtual listening simulation of small room acoustics.

Various physical aspects of room-acoustic simulation techniques have been extensively studied and refined, yet the perceptual attributes of the simulations have received relatively little attention. Here a method of evaluating the perceptual similarity between rooms is described and tested using 15 small-room simulations based on binaural room impulse responses (BRIRs) either measured from a real room or estimated using simple geometrical acoustic modeling techniques. Room size and surface absorption properties were varied, along with aspects of the virtual simulation including the use of individualized head-related transfer function (HRTF) measurements for spatial rendering. Although differences between BRIRs were evident in a variety of physical parameters, a multidimensional scaling analysis revealed that when at-the-ear signal levels were held constant, the rooms differed along just two perceptual dimensions: one related to reverberation time (T(60)) and one related to interaural coherence (IACC). Modeled rooms were found to differ from measured rooms in this perceptual space, but the differences were relatively small and should be easily correctable through adjustment of T(60) and IACC in the model outputs. Results further suggest that spatial rendering using individualized HRTFs offers little benefit over nonindividualized HRTF rendering for room simulation applications where source direction is fixed.

Three-dimensional rough surface scattering using finite elements.

In order to quantify the effects of three dimensional scattering, the scattering of a spherical incident wave from a rough, pressure release surface is modeled using a commercially available finite element (FE) code. The surface is generated by creating random realizations from a spatial power spectrum measured as part of the experimental validation of acoustic modeling techniques sea test conducted off the coast of Isola d’Elba in 2006. Since the FE model approaches an exact solution as the discretization density increases, it can be used as a benchmark for approximate methods. Therefore, the three‐dimensional (3‐D) FE model will be compared with a 3‐D Kirchhoff approximation solution as well as two‐dimensional solutions based on the integral equation, Kirchhoff, and FE models in order to determine the range of validity of the approximate methods.

The effects of scattering and poro-elasticity in reflection coefficient measurements.

Reflection coefficient measurements taken as part of the experimental validation of acoustic modeling techniques experiment in October 2006 off the coast of Isola d’Elba, Italy display frequency dependent behavior that cannot be described using elastic reflection models and spherical wave effects alone. Furthermore, analysis of the effects of scattering based on measured interface roughness does not predict the critical and supercritical angle behavior. Therefore, a poro‐elastic model is developed to account for the frequency dependent behavior. A comparison of the model including poro‐elasticity, spherical wave effects, and scattering with the data will be presented. [Work sponsored by Office of Naval Research, Ocean Acoustics.]

Investigation of an unusual noise phenomenon with horizontal and vertical hydrophone array data and three‐dimensional propagation modeling.

Acoustic data from an experiment conducted during Shallow Water 2006 (SW06) showed an unexplained broadband noise phenomenon. While the R/V Knorr towed a J‐15 acoustic source, which emitted a 93 Hz continuous‐wave signal, the R/V Endeavor performed Scanfish measurements in the vicinity to characterize internal wave phenomena. The WHOI Shark horizontal and vertical hydrophone array detected the 93 Hz signal, but eventually the signal was overpowered by broadband low‐frequency noise. The broadband noise may be associated with either research vessel but is uncharacteristically intense given that both vessels were greater than 25 km away from the Shark array. The complex acoustic environment due to the shelfbreak front, an internal wave packet, and the bathymetry of the continental shelf may have caused ducting of the noise field. Both three‐dimensional acoustic modeling and array processing techniques will be used to characterize the unexplained noise levels. In situ data from temperature sensors and Scanfish measurements provide environmental information needed to accurately model the sound speed field. These methods can also be applied to other SW06 experiments that were tailored to investigate the role of internal waves in acoustic propagation—such as those carried out on the R/V Sharp. [Work sponsored by the Office of Naval Research.]

The effects of roughness on the frequency dependence of specular scattering.

Specular scattering data were collected at grazing angles from 7 to 77 deg grazing and frequencies from 5 to 50 kHz at the experimental validation of acoustic modeling techniques (EVA) sea test near Isola in October 2006. High resolution microbathymetry data taken using a laser line scan system were also collected. The amplitude and phase of the specular scattered data exhibit a distinctive frequency dependence. This frequency dependence may be due to Bragg scattering from small interface ripples on the ocean bottom indicated by peaks in the measured interface roughness power spectrum. This study simulates the scattering using realizations of interface roughness in a finite element model to determine if the frequency dependence is due solely to scattering or if other mechanisms such as layering are important factors. Results will be compared to analytic approximations such as the Kirchhoff approximation as well as exact integral equations. [Work sponsored by the ONR, Ocean Acoustics.]

Perceived plausibility of a multi‐modal musical performance with introduced auditory‐visual spatial and temporal mismatch

One of the biggest problems for multi‐modal virtual musical performance is that of auditory‐visual mismatch. Often such performances place musicians in a dry, studio‐like environment and rely on acoustic room modeling techniques to place the musician's sound in an environment more conducive to musical performance. The problem, for example occurs when the musician sounds as if he/she is in a large performance space but from a visual standpoint appears to be located in a very small studio environment. This experiment aimed to reveal the impact of such mismatches on the perceived plausibility of a multi‐modal musical performance. Listeners were confronted with three contrasting musical excerpts presented in a virtual environment with varying degrees of auditory visual mismatch. In the first phase, listeners were able to adjust the acoustic modeling algorithm using the salient parameters of direct to reverberant ratio and reverberation time. As the visual space volume that the performance was presented in increased, the listeners repeatedly increased reverberation time and decreased direct to reverberant ratio. During the second phase, where varying levels of spatial and temporal mismatch were purposefully introduced, the level of repeatability and accuracy of the listening group decreased despite subjective responses indicating accurate perceived performance realism.

Bayesian inference in hierarchical non‐negative matrix factorisation models of musical sounds

There has been a surge of interest to efficient audio and music modeling using tools from statistical machine learning. One such technique, that has been particularly successful, is non‐negative matrix factorisation (NMF). However, a detailed theoretical understanding of this success is missing, as well as links to other modeling strategies such as sinusoidal or harmonic models. To fill this gap, we describe NMF in a statistical framework, as a hierarchical generative model consisting of an observation and a prior component. We show that particular choices lead to standard NMF algorithms as special cases, where parameter estimation is carried out via maximum likelihood. Starting from this view, we develop extensions that facilitate more realistic acoustic modeling (such as spectral smoothness or harmonicity of natural sounds) and alternative inference techniques via Gibbs sampling and variational Bayes, which allow us to do principled comparisons between alternative models via Bayesian model selection. Our novel construction, where we make use of Markov chains of Gamma random variables, retains conjugacy and enables us to develop models that fit better to real data while retaining attractive features of standard NMF such as fast convergence and easy implementation. We illustrate our approach on polyphonic pitch estimation.

Robust recognition and characterization of man-made objects in shallow water using time-frequency analysis

For underwater sonar, time-frequency analysis, in particular Wigner-Ville analysis, has been shown to be a relevant tool for discriminating a man made target (shell) from a natural one of the same shape (solid) and even to estimate some target characteristics (shell thickness, shear velocity...). This processing tool takes advantage of the evolutional, time dependent aspect of the echo spectrum. The estimated time-frequency patterns can be used for detection and wideband classification of sonar echoes in order to reduce false alarms. In particular, the so-called “coincidence pattern” appearing for specific frequency range is a robust time-frequency signature of man-made shells. A time-frequency analysis will be presented to understand echo formation mechanisms using a standard spherical shell model target model. The influence of the medium parameters as well as the source-receiver configuration will be investigated in free space and then extended to the case of a shallow water waveguide. The proposed approach will be tested using target scattering data collected during Experiments for Validation of Acoustic modeling techniques (EVA) sea test on the north shore of Isola D'Elba, Italy. Application to mine-hunting sonar systems will be discussed.

In situ reflection coefficient measurements from 5–50 kHz from the experiments for validation of acoustic modeling techniques (EVA) sea test

An accurate model of high-frequency reflection loss at the ocean bottom is critical to understanding shallow water sound propagation and the detection and classification of buried objects such as mines. The value of the observed reflection coefficient can be perturbed by such effects as scattering from a rough interface and dispersion in the ocean sediment. To obtain ground truth measurements for model development and verification, reflection coefficients from a rough water/sediment interface were measured as part of the experiments for validation of acoustic modeling techniques (EVA) sea test on the north shore of Isola dElba, Italy. Data at grazing angles from 10–80 deg and frequencies from 5–50 kHz were obtained. Experimental effects such as transducer response and spherical wave effects were quantified. The data will be analyzed to determine the influence of dispersion and scattering effects. [Work supported by ONR, Ocean Acoustics.]

Comparison of finite element model of scattering with experimental results from the Experimental Validation of Acoustic modeling techniques (EVA) sea test

Reflection coefficient measurements from 5 50 kHz at grazing angles from 10–80 degrees were taken as part of the Experiments for Validation of Acoustic modeling techniques (EVA) sea test on the north shore of Isola dElba, Italy. The microtopography of the seafloor was also measured using an ROV mounted laser line scan system. The seafloor roughness measurements were used as inputs in a finite element model to calculate the effects of scattering on the reflection coefficient measurement. The variance of the calculated reflection coefficient is compared with the measured variance to determine the contribution from interface scattering at different frequencies and angles. [Work supported by ONR, Ocean Acoustics.]

Acoustic modeling with contextual additive structure for hidden Markov model‐based speech recognition

This study proposes an acoustic modeling technique based on an additive structure of context dependencies for HMM‐based speech recognition. Context dependent models, e.g., triphone HMMs, are widely employed in speech recognition systems. Generally the dependency between phonetic contexts and acoustic feature distributions is direct, i.e., if a context is given, the Gaussian distribution is specified immediately. However, more complex dependencies can be considered. This study assumes an additive structure of acoustic feature components that have different context dependencies. For example, if the left and right phones are assumed as contexts of additive components, the generative process of the triphone feature is as follows: the additive components of left and right contexts are generated independently from each distribution and then added to the monophone feature. Since the output probability distribution is composed of the sum of the mean vectors and covariance matrices of additive components, a number of different distributions can be efficiently represented by a combination of fewer distributions. However, it is unknown which kinds of contexts have additive dependencies to acoustic features. Therefore, this study proposes a context clustering algorithm for the additive structure model. The proposed algorithm automatically extracts additive components by constructing multiple decision trees simultaneously.

Spatial Audio Measurement, Modeling and Composition

The author presents the results of an exploration of spatial composition through acoustic measurement and modeling techniques. He discusses the capture of actual acoustic spaces and the use of such captures to create virtual acoustic landscapes, and describes a series of sound-works composed using the acoustic captures.

Progress in the CU-HTK broadcast news transcription system

Broadcast news (BN) transcription has been a challenging research area for many years. In the last couple of years, the availability of large amounts of roughly transcribed acoustic training data and advanced model training techniques has offered the opportunity to greatly reduce the error rate on this task. This paper describes the design and performance of BN transcription systems which make use of these developments. First, the effects of using lightly supervised training data and advanced acoustic modeling techniques are discussed. The design of a real-time broadcast news recognition system is then detailed using these new models. As system combination has been found to yield large gains in performance, a range of frameworks that allow multiple recognition outputs to be combined are next described. These include the use of multiple types of acoustic models and multiple segmentations. As a contrast a system developed by multiple sites allowing cross-site combination, the SuperEARS system, is also described. The various models and recognition configurations are evaluated using several recent BN development and evaluation test sets. These new BN transcription systems can give gains of over 25% relative to the CU-HTK 2003 BN system

Hybrid NN/HMM acoustic modeling techniques for distributed speech recognition

Distributed speech recognition (DSR) where the recognizer is split up into two parts and connected via a transmission channel offers new perspectives for improving the speech recognition performance in mobile environments. In this work, we present the integration of hybrid acoustic models using tied-posteriors in a distributed environment. A comparison with standard Gaussian models is performed on the AURORA2 task and the WSJ0 task. Word-based HMMs and phoneme-based HMMs are trained for distributed and non-distributed recognition using either MFCC or RASTA-PLP features. The results show that hybrid modeling techniques can outperform standard continuous systems on this task. Especially the tied-posteriors approach is shown to be usable for DSR in a very flexible way since the client can be modified without a change at the server site and vice versa.

A comparison of acoustic coding models for speech-driven facial animation

This article presents a thorough experimental comparison of several acoustic modeling techniques by their ability to capture information related to orofacial motion. These models include (1) Linear Predictive Coding and Linear Spectral Frequencies, which model the dynamics of the speech production system, (2) Mel Frequency Cepstral Coefficients and Perceptual Critical Feature Bands, which encode perceptual cues of speech, (3) spectral energy and fundamental frequency, which capture prosodic aspects, and (4) two hybrid methods that combine information from the previous models. We also consider a novel supervised procedure based on Fisher’s Linear Discriminants to project acoustic information onto a low-dimensional subspace that best discriminates different orofacial configurations. Prediction of orofacial motion from speech acoustics is performed using a non-parametric k-nearest-neighbors procedure. The sensitivity of this audio–visual mapping to coarticulation effects and spatial locality is thoroughly investigated. Our results indicate that the hybrid use of articulatory, perceptual and prosodic features of speech, combined with a supervised dimensionality-reduction procedure, is able to outperform any individual acoustic model for speech-driven facial animation. These results are validated on the 450 sentences of the TIMIT compact dataset.

Generative factor analyzed HMM for automatic speech recognition

We present a generative factor analyzed hidden Markov model (GFA-HMM) for automatic speech recognition. In a standard HMM, observation vectors are represented by mixture of Gaussians (MoG) that are dependent on discrete-valued hidden state sequence. The GFA-HMM introduces a hierarchy of continuous-valued latent representation of observation vectors, where latent vectors in one level are acoustic-unit dependent and latent vectors in a higher level are acoustic-unit independent. An expectation maximization (EM) algorithm is derived for maximum likelihood estimation of the model. We show through a set of experiments to verify the potential of the GFA-HMM as an alternative acoustic modeling technique. In one experiment, by varying the latent dimension and the number of mixture components in the latent spaces, the GFA-HMM attained more compact representation than the standard HMM. In other experiments with varies noise types and speaking styles, the GFA-HMM was able to have (statistically significant) improvement with respect to the standard HMM.

An airgun array source signature model for environmental impact assessments

Environmental impact assessments for seismic surveys often include estimates of radially propagating sound levels, which are used to determine marine mammal impact zones. Sound levels may be estimated using computer-based acoustic modelling techniques but these require an accurate description of the survey source signature—arrays of airguns, in particular, have complex, highly directional source functions that depend on the array layout. To address this requirement, an airgun array source signature model has been developed for the purpose of underwater noise level prediction. The source model is based on published descriptions of the physics of airgun bubble oscillations and radiation [A. Ziolkowski, Geophys. J. R. Astron. Soc. 21, 137–161 (1970)] and includes the effects of port throttling, motion damping and bubble interactions. The output of the model is a collection of notional signatures which may be used to compute the source function of the array in any direction. Free parameters in the model have been fit to a large collection of existing airgun signature data, for airguns ranging from 5 to 185 in3. The output of the model is suitable for estimating sound levels resulting from airgun survey activity. [Work supported by NSERC.]

Optimal Utterance Selection for Unit Selection Speech Synthesis Databases

This paper describes techniques to find an optimal data set for building high quality unit-selection speech synthesis inventories. As the quality of unit-selection speech synthesis is dependent on the coverage of the database used in the selection, it is important to select the right data to record. In this paper we describe some simple techniques as well as a more complex acoustic modeling technique based on the database speaker's acoustic characteristics. Result of a simple evaluation procedure are presented justifying the technique.