Linking speech and language processing through prosody

Abstract

In most spoken language understanding systems today, the interface between speech recognition and natural language processing is little more than a serial connection. The top scoring recognition results (a list of sentence hypotheses) are passed to the natural language system, which typically processes these taking the highest ranking recognition hypothesis that is interpretable and ignoring any acoustic information other than rank. The system can be improved by choosing the utterance interpretation based on a combination of the acoustic likelihood of a hypothesized word string and an associated language score (e.g., parse probability or semantic preference score). However, further gain is offered by incorporating a prosody score as well, since prosody can provide a link between syntactic/semantic structure and certain types of acoustic variability. In this talk, two types of computational models of prosody will be described that integrate acoustic cues with text features based on recognizer sentence hypotheses and associated parser outputs. Also, methods for using prosodic information in automatic speech understanding systems will be described. Experimental results with speech understanding in the ATIS domain will be described for three different aspects of prosody: prosodic phrasing, prominence placement, and cues to disfluencies. [Work supported by NSF and ARPA.]