Acoustic Word Research Articles

Does working memory change with age? The interactions of concurrent articulation with the effects of word length and acoustic confusion

ABSTRACT The effects of acoustic confusion (phonological similarity), word length, and concurrent articulation (articulatory suppression) are cited as support for Working Memory's phonological loop component (e.g., Baddeley, 2000, Psychonomic Bulletin and Review, 7, 544). Research has focused on younger adults, with no studies examining whether concurrent articulation reduces the word length and acoustic confusion effects among older adults. In the current study, younger and older adults were given lists of similar and dissimilar letters (Experiment 1) or long and short words (Experiment 2) for immediate serial reconstruction of order. Items were presented visually or auditorily, with or without concurrent articulation. As expected, younger and older adults demonstrated effects of acoustic confusion, word length, and concurrent articulation. Further, concurrent articulation reduced the effects of acoustic confusion and word length equally for younger and older adults. This suggests that age-related differences occur in overall performance, but do not reflect an age-related deficiency in the functioning of the phonological loop component of working memory.

Elucidating the effects of ageing on remembering perceptually distorted word pairs

We investigated the effects of age, background babble, and acoustic distortion of the word itself on serial position memory in a series of experiments involving six different auditory environments (quiet, and 12-talker background babble presented between, overlapping, or concurrent with word presentation or with two kinds of distortion applied to the words). To control for hearing, the level of babble or distortion was adjusted so that younger and older adults could hear the words equally well. Although the presence of continuous and word-flanking background babble adversely affected memory in the early serial positions in both age groups, only older adults' memory was adversely affected in the later serial positions. Moreover, younger adults' memory was not affected by acoustic word distortion, whereas one of the two types of temporal distortion adversely affected memory for later serial positions in older adults. The exact pattern of impairment and its interaction with age suggests that memory in older adults is more affected than that in younger adults in complex listening situations because they either need more time or have to employ more attentional resources to segregate different auditory streams, thereby depleting the pool of resources available for memory encoding.

Determining and using acoustic confusability, acoustic perplexity and synthetic acoustic word error rate

Two statistics are disclosed for determining the quality of language models. These statistics are called acoustic perplexity and the synthetic acoustic word error rate (SAWER), and they depend upon methods for computing the acoustic confusability of words. It is possible to substitute models of acoustic data in place of real acoustic data in order to determine acoustic confusability. An evaluation model is created, a synthesizer model is created, and a matrix is determined from the evaluation and synthesizer models. Each of the evaluation and synthesizer models is a hidden Markov model. Once the matrix is determined, a confusability calculation may be performed. Different methods are used to determine synthetic likelihoods. The confusability may be normalized and smoothed and methods are disclosed that increase the speed of performing the matrix inversion and the confusability calculation. A method for caching and reusing computations for similar words is disclosed. Acoustic perplexity and SAWER are determined and applied.

When and how do listeners relate a sentence to the wider discourse? Evidence from the N400 effect

In two ERP experiments, we assessed the impact of discourse-level information on the processing of an unfolding spoken sentence. Subjects listened to sentences like Jane told her brother that he was exceptionally quick/slow, designed such that the alternative critical words were equally acceptable within the local sentence context. In Experiment 1, these sentences were embedded in a discourse that rendered one of the critical words anomalous (e.g. because Jane’s brother had in fact done something very quickly). Relative to the coherent alternative, these discourse-anomalous words elicited a standard N400 effect that started at 150–200 ms after acoustic word onset. Furthermore, when the same sentences were heard in isolation in Experiment 2, the N400 effect disappeared. The results demonstrate that our listeners related the unfolding spoken words to the wider discourse extremely rapidly, after having heard the first two or three phonemes only, and in many cases well before the end of the word. In addition, the identical nature of discourse- and sentence-dependent N400 effects suggests that from the perspective of the word-elicited comprehension process indexed by the N400, the interpretive context delineated by a single unfolding sentence and a larger discourse is functionally identical.

Theory and practice of acoustic confusability

In this paper we define two alternatives to the familiar perplexity statistic (hereafter lexical perplexity), which is widely applied both as a figure of merit and as an objective function for training language models. These alternatives, respectively acoustic perplexity and the synthetic acoustic word error rate, fuse information from both the language model and the acoustic model. We show how to compute these statistics by effectively synthesizing a large acoustic corpus, demonstrate their superiority (on a modest collection of models and test sets) to lexical perplexity as predictors of language model performance, and investigate their use as objective functions for training language models. We develop an efficient algorithm for training such models, and present results from a simple speech recognition experiment, in which we achieved a small reduction in word error rate by interpolating a language model trained by synthetic acoustic word error rate with a unigram model.

Learning visually grounded words and syntax of natural spoken language

Properties of the physical world have shaped human evolutionary design and given rise to physically grounded mental representations. These grounded representations provide the foundation for higher level cognitive processes including language. Most natural language processing machines to date lack grounding. This paper advocates the creation of physically grounded language learning machines as a path toward scalable systems which can conceptualize and communicate about the world in human-like ways. As steps in this direction, two experimental language acquisition systems are presented. The first system, CELL, is able to learn acoustic word forms and associated shape and color categories from fluent untranscribed speech paired with video camera images. In evaluations, CELL has successfully learned from spontaneous infant-directed speech. A version of CELL has been implemented in a robotic embodiment which can verbally interact with human partners. The second system, DESCRIBER, acquires a visually-grounded model of natural language which it uses to generate spoken descriptions of objects in visual scenes. Input to DESCRIBER’s learning algorithm consists of computer generated scenes paired with natural language descriptions produced by a human teacher. DESCRIBER learns a three-level language model which encodes syntactic and semantic properties of phrases, word classes, and words. The system learns from a simple ‘show-and-tell’ procedure, and once trained, is able to generate semantically appropriate, contextualized, and syntactically well-formed descriptions of objects in novel scenes.

Speech recognition system which creates acoustic models by concatenating acoustic models of individual words

A computerized speech recognition system creates acoustic models of phrases by concatenating acoustic models for individual words. The system stores an acoustic word model and spelling for each of its vocabulary words. When it receives the spelling of a multi-word phrase to be treated as a new vocabulary word, it stores that multi-word spelling as the spelling of the new vocabulary word, and a new acoustic model created by concatenating the acoustic word models of previous vocabulary words whose spellings correspond to words in the multi-word spelling as the acoustic model for the new word. The system can then perform speech recognition by comparing acoustic signals against the word models of stored vocabulary words, including those representing such multi-word phrases. Preferably when a multi-word model is formed, the individual acoustic models concatenated are modified to represent the coarticulation which takes place between words spoken continuously. This can be done by representing word models as sequences of individual phonemes, individual phonemes by phoneme-in-context models, and coarticulation by modifying the phoneme-in-context models of phonemes adjacent the boundary between concatenated words models to reflect the context of the phonemes on the other side of such word boundaries. The system can be a discrete word recognizer. The multi-word phrases can be selected by a user or be obtained from other programs running on the same computer as the speech recognizer, such as from one or more commands available in another program.

Pronunciation variants across system configuration, language and speaking style

This contribution aims at evaluating the use of pronunciation variants for different recognition system configurations, languages and speaking styles. This study is limited to the use of variants during speech alignment, given an orthographic transcription of the utterance and a phonemically represented lexicon and is thus focused on the modeling capabilities of the acoustic word models. To measure the need for variants we have defined the variant2+ rate which is the percentage of words in the corpus not aligned with the most common phonemic transcription. This measure may be indicative of the possible need for pronunciation variants in the recognition system. Pronunciation lexica have been automatically created so as to include a large number of variants (overgeneration). In particular, lexica with parallel and sequential variants were automatically generated in order to assess the spectral and temporal modeling accuracy. We first investigated the dependence of the aligned variants on the recognizer configuration. Then a cross-lingual study was carried out for read speech in French and American English using the BREF and the WSJ corpora. A comparison between read and spontaneous speech was made for French based on alignments of BREF (read) and Mask (spontaneous) data. Comparative alignment results using different acoustic model sets demonstrate the dependency between the acoustic model accuracy and the need for pronunciation variants. The alignment results obtained with the above lexica have been used to study the link between word frequencies and variants using different acoustic model sets.

Reduction of search space in speech recognition using phone boundaries and phone ranking

A method for estimating the probability of phone boundaries and the accuracy of the acoustic modelling in reducing a search-space in a speech recognition system. The accuracy of the acoustic modelling is quantified by the rank of the correct phone. The system includes a microphone for converting an utterance into an electrical signal, which is processed by an acoustic processor and label match which finds the best-matched acoustic label prototype. A probability distribution on phone boundaries is produced for every time frame using a first decision tree. These probabilities are compared to a threshold and some time frames are identified as boundaries between phones. An acoustic score is computed for all phones between every given pair of hypothesized boundaries, and the phones are ranked on the basis of this score. A second decision tree is traversed for every time frame to obtain the worst case rank of the correct phone at that time, and a short list of allowed phones is made for every time frame. A fast acoustic word match processor matches the label string from the acoustic processor to produce an utterance signal which includes at least one word. From recognition candidates produced by the fast acoustic match and the language model, the detailed acoustic match matches the label string from the acoustic processor against acoustic word models and outputs a word string corresponding to an utterance.

The effect of word-final phonemes on spoken word recognition

Three experiments tested subjects’ ability to recognize spoken words based on word-final phonological information. Experiment 1 replicated the finding [A. Salasoo and D. B. Pisoni, J. Memory Lang. 24, 210–231 (1985)] that when presented with phonological information incrementally beginning from word-offset (backward gating), subjects correctly identified spoken words in the absence of word-initial phonemes. Additionally, experiment 1 demonstrated that acoustic neighborhood size is a significant predictor of recognition probability for both forward and backward gated words. In experiment 2, word-initial and word-final acoustic word fragments were used as primes in a crossmodal identity priming task with a naming response. Both word-initial and word-final primes significantly facilitated subsequent naming reaction times. Experiment 3 replicated experiment 2 using an associative priming paradigm. The results are interpreted from a connectionist perspective. [Work supported by NICHD.]

Classification of news speech articles by keyword spotting

This paper proposes a method to automatically classify news speech articles using the word spotting technique. In word spotting, keyword probability Ps(w) ending at time t is computed by subtracting an accumulated probability of the best syllable sequence from the best syllable sequence + keyword probability. Keywords are extracted by finding the local peaks of the keyword probabilities. In article classification, the topic contribution of word (TCW) P(n|w) is computed in advance using ASAHI newspaper classification indices. Then a topic probability P(n)=ΣwP(w)×P(n|w) is computed using the TCW P(n|w) and keyword probability Ps(w) is substituted into P(w). This topic probability is the integration of acoustic word probability Ps(w) and a priori knowledge probability TCW. The news article can be classified by finding the highest topic probability P(n). The classification of news speech articles was carried out using 5-min NHK news programs over 30 days. The phoneme HMMs were trained using one-day news (5 min). In the experiment, 77 articles were classified using 183 keywords. The numbers of phoneme HMMs and syllables are 40 and 108, respectively. The classification rate is at present 49.4% (38/77). The correct rate of word spotting was 54.9% (567/1033) and false alarm was 1.2 fa/kw/h.

Hemispheric contributions to the integration of visual and auditory information in speech perception.

Differential hemispheric contributions to the perceptual phenomenon known as the McGurk effect were examined in normal subjects, 1 callosotomy patient, and 4 patients with intractable epilepsy. Twenty-five right-handed subjects were more likely to demonstrate an influence of a mouthed word on identification of a dubbed acoustic word when the speaker's face was lateralized to the LVF as compared with the RVF. In contrast, display of printed response alternatives in the RVF elicited a greater percentage of McGurk responses than display in the LVF. Visual field differences were absent in a group of 15 left-handed subjects. These results suggest that in right-handers, the two hemispheres may make distinct contributions to the McGurk effect. The callosotomy patient demonstrated reliable McGurk effects, but at a lower rate than the normal subjects and the epileptic control subjects. These data support the view that both the right and left hemisphere can make significant contributions to the McGurk effect.

A language model for very large-vocabulary speech recognition

We apply a trigram language model to an 86 000-word vocabulary speech recognition task. The recognition task consists of paragraphs chosen arbitrarily from a variety of sources, including newspapers, books, magazines, etc. The trigram language model parameters correspond to probabilities of words conditioned on the previous two words. The number of parameters to be estimated is enormous: 86 000 3 parameters in our case. Even a training set consisting of 60 million words is too small to estimate these parameters reliably. Parameter estimates using relative frequencies would assign a value of zero to a large fraction of the parameters. Many algorithms have been proposed to estimate probabilties of events not observed in the training text. We propose here a simple algorithm for estimating the probabilities of such events using Turing's formula. The resulting trigram language model reduces the acoustic recognition errors by 60%. We also show that the effectiveness of the trigram language model for correcting an acoustic word recognition error depends on whether or not the neighbouring word contexts occur in the training text corpus for the language model.

Selective Kana Agraphia: A Case Report

We present a Japanese man with selective Kana (phonogram) agraphia as a sequels of two cerebral infarctions in a part of the left angular gyros and its adjoining posterior superior temporal gyros and the left corona radiata. The agraphia of this patient showed the following features: (1) His writing difficulty was greater for Kana than for Kanji (ideogram) when a word was polysyllabic. (2) Kana errors consisted of perseveration and substitution with another letter, resulting in jargon agraphia. (3) This jargon agraphia often contained a correct number of letters for a target word.Based on these findings, we speculate that the basic defect of this agraphia lies in the process of converting an acoustic word image into a Kana motor grapheme sequence.

A Conversational-Mode Airline Information and Reservation System Using Speech Input and Output

We describe a conversational-mode, speech-understanding system which enables its user to make airline reservations and obtain timetable information through a spoken dialog. The system is structured as a three-level hierarchy consisting of an acoustic word recognizer, a syntax analyzer, and a semantic processor. The semantic level controls an audio response system making two-way speech communication possible. The system is highly robust and operates on-line in a few times real time on a laboratory minicomputer. The speech communication channel is a standard telephone set connected to the computer by an ordinary dialed-up line.

A conversational‐mode airline information and reservation system using speech input and output

We describe a conversational‐mode speech understanding system which enables its user to make airline reservations and obtain timetable information through a spoken dialog. The system is structured as a three level hierarchy consisting of an acoustic word recognizer, a syntax analyzer, and a semantic processor. The semantic level controls an audio response system making two‐way speech communication possible. The system is highly robust and operates on line in a few times real time on a laboratory minicomputer. The speech communication channel is a standard telephone set connected to the computer by an ordinary dialed‐up line.

The Effects of Syntactic Analysis on Word Recognition Accuracy

In this paper we examine the effects of an algorithm for syntactic analysis on word recognition accuracy. The behavior of the algorithm is studied by means of a computer simulation. We describe the syntactic analysis technique, the problem domain to which it was applied, and the details of the simulation. We then present the results of the simulation and their implications. We find, for example, that an acoustic word error rate of 10 percent is reduced to 0.2 percent after syntactic analysis, resulting in a sentence error rate of 1 percent. These figures are based on a 127-word vocabulary and an average of 10.3 words per sentence for 1000 sentences. We expect that these results are indicative of the performance which will be attained by a real speech recognition system which uses the syntactic analysis algorithm described herein.

Sur les mecanismes des troubles de la repetition et de la comprehension du langage dans l'aphasie acoustico-mnestique

Are the language disorders seen in acoustic-amnesic aphasia (classical conduction aphasia), in particular the characteristic defect of repetition, due to a short-term verbal memory defect, a deficit of acoustic word perception, or a general multimodal perceptual weakness? An attempt is made to answer these questions by reference to the findings in 8 cases of acoustic-amnesic aphasia. It would appear from these findings that the deficit in acoustic comprehension and repetition depends, in part at least, on a restriction in the amount of verbal material which may be processed by these patients. Frequency of usage of the words to be repeated is a factor of some importance and, in general, increase in frequency of usage augments perceptual capacity and hence comprehension. Finally, it would seem that the deficits of comprehension and repetition can be largely overcome if the formal structure of the verbal material is suitably adapted to the patient's perceptual capacity.

Central (Or Conduction) Aphasia in a Japanese Patient

A case of relatively pure conduction aphasia in a 66 year old woman is reported. A detailed linguistic analysis was performed and the patient's handling of the phonogrammic and ideogrammic aspects of the Japanese language was analysed. The patient showed a comparable pattern of difficultly in the following four operations: (1) naming of pictures (2) reading aloud of Kanji and Kana characters, (3) repetition of words and (4) transcription of heard words into Kanji and Kana characters. Of special note was the patient's superior written transcription of the ideographic symbols and the superior oral reading of the phonogrammic symbols. These results indicate that the patient had a basic defect in expressing the predetermined target word in verbal or writter form. From the analysis of Kana writings it is hypothesized that the in ability to establish a stable and distinct acoustic word image is responsible for this basic expressive difficulty leading to both paraphasic and paragraphic errors.

Word Length and Intelligibility

Intelligibility tests were conducted with monosyllabic, bisyllabic and trisyllabic words under conditions of known and unknown message sets. Longer words were found to be more intelligible than shorter words in both known and unknown message sets. Differences in intelligibility among the different lengths are interpreted in terms of acoustic discriminability and relative word frequency.