Continuous Speech Stream Research Articles

Why the body comes first: effects of experimenter touch on infants' word finding.

The lexicon of 6-month-olds is comprised of names and body part words. Unlike names, body part words do not often occur in isolation in the input. This presents a puzzle: How have infants been able to pull out these words from the continuous stream of speech at such a young age? We hypothesize that caregivers' interactions directed at and on the infant's body may be at the root of their early acquisition of body part words. An artificial language segmentation study shows that experimenter-provided synchronous tactile cues help 4-month-olds to find words in continuous speech. A follow-up study suggests that this facilitation cannot be reduced to the highly social situation in which the directed interaction occurs. Taken together, these studies suggest that direct caregiver-infant interaction, exemplified in this study by touch cues, may play a key role in infants' ability to find word boundaries, and suggests that early vocabulary items may consist of words often linked with caregiver touches. A video abstract of this article can be viewed at http://youtu.be/NfCj5ipatyE.

How Brain Waves Help Us Make Sense of Speech

Understanding a sentence that someone else utters may seem effortless to you, but it's actually a complex process because you have to parse that sentence into many parts to understand what it means. Speech consists of a hierarchy of components that each takes place on a different timescale. Speech cues such as intonation occur on a relatively long timescale, unfolding over hundreds of milliseconds. At the other end of the spectrum is the phoneme—the smallest unit of speech—which lasts only tens of milliseconds. Gross and co-workers show that hierarchical speech patterns (top) entrain hierarchical wave frequencies in the brain (bottom). Similarly, distinct sets of neurons in the brain fire rhythmically at different rates, and these oscillations can also be arranged in a hierarchy. For instance, slow delta-wave oscillations can influence the magnitude of faster theta oscillations, which in turn can alter the amplitude of even faster gamma oscillations. Rhythmic brain activity plays an important role in a variety of cognitive processes, including attention, memory, and decision-making. A new study published in PLOS Biology now offers insights into the important role that neural oscillations play in speech perception. Joachim Gross, Simon Garrod, and co-workers report novel evidence that hierarchically organized neural oscillations allow people to parse the hierarchical components of speech. The findings not only provide a more complete understanding of the role of brain oscillations in cognition, but they also reveal that people can efficiently understand speech by simultaneously processing different speech components on various timescales. Gross and his team used a brain imaging technique called magnetoencephalography to record neural activity in 22 volunteers while they listened to a seven-minute real-life story. The researchers found that neural oscillations were arranged in a hierarchy: delta oscillations influenced the magnitude of theta oscillations, which in turn affected the amplitude of gamma oscillations. These hierarchical oscillations matched the hierarchical components of speech. Slow delta-wave oscillations in brain regions that process auditory information were in sync with slow rhythmic changes in speech, corresponding to the timescale of prosody—the rhythm, stress, and intonation of speech. This temporal alignment was more pronounced in the right hemisphere of the brain than in the left hemisphere. Meanwhile, the magnitude of fast gamma-wave oscillations in auditory brain areas was influenced by changes in speech that took place on a relatively short timescale, corresponding to syllables. This effect was stronger in the left hemisphere than in the right hemisphere. Collectively, these findings are consistent with a theory called “asymmetric sampling in time” (AST), which proposes that the right hemisphere preferentially processes information over longer timescales, whereas the left hemisphere shows a bias for processing information over shorter timescales. Gross and his collaborators then examined how brain oscillations were affected by variations in speech over time. Speech is not strictly periodic; sometimes people unexpectedly stop talking or begin to talk more quickly. The researchers found that sudden, large changes in the physical features of speech reset the timing of oscillations in auditory brain areas, helping to sync up important speech events with oscillatory brain activity. This finding illustrates how oscillations adapt to changes in the rhythm of speech, allowing the brain to continue to efficiently process speech even when the physical features abruptly change. Taken together, the findings show that hierarchically organized brain oscillations work in concert to track speech components occurring at different timescales, helping to convert a continuous speech stream into meaningful internal representations. Gross J, Hoogenboom N, Thut G, Schyns P, Panzeri S, et al. (2013) Speech Rhythms and Multiplexed Oscillatory Sensory Coding in the Human Brain. doi:10.1371/journal.pbio.1001752

Detection and identification of speech sounds using cortical activity patterns

We have developed a classifier capable of locating and identifying speech sounds using activity from rat auditory cortex with an accuracy equivalent to behavioral performance and without the need to specify the onset time of the speech sounds. This classifier can identify speech sounds from a large speech set within 40ms of stimulus presentation. To compare the temporal limits of the classifier to behavior, we developed a novel task that requires rats to identify individual consonant sounds from a stream of distracter consonants. The classifier successfully predicted the ability of rats to accurately identify speech sounds for syllable presentation rates up to 10 syllables per second (up to 17.9±1.5bits/s), which is comparable to human performance. Our results demonstrate that the spatiotemporal patterns generated in primary auditory cortex can be used to quickly and accurately identify consonant sounds from a continuous speech stream without prior knowledge of the stimulus onset times. Improved understanding of the neural mechanisms that support robust speech processing in difficult listening conditions could improve the identification and treatment of a variety of speech-processing disorders.

The role of memory and representations in statistical learning

Numerous studies have examined learners’ ability to track auditory statistical cues (e.g., Saffran, Aslin, and Newport, 1996). It remains unknown, however, how learners manage this feat. This concern is non-trivial: computation of the transitional probabilities in a traditional statistical learning task would involve access to (at least) hundreds of memory traces that have been accumulated over a mere two minutes. The present experiments aim to elucidate the mechanisms underlying statistical learning. Adult participants are exposed to a 2-min continuous speech stream, composed of native phonetic units (study 1), semi-native phonetic units (study 2), or non-native phonetic units (study 3). Participants’ memories for words are then tested through forced-choice comparisons of words, part-words, and phantom words. In study 1, participants successfully segmented the native phonetic speech stream; however, they showed no preference for part-words or phantom words. Furthermore, asymmetries in performance by syllable position (onset, medial, coda) suggest that memory for the segmented words may be more specified/stable in the medial and coda positions. Preliminary results from study 3 (non-native) suggest that participants fail to segment the stream of less familiar sounds. Taken together, these results suggest that statistical learning proceeds via a “chunking”-type mechanism (e.g., Perruchet and Vinter, 1998).

Transition Probabilities and Different Levels of Prominence in Segmentation

A large body of empirical research demonstrates that people exploit a wide variety of cues for the segmentation of continuous speech in artificial languages, including rhythmic properties, phrase boundary cues, and statistical regularities. However, less is known regarding how the different cues interact. In this study we addressed the question of the relative importance of lexical stress, phrasal prominence, and transitional probabilities (TP) between adjacent syllables for the segmentation of an artificial language. We explored how duration increase, pitch rise, and the combination of duration and pitch on the antepenultimate, the penultimate, and the final syllable of a three‐syllabic word affect segmentation by native speakers of Italian. Our results indicate that, if the most frequent location of stress in the participants’ native language and a lengthened syllable in the artificial language do not coincide, segmentation is disrupted. If there is no conflict between the location of stress in the native language of the participant and the lengthened syllable in the artificial language, segmentation is neither impeded nor facilitated. Pitch marked the edges of the TP‐defined words in a continuous speech stream. When TPs and pitch cues are in conflict, segmentation fails; if pitch rise coincides with the edges of TP words, segmentation succeeds, but is not facilitated. Phrasal prominence comprising both pitch and duration facilitates segmentation when aligned with the word edges. Our findings show that language‐specific peculiarities of how nuclear pitch accents are realized in the native language of the listener might interact with statistical cues in the segmentation of an unfamiliar language.

Spoken keyword detection using autoassociative neural networks

Spoken keywords detection is essential to organize efficiently lots of hours of audio contents such as meetings, radio news, etc. These systems are developed with the purpose of indexing large audio databases or of detecting keywords in continuous speech streams. This paper addresses a new approach to spoken keyword detection using Autoassociative Neural Networks (AANN). The proposed work concerns the use of the distribution capturing ability of the Autoassociative neural network (AANN) for spoken keyword detection. It involves sliding a frame-based keyword template along the speech signal and using confidence score obtained from the normalized squared error of AANN to efficiently search for a match. This work formulates a new spoken keyword detection algorithm. The experimental results show that the proposed approach competes with the keyword detection methods reported in the literature and it is an alternative method to the existing key word detection methods.

The Phoneme Automatic Segmentation Algorithms Study of Tibetan Lhasa Words Continuous Speech Stream

In this paper, we adopt two methods to voice phoneme segmentation when building Tibetan corpus: One is the traditional artificial segmentation method, one is the automatic segmentation method based on the Mono prime HMM model. And experiments are performed to analyze the accuracy of both methods of segmentations. The results showed: Automatic segmentation method based tone prime HMM model helps to shorten the cycle of building Tibetan corpus, especially in building a large corpus segmentation and labeling a lot of time and manpower cost savings, and have greatly improved the accuracy and consistency of speech corpus annotation information.

Differing strategies in English and Japanese word segmentation: A computational-psycholinguistic approach to bootstrapping the lexicon

AbstractHow can six- to eight-month-olds find out where a word begins and where it ends in a continuous speech stream? A computer simulation reveals that the necessary information for segmenting word-like units is present though hidden in English Child-Directed-Speech. This holds even if the cognitive abilities of eight-month-olds constrain the range of possible segmentation algorithms. Applying transitional probability calculations to the incoming speech stream results in segmented chunks, most of which correspond to nonce formations. The key finding is, however, that the most frequent chunks from these formations are indeed words or phrases. Provided that infants prefer and recognize frequent items, it can be shown that a list of sound chains gradually augments a pseudo-lexicon of some eighty to one hundred entries. Now it can be further assumed that these entries are mapped to some new speech material. These mapping locate previously undetected word boundaries. In addition to that, from the pseudo-lexicon, other cues useful for segmentation - phonotactic constructions, prosody, or allophonic variants - could be unambiguously derived and used for complete segmentation before meanings are allocated to these chains. This segmentation mechanism does not seem to be universally true. A second line of computer simulations on Japanese reveals some indirect evidence against a universal segmentation mechanism based on transitional probabilities. To cope with the lack of representative samples of Japanese CDS, the segmentation algorithm was applied to Japanese adult speech (CSJ). For reasons of valid comparison, the simulation was also run on English adult speech (ICE-GB). While English adult speech still generates a viable lexicon, although at a significantly lower performance than English CDS, Japanese adult speech produces an error function that is, although above chance segmentation, insufficient for producing a lexicon

Chunking or not chunking? How do we find words in artificial language learning?

Chunking or Not Chunking? How Do We Find Words in Artificial Language Learning? Ana Franco (afranco@ulb.ac.be) Arnaud Destrebecqz (adestre@ulb.ac.be) Cognition, Consciousness and Computation Group Universite libre de Bruxelles, 50 ave. F.-D. Roosevelt, B1050 BELGIUM Abstract What is the nature of the representations acquired in implicit statistical learning? Recent results in the field of language learning have shown that adults and infants are able to find the words of an artificial language when exposed to a continuous auditory sequence consisting in a random ordering of these words. Such performance can only be based on processing the transitional probabilities between sequence elements. Two different kinds of mechanisms may account for these data: Participants either parse the sequence into smaller chunks corresponding to the words of the artificial language, or they become progressively sensitive to the actual values of the transitional probabilities. The two accounts are difficult to differentiate because they tend to make similar predictions in similar experimental settings. In this study, we present two experiments aimed at disentangling these two theories. In these experiments, participants had to learn two sets of pseudo-linguistic regularities (L1 and L2) presented in the context of a Serial Reaction Time (SRT) task. L1 and L2 were either unrelated, or the intra- words transitions of L1 became the inter-words transitions of L2. The two models make opposite predictions in these two situations. Our results indicate that the nature of the representations depends on the learning conditions. When cues are presented to facilitate parsing of the sequence, participants learned the words of the artificial language. However, when no cues were provided, their performance was strongly influenced by the actual values of the transitional probabilities. Keywords: implicit statistical learning; SRN; chunking; serial reaction time task Introduction A central issue in implicit learning research concerns the nature of the acquired knowledge. Does it reflect the abstract rules on which the training material is based or the surface features of the material, such as the frequencies of individual elements or chunks? According to some theorists, cognition can be viewed as rule-based symbol manipulation (Pinker & Price, 1988). From this perspective, learning would consist in the formation of new abstract, algebra-like rules. According to another theoretical position, information processing is essentially based on associative processes. In this view, learning would not depend on rule acquisition but on mechanisms capable of extracting the statistical regularities present in the environment (e.g., Elman, 1990). Over the last few years, a series of experimental results have provided new insights into the question of the nature of the representations involved in implicit learning. Research on language acquisition has shown that 8-months old infants are sensitive to statistical information (Jusczyk et al., 1994; Saffran, Aslin, & Newport, 1996; Saffran, Johnson, Aslin, & Newport, 1999) and capable of learning distributional relationships between linguistic units (Gomez & Gerken, 1999; Jusczyk, Houston, & Newsome, 1999; Saffran, Aslin, & Newport, 1996; Perruchet & Desaulty, 2008) presented in the continuous speech stream formed by an artificial language. Other studies have indicated that adults are also capable of extracting statistical regularities, and that these mechanisms are not restricted to linguistic material but also apply to auditory non-linguistic stimuli (Saffran, Johnson, Aslin, & Newport, 1999) or to visual stimuli (Fiser & Aslin, 2002). In the same way, implicit sequence learning studies have indicated that human learners are good at detecting the statistical regularities present in a serial reaction time (SRT) task. Altogether, these data suggest that statistical learning depends on associative learning mechanisms rather than on the existence of a “rule abstractor device” (Perruchet, Tyler, Galland, & Peereman, 2004). However, different models have been proposed to account for the data. According to the Simple Recurrent Network model (Elman, 1990; Cleeremans, & McClelland, 1991; Cleeremans, 1993), learning is based on the development of associations between the temporal context in which the successive elements occur and possible successors. Over training, the network learns to provide the best prediction of the next target in a given context, based on the transitional probabilities between the different sequence elements. On the other hand, chunking models, such as PARSER, consider learning as an attention-based parsing process that results in the formation of distinctive, unitary, rigid representations or chunks (Perruchet & Vinter, 1998). Thus, both models are based on processing statistical regularities, but only PARSER leads to the formation of “word-like” units. Although the representations assumed by these two classes of models are quite different, contrasting their assumptions is made difficult by the fact that they tend to make similar experimental predictions. For instance, in a typical artificial language learning experiment, participants are exposed to a continuous stream of plurisyllabic non-words (e.g., BATUBI, DUTABA…) presented in a random order, such that transitional probabilities between syllables are stronger intra-word

Perception-based vowel insertion by native Spanish-speaking learners of English

A hypothesis is proposed which claims that vowel prosthesis before word-initial [sC] clusters by L1 Spanish learners of English is a perception-based repair phenomenon conditioned by L1 biases for native patterns of intergestural timing in complex onsets. Data from a series of off-line perception tests involving fifty L1 Spanish-speaking children confirm the erroneous perception of an extra syllable in a majority of nonce tokens containing the target stimuli. Results of a set of follow-up discrimination and identification tasks prove that Spanish-speakers access information derived from native timing patterns in order to map continuous speech streams to categorical units before lexical storage occurs, supporting the viability of a bias derived from native timing configurations. The findings are of interest since they bear directly on contemporary research in the acquisition of L1 phasing relations and their effects on L2 phonological acquisition.

Phrasal prosody constrains word segmentation in French 16-month-olds.

Infants who are in the process of acquiring their mother tongue have to find a way of segmenting the continuous speech stream into word-sized units. We present an experiment showing that French 16-month-olds are able to exploit phonological phrase boundaries in order to constrain lexical access. Using the conditioned head-turning technique, we showed that infants trained to turn their head for a bisyllabic word responded more often to sentences that contained this word, than to sentences that contained both syllables of this word separated by a phonological phrase boundary. We compare these results with similar results obtained with English-speaking infants, and discuss their implication for lexical and syntactic acquisition.

Acoustic Markers of Prominence Influence Infants’ and Adults’ Segmentation of Speech Sequences

Two experiments investigated the way acoustic markers of prominence influence the grouping of speech sequences by adults and 7-month-old infants. In the first experiment, adults were familiarized with and asked to memorize sequences of adjacent syllables that alternated in either pitch or duration. During the test phase, participants heard pairs of syllables with constant pitch and duration and were asked whether the syllables had appeared adjacently during familiarization. Adults were better at remembering pairs of syllables that during familiarization had short syllables preceding long syllables, or high-pitched syllables preceding low-pitched syllables. In the second experiment, infants were familiarized and tested with similar stimuli as in the first experiment, and their preference for pairs of syllables was accessed using the head-turn preference paradigm.When familiarized with syllables alternating in pitch, infants showed a preference to listen to pairs of syllables that had high pitch in the first syllable. However, no preference was found when the familiarization stream alternated in duration. It is proposed that these perceptual biases help infants and adults find linguistic units in the continuous speech stream.While the bias for grouping based on pitch appears early in development, biases for durational grouping might rely on more extensive linguistic experience.

The contribution of language-specific knowledge in the selection of statistically-coherent word candidates

Much research has explored the extent to which statistical computations account for the extraction of linguistic information. However, it remains to be studied how language-specific constraints are imposed over these computations. In the present study we investigated if the violation of a word-forming rule in Catalan (the presence of more than one mid vowel within a word) may interfere with word extraction by statistical computations. Catalan native adult participants were presented with a continuous speech stream composed of trisyllabic nonsense words that violated this linguistic constraint. In a subsequent test, participants did not recognize the words from matched foils. Nevertheless, the same words were recognized if the test comprised foils that never appeared during familiarization, or if both words and foils were presented visually. Results suggest that background linguistic knowledge modulates the recognition of statistically-coherent words, but not the on-line computations leading to their extraction.

Exploiting Multiple Sources of Information in Learning an Artificial Language: Human Data and Modeling

This study investigates the joint influences of three factors on the discovery of new word-like units in a continuous artificial speech stream: the statistical structure of the ongoing input, the initial word-likeness of parts of the speech flow, and the contextual information provided by the earlier emergence of other word-like units. Results of an experiment conducted with adult participants show that these sources of information have strong and interactive influences on word discovery. The authors then examine the ability of different models of word segmentation to account for these results. PARSER (Perruchet & Vinter, 1998) is compared to the view that word segmentation relies on the exploitation of transitional probabilities between successive syllables, and with the models based on the Minimum Description Length principle, such as INCDROP. The authors submit arguments suggesting that PARSER has the advantage of accounting for the whole pattern of data without ad-hoc modifications, while relying exclusively on general-purpose learning principles. This study strengthens the growing notion that nonspecific cognitive processes, mainly based on associative learning and memory principles, are able to account for a larger part of early language acquisition than previously assumed.

Speech segmentation is facilitated by visual cues

Evidence from infant studies indicates that language learning can be facilitated by multimodal cues. We extended this observation to adult language learning by studying the effects of simultaneous visual cues (nonassociated object images) on speech segmentation performance. Our results indicate that segmentation of new words from a continuous speech stream is facilitated by simultaneous visual input that it is presented at or near syllables that exhibit the low transitional probability indicative of word boundaries. This indicates that temporal audio-visual contiguity helps in directing attention to word boundaries at the earliest stages of language learning. Off-boundary or arrhythmic picture sequences did not affect segmentation performance, suggesting that the language learning system can effectively disregard noninformative visual information. Detection of temporal contiguity between multimodal stimuli may be useful in both infants and second-language learners not only for facilitating speech segmentation, but also for detecting word-object relationships in natural environments.

Query-Driven Strategy for On-the-Fly Term Spotting in Spontaneous Speech

Spoken utterance retrieval was largely studied in the last decades, with the purpose of indexing large audio databases or of detecting keywords in continuous speech streams. While the indexing of closed corpora can be performed via a batch process, on-line spotting systems have to synchronously detect the targeted spoken utterances. We propose a two-level architecture for on-the-fly term spotting. The first level performs a fast detection of the speech segments that probably contain the targeted utterance. The second level refines the detection on the selected segments, by using a speech recognizer based on a query-driven decoding algorithm. Experiments are conducted on both broadcast and spontaneous speech corpora. We investigate the impact of the spontaneity level on system performance. Results show that our method remains effective even if the recognition rates are significantly degraded by disfluencies.

Neural responses to uninterrupted natural speech can be extracted with precise temporal resolution

The human auditory system has evolved to efficiently process individual streams of speech. However, obtaining temporally detailed responses to distinct continuous natural speech streams has hitherto been impracticable using standard neurophysiological techniques. Here a method is described which provides for the estimation of a temporally precise electrophysiological response to uninterrupted natural speech. We have termed this response AESPA (Auditory Evoked Spread Spectrum Analysis) and it represents an estimate of the impulse response of the auditory system. It is obtained by assuming that the recorded electrophysiological function represents a convolution of the amplitude envelope of a continuous speech stream with the to-be-estimated impulse response. We present examples of these responses using both scalp and intracranially recorded human EEG, which were obtained while subjects listened to a binaurally presented recording of a male speaker reading naturally from a classic work of fiction. This method expands the arsenal of stimulation types that can now be effectively used to derive auditory evoked responses and allows for the use of considerably more ecologically valid stimulation parameters. Some implications for future research efforts are presented.

Second language word segmentation in a fluent speech listening task.

Extracting words from a continuous speech stream is a complex task involving the integration of multiple linguistic cues. Like infants, adults have been shown to use both distributional information through statistical learning, and metrical and intonational information, when segmenting words in a second language. This study investigates adults’ native language influence on speech segmentation, and the extent to which linguistic knowledge is augmented by exposure to fluent, natural speech in a second language. In the experiment, English listeners completed a word learning task in an unfamiliar language, Finnish. Next, they listened to a fluent speech passage which contained learned words, as well as syllable sequences controlled for the same frequencies and transitional probabilities as the real words. English speakers have been shown to use acoustic stress cues to locate word boundaries, and while Finnish has a regular stress pattern, other phonotactic patterns could affect English listeners' segmentation abilities. Afterwards, they completed a forced‐choice identification task, choosing from pairs of real and non‐words, including incorrectly segmented syllable sequences, to identify the token which is more likely a word in Finnish. The behavioral data are analyzed for patterns reflecting the integration of native language linguistic knowledge and statistical learning abilities.

Statistical segmentation of tone sequences activates the left inferior frontal cortex: A near-infrared spectroscopy study

Word segmentation, that is, discovering the boundaries between words that are embedded in a continuous speech stream, is an important faculty for language learners; humans solve this task partly by calculating transitional probabilities between sounds. Behavioral and ERP studies suggest that detection of sequential probabilities (statistical learning) plays an important role in the process of word segmentation. To identify the brain regions that are engaged during statistical segmentation of tone sequences, we measured changes in blood oxygenation using multichannel near-infrared spectroscopy (NIRS) while participants were exposed to continuous, nonlinguistic auditory sequences, the elements of which were organized into fixed “tone-words,” but sequenced in random order. We habituated the participants to the tone-words by presenting the stimuli in a training phase prior to the presentation of the continuous tone sequences. After training, the statistical sequences, which included six tone-words, and the random sequences, which included 11 tones in random order, were presented alternately during NIRS recording. A large increase in the oxygenated hemoglobin concentration was observed in the left inferior frontal cortex (IFC) during the statistical sequence condition, but not during the random sequence condition. This suggests that the left IFC plays an important role in statistical segmentation of tone sequences, during which participants deploy the knowledge obtained in the training phase on the subsequent continuous sequence.

La conscience auto-organisatrice

This paper presents a shortened and updated version of a general model that the authors proposed in a target paper in Behavioral and Brain Sciences (2002). In sharp contrast to the standard cognitive framework, the model posits that the only representations people create and manipulate are those which form the momentary phenomenal experience. We propose that the isomorphism generally observed between these representations and the structure of the world is the end-product of a progressive organization that emerges thanks to elementary associative processes that take the conscious representations themselves as the stuff on which they operate, a thesis that we summarize in the concept of Self-Organizing Consciousness (SOC). As an illustration, we show that the SOC framework accounts for the discovery of words from a continuous speech stream, and for seemingly rule-governed