Non-speech Mode Research Articles

Cross-modal correspondences in sine wave: Speech versus non-speech modes.

The present study aimed to investigate whether or not the so-called "bouba-kiki" effect is mediated by speech-specific representations. Sine-wave versions of naturally produced pseudowords were used as auditory stimuli in an implicit association task (IAT) and an explicit cross-modal matching (CMM) task to examine cross-modal shape-sound correspondences. A group of participants trained to hear the sine-wave stimuli as speech was compared to a group that heard them as non-speech sounds. Sound-shape correspondence effects were observed in both groups and tasks, indicating that speech-specific processing is not fundamental to the "bouba-kiki" phenomenon. Effects were similar across groups in the IAT, while in the CMM task the speech-mode group showed a stronger effect compared with the non-speech group. This indicates that, while both tasks reflect auditory-visual associations, only the CMM task is additionally sensitive to associations involving speech-specific representations.

Introduction to the Special Issue: Teaching AAC Use to Individuals with Developmental and Physical Disabilities

Individuals with developmental and physical disabilities also often have severe communication impairment or complex communication needs (Beukelman and Mirenda 2013). The term severe communication impairment refers to situations where the person’s speech is so limited or unintelligible that it is unable to meet his or her everyday communication needs. Lack of sufficient, intelligible speech is prevalent among individuals with developmental and physical disabilities. McQueen et al. (1987), for example, found that approximately 65 % of children with significant intellectual disability had limited or no intelligible speech. Among children with autism spectrum disorder, Tager-Flusberg and Kasari (2013) noted that about 30 % are likely to Bremain with little expressive spoken language ability by the time they reach school age.^ (p. 468). The high prevalence of severe communication impairment among individuals with developmental and physical disabilities makes communication intervention a major habilitation priority for this population. AAC intervention requires identifying suitable non-speech communication modalities, as well as teaching the person to use non-speech modes for effective communication. To this end, AAC research is often focused on developing new and more effective intervention approaches. This special issue of the Journal of Developmental and Physical Disabilities was commissioned to attract papers that focused on developing new and more effective intervention approaches for individuals with developmental and physical disabilities and severe communication impairment. The goal of this special issue is to provide a forum for researchers using a broad range of AAC systems and intervention approaches to share the results of their recent efforts, be they empirical or conceptual. A secondary J Dev Phys Disabil (2016) 28:1–4 DOI 10.1007/s10882-015-9469-6

An evaluation of speech production in two boys with neurodevelopmental disorders who received communication intervention with a speech-generating device

BackgroundChildren with neurodevelopmental disorders often present with little or no speech. Augmentative and alternative communication (AAC) aims to promote functional communication using non-speech modes, but it might also influence natural speech production. MethodTo investigate this possibility, we provided AAC intervention to two boys with neurodevelopmental disorders and severe communication impairment. Intervention focused on teaching the boys to use a tablet computer-based speech-generating device (SGD) to request preferred stimuli. During SGD intervention, both boys began to utter relevant single words. In an effort to induce more speech, and investigate the relation between SGD availability and natural speech production, the SGD was removed during some requesting opportunities. ResultsWith intervention, both participants learned to use the SGD to request preferred stimuli. After learning to use the SGD, both participants began to respond more frequently with natural speech when the SGD was removed. ConclusionThe results suggest that a rehabilitation program involving initial SGD intervention, followed by subsequent withdrawal of the SGD, might increase the frequency of natural speech production in some children with neurodevelopmental disorders. This effect could be an example of response generalization.

Electrophysiological evidence for speech-specific audiovisual integration

Lip-read speech is integrated with heard speech at various neural levels. Here, we investigated the extent to which lip-read induced modulations of the auditory N1 and P2 (measured with EEG) are indicative of speech-specific audiovisual integration, and we explored to what extent the ERPs were modulated by phonetic audiovisual congruency. In order to disentangle speech-specific (phonetic) integration from non-speech integration, we used Sine-Wave Speech (SWS) that was perceived as speech by half of the participants (they were in speech-mode), while the other half was in non-speech mode. Results showed that the N1 obtained with audiovisual stimuli peaked earlier than the N1 evoked by auditory-only stimuli. This lip-read induced speeding up of the N1 occurred for listeners in speech and non-speech mode. In contrast, if listeners were in speech-mode, lip-read speech also modulated the auditory P2, but not if listeners were in non-speech mode, thus revealing speech-specific audiovisual binding. Comparing ERPs for phonetically congruent audiovisual stimuli with ERPs for incongruent stimuli revealed an effect of phonetic stimulus congruency that started at ~200ms after (in)congruence became apparent. Critically, akin to the P2 suppression, congruency effects were only observed if listeners were in speech mode, and not if they were in non-speech mode. Using identical stimuli, we thus confirm that audiovisual binding involves (partially) different neural mechanisms for sound processing in speech and non-speech mode.

Electrophysiological evidence for a multisensory speech-specific mode of perception

We investigated whether the interpretation of auditory stimuli as speech or non-speech affects audiovisual (AV) speech integration at the neural level. Perceptually ambiguous sine-wave replicas (SWS) of natural speech were presented to listeners who were either in ‘speech mode’ or ‘non-speech mode’. At the behavioral level, incongruent lipread information led to an illusory change of the sound only for listeners in speech mode. The neural correlates of this illusory change were examined in an audiovisual mismatch negativity (MMN) paradigm with SWS sounds. In an oddball sequence, ‘standards’ consisted of SWS/onso/coupled with lipread/onso/, and ‘deviants’ consisted of SWS/onso/coupled with lipread/omso/. The AV deviant induced a McGurk-MMN for listeners in speech mode, but not for listeners in non-speech mode. These results demonstrate that the illusory change in the sound by incongruent lipread information evoked an MMN which presumably takes place at a pre-attentive sensory processing stage.

Perception of intersensory synchrony in audiovisual speech: Not that special

Perception of intersensory temporal order is particularly difficult for (continuous) audiovisual speech, as perceivers may find it difficult to notice substantial timing differences between speech sounds and lip movements. Here we tested whether this occurs because audiovisual speech is strongly paired (“unity assumption”). Participants made temporal order judgments (TOJ) and simultaneity judgments (SJ) about sine-wave speech (SWS) replicas of pseudowords and the corresponding video of the face. Listeners in speech and non-speech mode were equally sensitive judging audiovisual temporal order. Yet, using the McGurk effect, we could demonstrate that the sound was more likely integrated with lipread speech if heard as speech than non-speech. Judging temporal order in audiovisual speech is thus unaffected by whether the auditory and visual streams are paired. Conceivably, previously found differences between speech and non-speech stimuli are not due to the putative “special” nature of speech, but rather reflect low-level stimulus differences.

Causal inference in audiovisual speech: Comment on “Crossmodal influences on visual perception” by L. Shams

Shams and Kim (S&K) [1], argue that visual perception can be affected by sound and touch, and that such alterations can occur at early stages of processing. This view is by now well-documented, but here I would like to draw some parallels to another domain – the processing of audiovisual speech – that allows one to make interesting generalizations to the auditory domain. It is well-known that the identification of an auditory utterance can be strongly biased by concurrent lipread information and that the lipread information may even change “sensory-specific” auditory cortex [2,3]. Less well-known is that perception of auditory speech is – like visual perception – also highly adaptive even in the mature brain. Recent research has demonstrated that adult listeners readily use lipread information to adjust the phonetic boundary between two speech categories [4]: For example, an auditory ambiguous sound halfway between /b/ and /d/ is perceived as /b/ if the sound was on a previous occasion combined with lipread /b/, but as /d/ if the same ambiguous sound was on the previous occasion combined with lipread /d/. Lipread information thus can serve as a ‘teacher’ for auditory perception. Audiovisual speech also provides a unique test case to examine the problem of causal inference without the need to change potentially important characteristic of the stimulus. As S&K rightly point out, multisensory integration only makes sense if the sensory signals are caused by the same object. This is a non-trivial inference problem for the brain as it typically does not have a clue about the causal structure of an event. S&K adopt a Bayesian approach to examine causal inference and interactions in the audiovisual spatial and numerical domain. We used another procedure to address the same question for speech, namely by creating Sine Wave Speech’ (SWS) [5]. In SWS the spectral richness of speech is reduced to a few sinusoids, and naive listeners typically perceive the sounds as non-speech whistles. Only when informed that the sounds are actually derived from an utterance, listeners perceive the sound as speech and then cannot switch back to non-speech mode again. The interesting part of this is that only listeners in speech mode are affected by lipread information, – both immediate identification and adaptation aftereffects – while listeners in non-speech mode do not show these intersensory integration effects. This finding demonstrates the causal inference problem in the clearest way: Lipread information affects sound identification only if the two information sources are assigned to the same event: namely an articulatory gesture.

Individuals with Intellectual Disabilities and Augmentative and Alternative Communication: Analysis of Survey Data on Uptake of Aided AAC, and Loneliness Experiences

This study describes uptake of augmentative and alternative communication systems by adults with intellectual disabilities; their ability to self-report at interview; differences in self-reported loneliness experiences by communication mode; and predictors of loneliness. We analyzed National Core Indicators data from 26 US states involving over 13,000 service users grouped by primary means of expression: natural speech, gestures/body language, manual signs, or communication aid/device (aided AAC). Uptake of aided AAC was low; only 0.8% of participants used aided AAC as their primary means of expression. Valid interview responding was higher in turn for participants communicating with natural speech, aided AAC, and other modes. Almost half the participants were lonely, but loneliness did not differ by communication mode used; social contact and social climate variables predicted loneliness. Individuals who used aided AAC communicated more effectively than users of other non-speech modes, supporting more widespread use of aided AAC. Loneliness was prevalent but aided AAC users were not lonelier. Interventions to reduce loneliness are discussed.

Phonetic recalibration only occurs in speech mode

Upon hearing an ambiguous speech sound dubbed onto lipread speech, listeners adjust their phonetic categories in accordance with the lipread information (recalibration) that tells what the phoneme should be. Here we used sine wave speech (SWS) to show that this tuning effect occurs if the SWS sounds are perceived as speech, but not if the sounds are perceived as non-speech. In contrast, selective speech adaptation occurred irrespective of whether listeners were in speech or non-speech mode. These results provide new evidence for the distinction between a speech and non-speech processing mode, and they demonstrate that different mechanisms underlie recalibration and selective speech adaptation.

Neural correlates of switching from auditory to speech perception

Many people exposed to sinewave analogues of speech first report hearing them as electronic glissando and, later, when they switch into a ‘speech mode’, hearing them as syllables. This perceptual switch modifies their discrimination abilities, enhancing perception of differences that cross phonemic boundaries while diminishing perception of differences within phonemic categories. Using high-density evoked potentials and fMRI in a discrimination paradigm, we studied the changes in brain activity that are related to this change in perception. With ERPs, we observed that phonemic coding is faster than acoustic coding: The electrophysiological mismatch response (MMR) occurred earlier for a phonemic change than for an equivalent acoustic change. The MMR topography was also more asymmetric for a phonemic change than for an acoustic change. In fMRI, activations were also significantly asymmetric, favoring the left hemisphere in both perception modes. Furthermore, switching to the speech mode significantly enhanced activation in the posterior parts of the left superior gyrus and sulcus relative to the non-speech mode. When responses to a change of stimulus were studied, a cluster of voxels in the supramarginal gyrus was activated significantly more by a phonemic change than by an acoustic change. These results demonstrate that phoneme perception in adults relies on a specific and highly efficient left-hemispheric network, which can be activated in top-down fashion when processing ambiguous speech/non-speech stimuli.

Priming effects in speech and nonspeech modes of perception

A phonemic perception was assumed to be a process in which representation of speech signals is matched to phonemic representation in long-term memory. Since stimulation is supposed to activate relevant memory representation, priming effects were compared between speech sounds and their sinewave analogs, either in the speech mode or nonspeech mode of perception. An analog target imitating syllables /ba/ or /ga/ in Session 1 or their original speech target in Session 2 was preceded by one of 5 primers, which were these 4 target sounds, and a 1,000 Hz pure tone, with one of 6 interstimulus intervals (ISIs). In Session 1, half of the participants were asked to identify analogs as ill-synthesized syllables, “BA” or “GA,” and the other half were asked to label them as computer sounds, named “A” or “B.” In Session 2, all of them identified original syllables. Regardless of target types and ISIs, subjects given speech instructions showed faster identification of targets preceded by phonemically identical primers, while subjects given nonspeech instructions identified targets faster only when targets were preceded by physically identical primers. These results support that speech sounds and their analogs are accessible to common memory representation.

A Total Communication Approach Towards Meeting the Communication Needs of People with Learning Disabilities

The promotion of effective communication must include an overall strategy, implemented within a multi-disciplinary framework, and systems for evaluation (Van der Gaag, 1998). The ultimate goal of introducing alternative or augmentative communication the use of non-speech modes to supplement or substitute verbal communication (von Tetzchner & Hygum Jensen, 1996) is to enable the individual to ‘communicate freely in all ordinary communication situations and a range of less ordinary ones’ (Martinsen & von Tetzchner, 1996 p47). This requires consideration of the communication in use by all those involved in the communication exchange and of the wider issues in the communication environment (Bradshaw, 1998). Jones describes the implementation of a total communication strategy across a number of organisations. The Somerset Total Communication project demonstrates the usefulness and potential of the systematic introduction of multimodal communication, using a co-ordinated, multi-agency approach with a local authority or service system.

References to people in the communications of female and male youths with mental retardation

Gender-related differences have consistently been reported in the language of adults and children with no disabilities. One well-replicated finding is that females discuss people and relationships more often than do males, particularly in conversations with other females. These stylistic variations in language are considered to have implications for the adaptive functioning of language users, most particularly females. Although studied thus far only in nondisabled individuals, such issues of language style use may be of equal or greater concern for those with mental retardation. How does a cognitive impairment intensify or reduce gender-linked language styles and their effects? Language transcripts were obtained from eight male and eight female participants with retardation, interacting separately with one male and one female adult partner. Half of the participants used speech as their primary mode of communication: the others relied on vocalization, gesture, or augmented modes. Participants using speech showed gender-linked language patterns similar to nondisabled individuals, with females discussing people significantly more often than males. Females using nonspeech modes, in contrast, showed a severe reduction in person-referencing that was not accountable by their expressive speech limitations.

Cross-modal context effects on the perception of /r/ and /l/ in a speech and nonspeech mode

Norrix and Green [J. Acoust. Soc. Am. 99, 2591–2592 (1996)] provided evidence for cross-modal context effects on the perception of /r/ and /l/ in a stop cluster. Tokens from a synthetic /iri–ili/ continuum were dubbed onto a visual /ibi/. When presented in an auditory-visual (AV) condition, the tokens were perceived as ranging from /ibri/ to /ibli/. Results indicated a reliable shift in the AV condition relative to an auditory-only (AO) condition. This shift was in accord with acoustic consequences of articulating /r/ and /l/ in a stop cluster. In the current study, sine-wave analogs of the /iri–ili/ tokens were constructed and presented to two groups of observers in an AO and AV condition. Group One was told they would hear schematic speech sounds and instructed to identify what they heard as /r/ or /l/. Group Two made up their own criteria for classifying the tokens as nonspeech sounds. Results indicated a reliable shift in the /r–l/ boundary between the AO and AV conditions for the speech group only and suggest that the influence of the visual articulatory context depends upon listeners interpreting the auditory tokens as speech. [Work supported by NIDCD, NIH.]

Masking and stimulus intensity effects on duplex perception: A confirmation of the dissociation between speech and nonspeech modes

Using the phenomenon of duplex perception, previous researchers have shown that certain manipulations affect the perception of formant transitions as speech but not their perception as nonspeech "chirps," a dissociation that is consistent with the hypothesized distinction between speech and nonspeech modes of perception [Liberman et al., Percept. Psychophys. 30, 133-143 (1981); Mann and Liberman, Cognition 14, 211-235 (1983)]. The present study supports this interpretation of duplex perception by showing the existence of a "double dissociation" between the speech and chirp percepts. Five experiments compared the effects of stimulus onset asynchrony, backward masking, and transition intensity on the two sides of duplex percepts. It was found that certain manipulations penalize the chirp side but not the speech side, whereas other manipulations had the opposite effect of penalizing the speech side but not the chirp side. In addition, although effects on the speech side of duplex percepts have appeared to be much the same as in the case of normal (electronically fused) speech stimuli, the present study discovered that manipulations that impaired the chirp side of duplex percepts had considerably less effect on the perception of isolated chirps. Thus it would seem that duplex perception makes chirp perception more vulnerable to the effects of stimulus degradation. Several explanations of the data are discussed, among them, the view that speech perception may take precedence over other forms of auditory perception [Mattingly and Liberman, in Signals and Sense: Local and Global Order in Perceptual Maps, edited by G.M. Edelman, W.E. Gall, and W.M. Cowan (Wiley, New York, in press); Whalen and Liberman, Science 237, 169-171 (1987)].

Augmentative and alternative communication: Does it have a future in aphasia rehabilitation?

Abstract Augmentative and alternative communication (AAC) techniques have been used in aphasia treatment for the past two decades. This paper examines the multiple ways in which non-speech modes have been applied—as an alternate means of communication, a means of facilitating the re-acquisition of spoken language skills, and to enhance specific language and communication functions—and the results that have been realized. The assumption that AAC techniques offer minimal assistance to the adult with language impairment is critically examined. Future directions for clinical practice, research and technology are discussed.

Discrimination of duration ratios in bisyllabic tokens by native English and Estonian listeners

In an earlier study [ Journal of Phonetics (1987), 15 , 349–363], we examined the ability of Estonian and English speakers to discriminate among pairs of noise bursts whose durations were in the ratios of 1 : 2, 2: 3, 3 : 2, and 2 : 1. The results obtained showed that both groups of listeners clearly recognized only two contrastive patterns: 1 : 2 and 2 : 3 vs. 3 : 2 and 2: 1. Since there may be a difference in the perception of duration ratios when listeners are in a speech mode of perception rather than a non-speech mode, the present study is a replication of the earlier study using bisyllabic tokens as stimuli instead of noise-burst sequences. Again, the obtained results show the same contrastive patterns. The data support a reanalysis of the three-way quantity contrast in Estonian into two-way contrast based on duration (short-long vs. long-short) and a contrast based upon F 0 contours.

The social-verbal competence of AAC users

This study examines the performance of AAC users on a formal measure of social-verbal competence. The Let's Talk lnventory for Adolescents(Wiig, 1982, Columbus, OH, Charles E. Merrill) was administered to four individuals who communicate via Blissymbolics. Forty speech acts, representing the communicative functions of Ritualizing, Informing, Controlling, and Feeling, were elicited in each of two registers (Peer and Authority). The data were analyzed quantitatively and qualitatively using the following procedures: (a) a comparison of overall and subscores in each communicative function and register with those presented in the test manual for normally developing 7 to 14 year olds; (b) analysis of correct responses in terms of syntactic structure, attenuation (the softening of the speech act), and amplification, and descriptive comparison of the subjects' use of these features with that of normally developing 7 to 14 year olds; (c) analysis of speech acts which were passed by all subjects in both contexts and those which were passed by fewer than two subjects in each context; (d) analysis of errors re: available vocabulary and understanding of the situation. The results showed variation in the subjects' ability to fulfill speech acts in the four communicative functions. As a group, the subjects were most successful in the Informing function, followed by Ritualizing, Controlling, and Feeling. Although the Blissymbol users produced many of the same syntactic structures and attenuation devices as were found among the normal groups, differences related to the nonspeech mode were evident. An elicited speech act format is felt to be a valuable addition to the body of measures for evaluating the communication abilities of nonspeaking individuals.

The ABCs of categorical perception

Studies of speech perception first revealed a surprising discontinuity in the way in which stimulus values on a physical continuum are perceived. Data which demonstrate the effect in nonspeech modes have challenged the contention that categorical perception is a hallmark of the speech mode, but the psychophysical models that have been proposed have not resolved the issues raised by empirical findings. This study provides data from judgments of four sensory continua, two visual and two tactual-kinesthetic, which show that the adaptation level for a set of stimuli serves as a category boundary whether stimuli on the continuum differ by linear or logarithmic increments. For all sensory continua studied, discrimination of stimuli belonging to different perceptual categories was more accurate than discrimination of stimuli belonging to the same perceptual category. Moreover, shifts in the adaptation level produced shifts in the location of the category boundary. The concept of Adaptation-level Based Categorization (ABC) provides a unified account of judgmental processes in categorical perception without recourse to post hoc constructs such as implicit anchors or external referents.

Selecting Augmentative Communication Systems on the Basis of Current Communicative Means and Functions

Use of augmentative non-speech systems of communication have increasingly been reported as effective in promoting communicative development in individuals with autism and related handicaps. While various forms of sign language have constituted the primary non-speech communication mode, other approaches have also been used, including plastic symbol, written word and pictorial systems. While the reported results have overall been encouraging, questions remain as to whether one particular system, for instance, sign language is most suitable to the autistic population as a whole, and at which point and how it should be taught. This paper examines to what extent individuals with autism may share some common characteristics that would make them prime candidates for one particular system over another. Given that visuo-spatial abilities tend to be well developed, it is argued that a case could be made for the use of communication modes that incorporate non-transient stimuli. Nevertheless, given the heterogeneity ...