Gestural Scores Research Articles

Language outcomes from the UK-CDI Project: can risk factors, vocabulary skills and gesture scores in infancy predict later language disorders or concern for language development?

At the group level, children exposed to certain health and demographic risk factors, and who have delayed language in early childhood are, more likely to have language problems later in childhood. However, it is unclear whether we can use these risk factors to predict whether an individual child is likely to develop problems with language (e.g., be diagnosed with a developmental language disorder). We tested this in a sample of 146 children who took part in the UK-CDI norming project. When the children were 15-18 months old, 1,210 British parents completed: (a) the UK-CDI (a detailed assessment of vocabulary and gesture use) and (b) the Family Questionnaire (questions about health and demographic risk factors). When the children were between 4 and 6 years, 146 of the same parents completed a short questionnaire that assessed (a) whether children had been diagnosed with a disability that was likely to affect language proficiency (e.g., developmental disability, language disorder, hearing impairment), but (b) also yielded a broader measure: whether the child's language had raised any concern, either by a parent or professional. Discriminant function analyses were used to assess whether we could use different combinations of 10 risk factors, together with early vocabulary and gesture scores, to identify children (a) who had developed a language-related disability by the age of 4-6 years (20 children, 13.70% of the sample) or (b) for whom concern about language had been expressed (49 children; 33.56%). The overall accuracy of the models, and the specificity scores were high, indicating that the measures correctly identified those children without a language-related disability and whose language was not of concern. However, sensitivity scores were low, indicating that the models could not identify those children who were diagnosed with a language-related disability or whose language was of concern. Several exploratory analyses were carried out to analyse these results further. Overall, the results suggest that it is difficult to use parent reports of early risk factors and language in the first 2 years of life to predict which children are likely to be diagnosed with a language-related disability. Possible reasons for this are discussed.

Temporopolar volumes are associated with the severity of social impairment and language development in children with autism spectrum disorder with developmental delay.

Children with autism spectrum disorder (ASD) and developmental delay (DD; ASD + DD) have more severe clinical symptoms than those with ASD without DD (ASD-only). However, little is known about the underlying neuroimaging mechanisms. The aim of this study was to explore the volumetric difference between patients with ASD + DD and ASD-only and investigate the relationship between brain alterations and clinical manifestations. A total of 184 children with ASD aged 2-6 years were included in this study, who were divided into two groups according to their cognitive development: ASD + DD and ASD-only. Clinical symptoms and language development were assessed using the Autism Diagnostic Observation Schedule (ADOS), Childhood Autism Rating Scale (CARS), and the Putonghua Communicative Development Inventory. Of the 184 children, 60 age-matched males (30 ASD + DD and 30 ASD-only patients) with high-resolution structural neuroimaging scans were included for further voxel-based morphometry analysis to examine the relationship between clinical symptoms and gray matter volumes. The ASD + DD group had higher CARS and ADOS scores, lower gesture scores, and poorer performance in "responding to joint attention" (RJA) and "initiating joint attention" than the ASD-only group. Larger gray matter volumes in the temporal poles of the right and left middle temporal gyri were associated with the co-occurrence of DD in patients with ASD. Moreover, temporopolar volumes were correlated with CARS and ADOS scores, gesture scores, and RJA ability. Pre-language development significantly mediated the relationship between temporopolar volumes and both CARS and ADOS scores; RJA ability, but not gesture development, contributed to this mediating effect. In this study, we found that temporopolar volumes were enlarged in patients with ASD who had comorbid DD, and these patients showed an association between symptom severity and language ability during the pre-language stage. Offering early interventions focused on RJA and the temporal pole may help improve clinical symptoms.

On the Emergence of Phonological Knowledge and on Motor Planning and Motor Programming in a Developmental Model of Speech Production.

A broad sketch for a model of speech production is outlined which describes developmental aspects of its cognitive-linguistic and sensorimotor components. A description of the emergence of phonological knowledge is a central point in our model sketch. It will be shown that the phonological form level emerges during speech acquisition and becomes an important representation at the interface between cognitive-linguistic and sensorimotor processes. Motor planning as well as motor programming are defined as separate processes in our model sketch and it will be shown that both processes revert to the phonological information. Two computational simulation experiments based on quantitative implementations (simulation models) are undertaken to show proof of principle of key ideas of the model sketch: (i) the emergence of phonological information over developmental stages, (ii) the adaptation process for generating new motor programs, and (iii) the importance of various forms of phonological representation in that process. Based on the ideas developed within our sketch of a production model and its quantitative spell-out within the simulation models, motor planning can be defined here as the process of identifying a succession of executable chunks from a currently activated phoneme sequence and of coding them as raw gesture scores. Motor programming can be defined as the process of building up the complete set of motor commands by specifying all gestures in detail (fully specified gesture score including temporal relations). This full specification of gesture scores is achieved in our model by adapting motor information from phonologically similar syllables (adapting approach) or by assembling motor programs from sub-syllabic units (assembling approach).

Searching Names in Contact List by Three Touch-Screen Gestures

We compared three list scrolling gestures including sliding to scroll, moving a regular scrollbar, and scrolling through alphabetic index, in a name locating task on mobile phone. We found that scrolling through alphabetic index was the fastest independent of target locations. Moreover. for near-reaching targets RT was shorter for sliding to scroll than moving scrollbar, but it took sliding longer than moving scrollbar for far-reaching targets. Gesture evaluation scores of alphabetic index increased and those of sliding to scroll decreased over time. The different pre- and post-task rating patterns for gestures are due to their differential fatigue and emotional changes. The present study provides insights into users’ mental models for three list scrolling gestures and implications for future gesture design.

Lexicon on Board: A MEG Study Based on Expressive Picture-Naming

The cognitive processes associated with word retrieval and the investigation of word-picture differences are complex and not fully understood. Uttering a word entails orchestrating several steps as visual object recognition, accessing a lexical concept, lemma selection, lemma retrieval, accessing the morpheme(s) and generating the phonological word, and finally retrieving syllabic gestural scores and articulation. To study the mechanisms of word retrieval in lemma selection phase, we compared the three groups of monolingual, semi-bilingual and bilingual learners with respect to their topography and strength of Functional Connectivity (FC)values of the most highlighted pair of activated nodes in the time range of 0-150 ms in different frequency bands (delta, theta, alpha, beta, and gamma) upon application of the stimuli. We have seen no significant difference between difference frequency bands (p > 0.05) at the most highlighted FC pairs. However, we observed higher gamma values signifying the semantic activation of the word. We could not find any significant difference between the groups in terms of FC values at designated pairs of nodes signifying that different amount of exposure could not affect electrophysiological patterns in the preliminary step of word production. Acknowledgements This study was supported by the Portuguese Foundation for Science and Technology and the Portuguese Ministry of Science, Technology and Higher Education (UID/PSI/01662/2019), through the national funds (PIDDAC).

Modeling speaker-specific vocal tract kinematics from gestural scores

The theory of Task Dynamics provides a method of predicting articulatory kinematics from a discrete phonologically-relevant representation (“gestural score”). However, because the implementations of that model (e.g., Nam et al., 2004) have generally used a simplified articulatory geometry (Mermelstein et al., 1981) whose forward model (from articulator to constriction coordinates) can be analytically derived, quantitative predictions of the model for individual human vocal tracts have not been possible. Recently, methods of deriving individual speaker forward models from real-time MRI data have been developed (Sorensen et al., 2019). This has further allowed development of task dynamic models for individual speakers, which make quantitative predictions. Thus far, however, these models (Alexander et al., 2019) could only synthesize limited types of utterances due to their inability to model temporally overlapping gestures. An updated implementation is presented, which can accommodate overlapping gestures and incorporates an optimization loop to improve the fit of modeled articulatory trajectories to the observed ones. Using an analysis-by-synthesis approach, the updated implementation can be utilized: (1) to refine the hypothesized speaker-general gestural parameters (target, stiffness) for individual speakers; (2) to test different degrees of temporal overlapping among multiple gestures such as a CCVC syllable. [Work supported by NSF, Grant 1908865.]

Children's language abilities predict success in remote communication contexts.

Remote communicative contexts are part of everyday social, familial, and academic interactions for the modern child. We investigated the ability of second-graders to engage in remote discourse, and we determined whether language ability, theory of mind, and shy temperament predicted their success. Fifty 7-to-9-year-old monolingual English speakers with a wide range of language abilities participated in standardized testing and an expository discourse task in which they taught two adults to solve the Tower of London, one in an audiovisual condition to simulate video chat and a second in an audio-only condition to simulate phone communication. The discourse was scored with a rubric of 15 items deemed relevant to the explanation. Children included 27% to 87% of the items, with more items communicated via gesture than spoken word in both conditions. Gesture scores and spoken scores were highly correlated. Children specified more rubric items overall in the audio condition and more rubric items in the spoken modality when in the audio condition than the audiovisual condition. Performance in both conditions was positively associated with scores on independent measures of language ability. There was no relationship between performance and theory of mind, shy temperament, ability to solve the Tower of London, age, or sex. We conclude that 7-to-9-year-olds adjust the modality and content of their message to suit their remote partner's needs, but their success in remote discourse contexts varies significantly from individual to individual. Children with below-average language skills are at risk for functional impairments in remote communication.

The nonlinear gestural model of speech apraxia: clinical implications and applications

ABSTRACT Rationale: In the approach presented here, adult apraxia of speech (AoS) is understood as a disorder affecting the language-specific motor patterns a speaker acquires during childhood and adolescence. The phonological factors that influence articulatory accuracy in AoS reflect the relative vulnerability of their accompanying motor planning variables to apraxic impairment, depending on the extent to which they have been consolidated in speech motor learning. Since these factors interact strongly, the Nonlinear Gestural (NLG) model combines them into an integrative computational framework. We thereby aim at explaining word production accuracy in AOS from a word’s structural properties across multiple hierarchical layers, – from articulatory gestures to metrical patterns. The NLG model: In a review of earlier articles, the NLG-model is first introduced by describing five core principles underlying its design, whereupon the computational and empirical underpinnings of the model are explained. Finally, the model shape is delineated by specifying its empirically derived coefficients, with a focus on clinical implications. Clinical applications: A web-based NLG calculator is described that allows clinicians to compute NLG scores for German words or pseudowords and create displays of their gestural scores, as a prerequisite of incorporating the NLG approach in their diagnostic and therapeutic work. Furthermore, we describe how the structural properties of words, as represented by their NLG scores, can be linked to other clinically relevant lexical variables. Potential applications of these tools involve (i) the design of assessment and treatment materials following a hierarchy of articulatory difficulty, (ii) the interpretation of error patterns following the NLG hierarchy as a signature of the apraxic pathomechanism, with possible differential diagnostic applications, and (iii) computation of a distance metric to describe the relationship between target words and their realisations by patients with AoS. Expansions to other types of speech sound impairment in adults and children are discussed.

Coordinative movement of articulators in bilabial stop /p/*

Speech articulators are coordinated for the purpose of segmental constriction in terms of a task. In particular, vertical jaw movements repeatedly contribute to consonantal as well as vocalic constriction. The current study explores vertical jaw movements in conjunction with bilabial constriction in bilabial stop /p/ in the context /a/-to-/a/. Revisiting kinematic data of /p/ collected using the electromagenetic midsagittal articulometer (EMMA) method from seven (four female and three male) speakers of Seoul Korean, we examined maximum vertical jaw position, its relative timing with respect to the upper and lower lips, and lip aperture minima. The results of those dependent variables are recapitulated in terms of linguistic (different word boundaries) and paralinguistic (different speech rates) factors as follows. Firstly, maximum jaw height was lower in the across-word boundary condition (across-word < within-word), but it did not differ as a function of different speech rates (comfortable = fast). Secondly, more reduction in the lip aperture (LA) gesture occurred in fast rate, while word-boundary effects were absent. Thirdly, jaw raising was still in progress after the lips’ positional extrema were achieved in the within-word condition, while the former was completed before the latter in the across-word condition. Lastly, relative temporal lags between the jaw and the lips (UL and LL) were more synchronous in fast rate, compared to comfortable rate. When these results are considered together, it is possible to posit that speakers are not tolerant of lenition to the extent that it is potentially realized as a labial approximant in either word-boundary condition while jaw height still manifested lower jaw position in the across-word boundary condition. Early termination of vertical jaw maxima before vertical lower lip maxima across-word condition may be partly responsible for the spatial reduction of jaw raising movements. This may come about as a consequence of an excessive number of factors (e.g., upper lip height (UH), lower lip height (LH), jaw angle (JA)) for the representation of a vector with two degrees of freedom (x, y) engaged in a gesture-based task (e.g., lip aperture (LA)). In the task-dynamic application toolkit, the jaw angle parameter can be assigned numerical values for greater weight in the across-word boundary condition, which in turn gives rise to lower jaw position. Speech rate-dependent spatial reduction in lip aperture may be able to be resolved by means of manipulating activation time of an active tract variable in the gestural score level.

Coordinative movement of articulators in bilabial stop /p/*

Speech articulators are coordinated for the purpose of segmental constriction in terms of a task. In particular, vertical jaw movements repeatedly contribute to consonantal as well as vocalic constriction. The current study explores vertical jaw movements in conjunction with bilabial constriction in bilabial stop /p/ in the context /a/-to-/a/. Revisiting kinematic data of /p/ collected using the electromagenetic midsagittal articulometer (EMMA) method from seven (four female and three male) speakers of Seoul Korean, we examined maximum vertical jaw position, its relative timing with respect to the upper and lower lips, and lip aperture minima. The results of those dependent variables are recapitulated in terms of linguistic (different word boundaries) and paralinguistic (different speech rates) factors as follows. Firstly, maximum jaw height was lower in the across-word boundary condition (across-word < within-word), but it did not differ as a function of different speech rates (comfortable = fast). Secondly, more reduction in the lip aperture (LA) gesture occurred in fast rate, while word-boundary effects were absent. Thirdly, jaw raising was still in progress after the lips’ positional extrema were achieved in the within-word condition, while the former was completed before the latter in the across-word condition. Lastly, relative temporal lags between the jaw and the lips (UL and LL) were more synchronous in fast rate, compared to comfortable rate. When these results are considered together, it is possible to posit that speakers are not tolerant of lenition to the extent that it is potentially realized as a labial approximant in either word-boundary condition while jaw height still manifested lower jaw position in the across-word boundary condition. Early termination of vertical jaw maxima before vertical lower lip maxima across-word condition may be partly responsible for the spatial reduction of jaw raising movements. This may come about as a consequence of an excessive number of factors (e.g., upper lip height (UH), lower lip height (LH), jaw angle (JA)) for the representation of a vector with two degrees of freedom (x, y) engaged in a gesture-based task (e.g., lip aperture (LA)). In the task-dynamic application toolkit, the jaw angle parameter can be assigned numerical values for greater weight in the across-word boundary condition, which in turn gives rise to lower jaw position. Speech rate-dependent spatial reduction in lip aperture may be able to be resolved by means of manipulating activation time of an active tract variable in the gestural score level.

Libras and Articulatory Phonology

This paper proposes a new approach to the phonological representation of Brazilian Sign Language (Libras). We depart from the observation that traditional analyses have overlooked features of signed languages which have no (exact) correspondence in spoken languages. Moreover, traditional approaches impose spoken language theoretical constructs on signed languages analyses and, by doing so, they disregard the possibility that signed languages follow different principles, as well as that analytical categories for spoken languages may be inaccurate for signed languages. Therefore, we argue that an approach grounded on a general theory of movement can account for signed language phonology in a more accurate way. Following Articulatory Phonology, we propose the analytical primes for a motor-oriented phonological approach to Libras, i.e., we determine which are the articulatory gestures that constitute the lexical items in a signed language. Besides, we propose a representation for the sign BEETLE-CAR in terms of a gestural score, and explain how gestures coordinate in relation to each other. As it is discussed, this approach allows us to more satisfactorily explain cases of variation attested in our data.

Early Cognitive, Structural, and Microstructural Changes in Presymptomatic C9orf72 Carriers Younger Than 40 Years

Presymptomatic carriers of chromosome 9 open reading frame 72 (C9orf72) mutation, the most frequent genetic cause of frontotemporal lobar degeneration and amyotrophic lateral sclerosis, represent the optimal target population for the development of disease-modifying drugs. Preclinical biomarkers are needed to monitor the effect of therapeutic interventions in this population. To assess the occurrence of cognitive, structural, and microstructural changes in presymptomatic C9orf72 carriers. The PREV-DEMALS study is a prospective, multicenter, observational study of first-degree relatives of individuals carrying the C9orf72 mutation. Eighty-four participants entered the study between October 2015 and April 2017; 80 (95%) were included in cross-sectional analyses of baseline data. All participants underwent neuropsychological testing and magnetic resonance imaging; 63 (79%) underwent diffusion tensor magnetic resonance imaging. Gray matter volumes and diffusion tensor imaging metrics were calculated within regions of interest. Anatomical and microstructural differences between individuals who carried the C9orf72 mutation (C9+) and those who did not carry the C9orf72 mutation (C9-) were assessed using linear mixed-effects models. Data were analyzed from October 2015 to April 2017. Differences in neuropsychological scores, gray matter volume, and white matter integrity between C9+ and C9- individuals. Of the 80 included participants, there were 41 C9+ individuals (24 [59%] female; mean [SD] age, 39.8 [11.1] years) and 39 C9- individuals (24 [62%] female; mean [SD] age, 45.2 [13.9] years). Compared with C9- individuals, C9+ individuals had lower mean (SD) praxis scores (163.4 [6.1] vs 165.3 [5.9]; P = .01) and intransitive gesture scores (34.9 [1.6] vs 35.7 [1.5]; P = .004), atrophy in 8 cortical regions of interest and in the right thalamus, and white matter alterations in 8 tracts. When restricting the analyses to participants younger than 40 years, compared with C9- individuals, C9+ individuals had lower praxis scores and intransitive gesture scores, atrophy in 4 cortical regions of interest and in the right thalamus, and white matter alterations in 2 tracts. Cognitive, structural, and microstructural alterations are detectable in young C9+ individuals. Early and subtle praxis alterations, underpinned by focal atrophy of the left supramarginal gyrus, may represent an early and nonevolving phenotype related to neurodevelopmental effects of C9orf72 mutation. White matter alterations reflect the future phenotype of frontotemporal lobar degeneration/amyotrophic lateral sclerosis, while atrophy appears more diffuse. Our results contribute to a better understanding of the preclinical phase of C9orf72 disease and of the respective contribution of magnetic resonance biomarkers. clinicaltrials.gov Identifier: NCT02590276.

Links between Gestures and Multisensory Processing: Individual Differences Suggest a Compensation Mechanism.

Speech-associated gestures represent an important communication modality. However, individual differences in the production and perception of gestures are not well understood so far. We hypothesized that the perception of multisensory action consequences might play a crucial role. Verbal communication involves continuous calibration of audio–visual information produced by the speakers. The effective production and perception of gestures supporting this process could depend on the given capacities to perceive multisensory information accurately. We explored the association between the production and perception of gestures and the monitoring of multisensory action consequences in a sample of 31 participants. We applied a recently introduced gesture scale to assess self-reported gesture production and perception in everyday life situations. In the perceptual experiment, we presented unimodal (visual) and bimodal (visual and auditory) sensory outcomes with various delays after a self-initiated (active) or externally generated (passive) button press. Participants had to report whether they detected a delay between the button press and the visual stimulus. We derived psychometric functions for each condition and determined points of subjective equality, reflecting detection thresholds for delays. Results support a robust link between gesture scores and detection thresholds. Individuals with higher detection thresholds (lower performance) reported more frequent gesture production and perception and furthermore profited more from multisensory information in the experimental task. We propose that our findings indicate a compensational function of multisensory processing as a basis for individual differences in both action outcome monitoring and gesture production and perception in everyday life situations.

Do speakers have access to a mental syllabary? ERP comparison of high frequency and novel syllable production

The transformation of an abstract phonological code into articulation has been hypothesized to involve the retrieval of stored syllable-sized motor plans. Accordingly, gestural scores for frequently used syllables are retrieved from memory whereas gestural scores for novel and possibly low frequency syllables are assembled on-line. The present study was designed to test this hypothesis. Participants produced disyllabic pseudowords with high frequency, low frequency and non-existent (novel) initial syllables. Behavioral results revealed slower production latencies for novel than for high frequency syllables. Event-related potentials diverged in waveform amplitudes and global topographic patterns between high frequency and low frequency/novel syllables around 170ms before the onset of articulation. These differences indicate the recruitment of different brain networks during the production of frequent and infrequent/novel syllables, in line with the hypothesis that speakers store syllabic-sized motor programs for frequent syllables and assemble these motor plans on-line for low frequency and novel syllables.

A procedure for estimating gestural scores from speech acoustics

Speech can be represented as a constellation of constricting vocal tract actions called gestures, whose temporal patterning with respect to one another is expressed in a gestural score. Current speech datasets do not come with gestural annotation and no formal gestural annotation procedure exists at present. This paper describes an iterative analysis-by-synthesis landmark-based time-warping architecture to perform gestural annotation of natural speech. For a given utterance, the Haskins Laboratories Task Dynamics and Application (TADA) model is employed to generate a corresponding prototype gestural score. The gestural score is temporally optimized through an iterative timing-warping process such that the acoustic distance between the original and TADA-synthesized speech is minimized. This paper demonstrates that the proposed iterative approach is superior to conventional acoustically-referenced dynamic timing-warping procedures and provides reliable gestural annotation for speech datasets.

'Blood Piano': Music, Femininity and the Abject – A Practice-Led Enquiry

Recent theoretical study in the field of musical analysis asserts that music performance may be understood as a transgressive encounter based on the writings of George Bataille (Bataille, 2008 [1957]; Rebelo, 2006; Waterman, 2008). Through identifying an overlap between choreographic methodology and the examination of instrumental gesture, I propose ways in which these theoretical findings may be translated into practice-led research. Arising within the context of gesture-based research as a stimuli for choreographic activity, I discuss musical practice as a mode of simulation marked out by physical mimesis. I propose the manifestation of effort through physical gesture in music performance as an expressive component conducive to live art interpretation where the performer's body plays a key role. Richard Leppert's discussion of music and femininity as historically synonymous furthers this research by contextualising the abject within the framework of the transgressive (Leppert, 1995 [1993]; Kristeva, 1982 [1980]). The gestural repertoires developed through instrumental practice with particular reflection on piano performance are well documented by Henry Shaffer, Francois Delalande, Jane Davidson, Eric Clarke and Mark Thompson (Shaffer, 1976, 1981, 1984; Delalande, 1988, 1995; Davidson, 2007; Clarke, 2006; Thompson, 2007). The kinds of gestural scores generated in such analyses - such as Delalande's classification of gesture types based on the playing of Glenn Gould - would usefully lend themselves to a choreographic interpretation (Delalande, 1988). Indeed, annotations of musical scores with gestural motifs, such as Thompson's analysis of Gould's playing of the 'Goldberg Variations' by J.S. Bach are redolent of the gestural scores generated in workshops led by choreographer Jonathan Burrows. Burrows, in collaboration with composer Matteo Fargion, has explored a number of choreographic interventions informed by musical structure. In a Burrows/Fargion workshop as part of the Bodysurf project at Findhorn in 2009, participants were encouraged to develop score-based exercises adapting rhythmical frameworks arising within John Cage's 'Lecture on Nothing' (1959). Within this framework, individuals were encouraged to explore different combinations of sitting gestures, vocal sounds and text while at the same time adhering to patterns in keeping with Kevin Volans's notion of variability. 1 However, it has yet to be explored how gestural analyses within music performance research might be usefully translated into choreographed material that could be used to comment upon the physical aspects of music making, with special attention to ways in which material could be made available to non-instrumentalists.

Gestures in optimality theory and the laryngeal phonology of Faroese

This paper presents an Optimality Theoretic analysis of aspiration in the Scandinavian language Faroese using gestural score representations from Articulatory Phonology. In Faroese, a strong parallel exists between pre- and post-aspiration such that neither realization appears to be inherently preferred. This symmetry is straightforwardly captured by alignment within gestural scores, but runs counter to the treatments of pre-aspiration in traditional linear and autosegmental theories, where such segments are inherently more complex than their post-aspirated counterparts. An OT analysis that makes use of the alignment-based distinction in gestural scores is therefore proposed for the Faroese data. The account incorporates cross-linguistic preferences for post-aspiration and the correlation observed between stress and aspiration, yet pre-aspiration surfaces in many of the Faroese environments. Laryngeal alignment preferences are also analyzed as the basis of apparent contrast neutralization and sonorant devoicing in consonant sequences, and provide the framework for a grounded analysis of the interaction seen among consonant length, vowel quality and aspiration. Based on the analytic benefits of incorporating rich representational structure into the OT framework seen here and elsewhere, the potential for such theories to contribute to the advancement of phonological theory appears to be considerable.

Virtual Gesture Control and Synthesis of Music Performances: Qualitative Evaluation of Synthesized Timpani Exercises

The increasing availability of software for creating real-time simulations of musical instrument sounds allows for the design of new visual and sounding media. These past decades have especially focused on the control of real and virtual instruments by natural gestures. In this paper, we present and extensively evaluate a framework (Figure 1) for the control of virtual percussion instruments, by modeling and simulating virtual percussionists gestures. By positioning the virtual performer at the center of the gesture-sound synthesis system, we aim at providing original tools to analyze and synthesize instrumental gesture performances. Our physics-based approach for gesture simulation brings some insight into the effect of biomechanical parameters of the gesture on the instrumental performance. Simulating both gesture and sound by physical models leads also to a coherent and human-centered interaction and provides new ways of exploring the mapping between gesture and sound. The use of motion capture data enables the realistic synthesis of both pre-recorded and novel percussion sequences from the specification of gesture scores. Such scores involve motion editing techniques applied to simple beat attacks. We therefore propose an original gesture language based on the instrumental playing techniques. This language is characterized by expressivity, interactivity with the user, and the possibility to take into account co-articulation between gesture units. Finally, providing 3D visual rendering synchronized with sound rendering allows us to observe virtual performances to the light of real ones, and to qualitatively evaluate both pedagogical and compositional capabilities of such a system.

Early indicators of autism spectrum disorders at 12 and 24 months of age: A prospective, longitudinal comparative study

Prospective questionnaire data from a longitudinal population sample on children with autism spectrum disorders (ASD), developmental delay, specific language impairment, or typical development (TD), were collected at ages eight, 12 and 24 months, via the Communication and Symbolic Behavior Scale Developmental Profile (CSBS) - Infant Toddler Checklist, and the Actions and Gesture section of the MacArthur-Bates Communicative Development Inventory (CDI):Words and Gestures. The four groups were compared at four years of age to identify whether any early behaviours differentiated the groups. While children with ASD differed from TD children on most social communicative measures by 12 months of age, the only social communication characteristic which could differentiate the children with ASD from the other groups were gesture scores on the CDI at 12 months and the CSBS at 24 months. Significant markers of ASD were identifiable in this community sample at an early age, although discrimination between clinical groups was rarely evident.

ASL sign lowering as undershoot: A corpus study

AbstractAmerican Sign Language (ASL) signs that are located on the forehead in their canonical form are often articulated lower during natural signing. Previous studies have examined this phenomenon from a phonetic perspective, treating it as a form of undershoot, and from a variationist sociolinguistic perspective, treating it as a categorical process. This study sees if the findings and explanations of these studies can be extended to the lowering of signs formed in locations other than the forehead. In a corpus of natural conversational signing from six signers, we measure the vertical displacement of over 3000 tokens of signs canonically formed at the face, head, or neck. While there is some apparent evidence for a categorical lowering process in a minority of signs and considerable evidence for undershoot, neither alone can explain the full range of displacement patterns across all signs and locations. Undershoot must be carefully planned and controlled: no matter how sloppy the signing, signers systematically avoid contacting their eyes. The results can be explained if there is a somatosensory forward-modelling mechanism that can veto undesirable gestural scores and whose decisions are incorporated during learning into the phonological distributions representing the locations of signs.