Gesture Acquisition Research Articles

Shall We Play the Same? Pedagogical Perspectives on Infants' and Children's Imitation of Musical Gestures.

Imitation, both gestural and vocal, has been acknowledged to be at the origin of human communication (Donald, 1991). Music is often considered to be the first means of communication of emotion via both vocal and gestural synchronization (Malloch, 1999; Malloch and Trevarthen, 2009). Instrumental music is part of the human heritage for more than 35,000 years before our era (Aimé et al., 2020). However, very little is known about the acquisition of gestures that produce sounds (i.e., musical gestures) and their role in the development of music and musicality. In the present paper, we propose that studying early synchronous imitation of musical gestures is essential both for investigating the development of the early action–perception system and for outlining early music interventions during infancy. We designed double musical objects which can be used in preschool music education for prompting synchronic imitation of musical gestures between adult and child, and between dyads of infants. We conclude by proposing a novel pedagogical perspective in music education for the early years which links the privileged orientation of infants and children towards sound discoveries with the development of perception-action coupling via imitation of musical gestures.

A new window onto animal culture

AbstractScientific interest in the diversity of gestural signalling dates back to the figure of Charles Darwin. More than a hundred years later, there is a considerable body of work describing human gestural diversity across languages and cultures. However, the question of communicative culture in our closest living relatives, the nonhuman primates, is relatively unexplored. Here, we will stir new interest into this topic by (i) briefly summarizing the current knowledge of animal culture, and (ii) presenting the current knowledge on gesture cultures, diversity and usage in the most common model for early hominid behaviour, the chimpanzee (Pan troglodytes). We will focus particularly on well-established behaviours being customary in some and absent in other chimpanzee communities, and recently discovered social customs that have been suggested to differ in their form, and/or meaning across populations. We also introduce latest findings on chimpanzees’ gestural diversity, providing further evidence for the role social negotiation plays in gestural acquisition. We conclude that the field has been hampered by misconstruing great ape gestures as fixed action patterns, a strong research bias on the perspective of signalers, and a lack of coherent methodology to assess the meaning and context of gestures across sites. We argue for systematic cross-site comparisons by viewing communicative exchanges as negotiations, enabling a unique perspective onto the evolutionary trajectory of culture and communication.

The origins of gestures and language: history, current advances and proposed theories.

Investigating in depth the mechanisms underlying human and non-human primate intentional communication systems (involving gestures, vocalisations, facial expressions and eye behaviours) can shed light on the evolutionary roots of language. Reports on non-human primates, particularly great apes, suggest that gestural communication would have been a crucial prerequisite for the emergence of language, mainly based on the evidence of large communication repertoires and their associated multifaceted nature of intentionality that are key properties of language. Such research fuels important debates on the origins of gestures and language. We review here three non-mutually exclusive processes that can explain mainly great apes' gestural acquisition and development: phylogenetic ritualisation, ontogenetic ritualisation, and learning via social negotiation. We hypothesise the following scenario for the evolutionary origins of gestures: gestures would have appeared gradually through evolution via signal ritualisation following the principle of derived activities, with the key involvement of emotional expression and processing. The increasing level of complexity of socioecological lifestyles and associated daily manipulative activities might then have enabled the acquisition and development of different interactional strategies throughout the life cycle. Many studies support a multimodal origin of language. However, we stress that the origins of language are not only multimodal, but more broadly multicausal. We propose a multicausal theory of language origins which better explains current findings. It postulates that primates' communicative signalling is a complex trait continually shaped by a cost-benefit trade-off of signal production and processing of interactants in relation to four closely interlinked categories of evolutionary and life cycle factors: species, individual and context-related characteristics as well as behaviour and its characteristics. We conclude by suggesting directions for future research to improve our understanding of the evolutionary roots of gestures and language.

A systematic review on hand gesture recognition techniques, challenges and applications.

BackgroundWith the development of today’s technology, and as humans tend to naturally use hand gestures in their communication process to clarify their intentions, hand gesture recognition is considered to be an important part of Human Computer Interaction (HCI), which gives computers the ability of capturing and interpreting hand gestures, and executing commands afterwards. The aim of this study is to perform a systematic literature review for identifying the most prominent techniques, applications and challenges in hand gesture recognition.MethodologyTo conduct this systematic review, we have screened 560 papers retrieved from IEEE Explore published from the year 2016 to 2018, in the searching process keywords such as “hand gesture recognition” and “hand gesture techniques” have been used. However, to focus the scope of the study 465 papers have been excluded. Only the most relevant hand gesture recognition works to the research questions, and the well-organized papers have been studied.ResultsThe results of this paper can be summarized as the following; the surface electromyography (sEMG) sensors with wearable hand gesture devices were the most acquisition tool used in the work studied, also Artificial Neural Network (ANN) was the most applied classifier, the most popular application was using hand gestures for sign language, the dominant environmental surrounding factor that affected the accuracy was the background color, and finally the problem of overfitting in the datasets was highly experienced.ConclusionsThe paper will discuss the gesture acquisition methods, the feature extraction process, the classification of hand gestures, the applications that were recently proposed, the challenges that face researchers in the hand gesture recognition process, and the future of hand gesture recognition. We shall also introduce the most recent research from the year 2016 to the year 2018 in the field of hand gesture recognition for the first time.

Gesturing towards the future: cognition, big data, and the future of comparative gesture research.

The field of ape gesture research has grown significantly in the past two decades, but progress on the question of gesture development has been limited by methodological and terminological disagreements, small sample sizes, and a lack of fine-grained longitudinal data. The main theories of gesture acquisition are often portrayed as mutually exclusive, but only some theories actually detail learning mechanisms, and differences in the level of analysis may help explain some of the apparent disagreements. Gesture research would benefit greatly from the articulation of more testable hypotheses. We propose two hypotheses that follow from dominant theories of gesture acquisition. We urge scholars to collect new data and leverage existing data in ways that maximize the potential for comparison across datasets and articulation with studies of other communicative modalities. Finally, we advocate for a transition away from using intentionality as a marker of the 'special status' of gesture, and towards using gesture as a window onto the lives and minds of apes.

The development of gestural communication in great apes

Great apes deploy gestural signals in goal-directed and flexible ways across a wide range of social contexts. Despite growing evidence for profound effects of developmental experience on social cognition, socio-ecological factors shaping gesture use are still poorly understood, particularly in apes living in their natural environment. After discussing current ambiguities in terminology and methods, we review recent work implementing a longitudinal and/or cross-sectional approach in great ape gesture acquisition (phylogenetic and ontogenetic origins) and development (ontogenetic trajectories). To understand whether and to what extent the socio-ecological environment influences gestural communication, it is essential to distinguish between the gestural repertoire and gesture usage, which represent different levels of analysis. While the majority of the apes’ gestural repertoire seems to be innate, accumulating evidence shows that the communicative usage of these signals is substantially affected by interactional experiences throughout ontogeny. Nevertheless, since great ape communication is intrinsically multimodal, future developmental research on gesture should incorporate other modes of communication. Great apes navigate their differentiated social relationships by means of a large and flexible repertoire of gestures. However, gestural ontogeny is still poorly understood, particularly in primates living in their natural environment. We first discuss how the different usages of the term ‘gesture’ have led to a number of apparently disparate views, but highlight that these perspectives each provide their own contribution and may be reconciled by considering them as different levels of explanation. We then review recent studies on the various individual and social factors shaping the gestural use in great apes throughout development, highlighting the impact of socio-ecological factors. While the majority of the apes’ gestural repertoire seems to be innate, the communicative usage of these signals is substantially affected by interactional experiences throughout ontogeny. Given that ape gestural signals are inherently multimodal and are then often combined with other communicative signals, a broad multimodal perspective on gesture is important in order to gain a thorough understanding of the developmental processes underlying great ape communication.

Variability in Verbal and Nonverbal Communication in Infants at Risk for Autism Spectrum Disorder: Predictors and Outcomes.

Early communication impairment is among the most-reported first concerns in parents of young children with autism spectrum disorder (ASD). Using a parent-report questionnaire, we derived trajectory groups for early language and gesture acquisition in siblings at high risk for ASD and in children at low risk, during their first 2 years of life. Developmental skills at 6months were associated with trajectory group membership representing growth in receptive language and gestures. Behavioral symptoms also predicted gesture development. All communication measures were strongly related to clinical and developmental outcomes. Trajectory groups further indicated slowest language/gesture acquisition in infants with later ASD diagnoses, in particular when associated with language delay. Overall, our results confirm considerable variability in communication development in high-risk infants.

How primates acquire their gestures: evaluating current theories and evidence.

Mechanisms underlying gesture acquisition in primates are largely unstudied, yet heavily debated. While some studies suggest that gestural repertoires are largely innate, others emphasize that gestures emerge and are shaped in social interactions with other conspecifics. There is agreement, however, regarding the negligible role of imitation for the acquisition of gestures. The aim of this paper is to provide an overview of the current knowledge about gesture acquisition in nonhuman primates, to introduce the corresponding mechanisms suggested to be involved, and to discuss how findings from current studies support the different theories of gesture acquisition. We suggest that seemingly inconsistent findings across different research groups can be reconciled by pointing to differences between their research foci as well as methods of data collection. The additional comparison of the developmental pathways of gestural and facialcommunication highlights thecomplex interplay of genetic as well as social factors involved in shaping a species repertoire. We close by proposing that extending longitudinal studies, which capture the onset and usage of gestures in young primates, and which include the comparisons of severalspecies and groups in different environments, will enable us to better understand developmental pathways of gestural communication in primates.

Gestural acquisition in great apes: the Social Negotiation Hypothesis

Scientific interest in the acquisition of gestural signalling dates back to the heroic figure of Charles Darwin. More than a hundred years later, we still know relatively little about the underlying evolutionary and developmental pathways involved. Here, we shed new light on this topic by providing the first systematic, quantitative comparison of gestural development in two different chimpanzee (Pan troglodytes verus and Pan troglodytes schweinfurthii) subspecies and communities living in their natural environments. We conclude that the three most predominant perspectives on gestural acquisition—Phylogenetic Ritualization, Social Transmission via Imitation, and Ontogenetic Ritualization—do not satisfactorily explain our current findings on gestural interactions in chimpanzees in the wild. In contrast, we argue that the role of interactional experience and social exposure on gestural acquisition and communicative development has been strongly underestimated. We introduce the revised Social Negotiation Hypothesis and conclude with a brief set of empirical desiderata for instigating more research into this intriguing research domain.

KIND‐DAMA: A modular middleware for Kinect‐like device data management

SummaryIn the last decades, we have witnessed a growing interest toward touchless gestural user interfaces. Among other reasons, this is due to the large availability of different low‐cost gesture acquisition hardware (the so‐called “Kinect‐like devices”). As a consequence, there is a growing need for solutions that allow to easily integrate such devices within actual systems. In this paper, we present KIND‐DAMA, an open and modular middleware that helps in the development of interactive applications based on gestural input. We first review the existing middlewares for gestural data management. Then, we describe the proposed architecture and compare its features against the existing similar solutions we found in the literature. Finally, we present a set of studies and use cases that show the effectiveness of our proposal in some possible real‐world scenarios.

Gestural development of chimpanzees in the wild: the impact of interactional experience

To understand the complexity involved in animal signalling, studies have mainly focused on repertoire size and information conveyed in vocalizations of birds and nonhuman primates. However, recent studies on gestural abilities of nonhuman primates have shown that we also need a detailed understanding of other communicative modalities and underlying cognitive skills to grasp this phenomenon in detail. Here, we thus examined gestural signalling of chimpanzees, Pan troglodytes, living in two communities in the wild (Kanyawara, Uganda; Tai South, Cote d'Ivoire) with a special focus on the influence of the social environment on signal development. Specifically, we investigated to what extent specific social factors, namely behavioural context, interaction rates and maternal proximity, affect gestural production (i.e. gesture frequency, sequences and repertoire size). We used a combination of video recordings and focal scans obtained from 11 infants aged between 9 and 69 months during 1145 h of observation throughout two consecutive field periods. Overall, we found that social play was the context in which the highest number of gestures occurred. While gesture frequency and repertoire size increased with higher interaction rates with nonmaternal conspecifics and the number of previous interaction partners, no effect was found for interaction rates with mothers. Our results thus imply that infants of social mothers may have a head start in life. Moreover, we provide hitherto undocumented evidence for sex differences in gestural signalling, which may reflect the differential importance of early socialization for chimpanzee males and females. Gestural development thus relies heavily on interactional experiences with conspecifics, which adds support for gestural acquisition via the learning mechanism of ‘social negotiation’ in great apes.

Statistical models for the indirect acquisition of violin bowing controls from audio analysis

Acoustical studies of sound production in bowed string instruments show that there is a direct relationship between bowing controls and the sound produced. During note sustains the three major bowing parameters that determine the characteristics of the sound are the bowing force (bforce), the bowing distance to the bridge (bbd) and the bowing velocity (bvel). During note attacks the third parameter is acceleration (bacc) rather than velocity. We are interested in understanding this correspondence between bowing controls and sound but in the opposite direction, i.e., mapping sound features extracted from an audio recording to the original bowing controls used. This inverse process is usually called indirect acquisition of gestures and is of great interest in different fields ranging from acoustics and sound synthesis to motor learning or augmented performances. Indirect acquisition is based on the processing of the audio signal and it is usually informed on acoustical or physical properties of the sound or sound production mechanism. In this paper, we present indirect acquisition methods of violin controls from an audio signal based on training of statistical models with a database of multimodal data (control and sound) from violin performances.

Hand gesture recognition using valley circle feature and Hu’s moments technique for robot movement control

Abstract Hand gesture recognition is a simpler and more natural way of human computer interaction. The goal of this paper is to detect the continuous gestures and use them to convey information for the robot movement control. So the hand gesture recognition requires fast and extremely robust. In this paper, three strategies were used to realize the hand gesture recognition: (1) the valley circle (VC) was created for the first stage of 6 fingertip numbers classification; (2) the hybrid feature vector of Hu’s moments, convexity and compactness (HCC) were constructed for the second stage of gesture recognition of the remainder unknown gesture classes; (3) a new template matching recognition (NTMR) algorithm was proposed to realize 10 gesture classes recognition. To test the hand gesture recognition method, the robot movement control system was built. It is experimental proved that the NTMR algorithm is effective and corrective for the hand gesture recognition. It increased the recognition accuracy by 4% and decreased the recognition duration by 112 ms compared with Hu’s moment method. It had good performances of the real-time hand gesture acquisition and information conveyance, and it had the invariant properties when the gesture was rotated and shifted and scaled.

Inertial Motion Sensing Glove for Sign Language Gesture Acquisition and Recognition

The most popular systems for automatic sign language recognition are based on vision. They are user-friendly, but very sensitive to changes in regard to recording conditions. This paper presents a description of the construction of a more robust system—an accelerometer glove—as well as its application in the recognition of sign language gestures. The basic data regarding inertial motion sensors and the design of the gesture acquisition system as well as project proposals are presented. The evaluation of the solution presents the results of the gesture recognition attempt by using a selected set of sign language gestures with a described method based on Hidden Markov Model (HMM) and parallel HMM approaches. The proposed usage of parallel HMM for sensor-fusion modeling reduced the equal error rate by more than 60%, while preserving 99.75% recognition accuracy.

Editorial: Is the Language Faculty Nonlinguistic?

Editorial: Is the Language Faculty Nonlinguistic?

Should I stay or should I go? Initiation of joint travel in mother-infant dyads of two chimpanzee communities in the wild.

It is well established that great apes communicate via intentionally produced, elaborate and flexible gestural means. Yet relatively little is known about the most fundamental steps into this communicative endeavour—communicative exchanges of mother–infant dyads and gestural acquisition; perhaps because the majority of studies concerned captive groups and single communities in the wild only. Here, we report the first systematic, quantitative comparison of communicative interactions of mother–infant dyads in two communities of wild chimpanzees by focusing on a single communicative function: initiation of carries for joint travel. Over 156 days of observation, we recorded 442 actions, 599 cases of intentional gesture production, 51 multi-modal combinations and 80 vocalisations in the Kanyawara community, Kibale National Park, Uganda, and the Taï South community, Taï National Park, Côte d’Ivoire. Our results showed that (1) mothers and infants differed concerning the signal frequency and modality employed to initiate joint travel, (2) concordance rates of mothers’ gestural production were relatively low within but also between communities, (3) infant communicative development is characterised by a shift from mainly vocal to gestural means, and (4) chimpanzee mothers adjusted their signals to the communicative level of their infants. Since neither genetic channelling nor ontogenetic ritualization explains our results satisfactorily, we propose a revised theory of gestural acquisition, social negotiation, in which gestures are the output of social shaping, shared understanding and mutual construction in real time by both interactants.Electronic supplementary materialThe online version of this article (doi:10.1007/s10071-015-0948-z) contains supplementary material, which is available to authorized users.

Natural interaction for online documentation in industrial maintenance

This paper presents a novel application of natural interaction techniques for technical documentation navigation in augmented reality maintenance. The objective is to provide technical information where and when needed in order to reduce the time spent in retrieving procedural data from manuals. The system integrates a video and depth camera for user gesture acquisition, a webcam for marker-based tracking and a see through head mounted display for digital information overlay. We designed a novel hand free gesture-based interface to access technical documentation while performing maintenance tasks. We implemented a specific flexible system to prototype natural interfaces using UML configurable state machines. It supports 2D and 3D content navigation, CAD assembly explosion, real time 3D sectioning and operation browsing. We differentiate the interaction metaphors for 2D documents and 3D CAD models in order to be intuitive and easy to learn. A case study and a user test demonstrated the feasibility and the good acceptance of the presented gesture interface.

A Smart phone Identification Method Based on Gesture

In order to promote the practicality of smart phone identification based on gesture, we introduce weighted morphological characteristics and early termination of authentication into dynamic time warping (DTW) algorithm, which based on characteristic moving data captured by in-built accelerometer and gyroscopes, and put forward an effective identification algorithm (ME-DTW) for smartphone. The algorithm controls the contribution made by the difference between the latitudes to the total Euclidean distance by smartphone morphological characteristics. It also introduces restricted areas touch trigger gesture acquisition scheme and authentication gesture length selection scheme based on normal distribution, which effectively improve the identification accuracy and efficiency. Experimental results show that: when others imitate gestures attack false acceptance rate tends to 0%, the personal identification false rejection rate remained at 3.29%, which can meet most practical security needs

The Design, Implementation, and Testing of a Sensorized MRI-Compatible Cello

The design and testing of optical sensors for a cello compatible with magnetic resonance imaging (MRI) are presented. The interface is used in neuroimaging experiments, allowing for the first time, the acquisition of cello performance gestures captured inside the MRI scanner. Left-hand fingering and right-hand bowing gestures were captured with optical sensors embedded in the fingerboard, bridge, and bow. Finger-string interaction was sensed through diffuse reflection using plastic optical fibres potted in a 3D-printed fingerboard. Acoustic string vibration was sensed through the transmissive amplitude modulation of an emitting–receiving fibre pair mounted in a specially designed bridge. Bow position and force was transduced with a combination of optical sensors, including optical flow, macrobending loss under compressive strain, and with a three-axis Faraday rotator. The sensors provided sufficient resolution and dynamic range to measure cello gestures inside the MRI scanner, and no interference from the scanner was noted in the acquired signals. Further MRI compatibility testing confirmed that the interface and sensors had no meaningful effect on field homogeneity and no image artifacts were noted. The interface is currently being used in neuroscience experiments.

On free-hand TV control: experimental results on user-elicited gestures with Leap Motion

We present insights from a gesture elicitation study conducted for TV control, during which 18 participants contributed gesture commands and rated the execution difficulty and recall likeliness of free-hand gestures for 21 television control tasks. Our study complements previous work on gesture interaction design for the TV set with the first exploration of fine-grained resolution 3-D finger movements and hand gestures. We report lower agreement rates than previous gesture studies ( $${\mathcal {AR}}=.158$$ ) with 72.8 % recall rate and 15.8 % false positives, results that are explained by the complexity and variability of unconstrained finger and hand gestures. However, our observations also confirm previous findings, such as people preferring related gestures for dichotomous tasks and more disagreement occurring for abstract tasks, such as “open browser” or “show the list of channels” for our specific TV scenario. To reach a better understanding of our participants’ preferences for articulating finger and hand gestures, we defined five measures for Leap Motion gestures, such as gesture volume and finger-to-palm distance, which we employed to evaluate gestures performed by our participants. We also contribute a set of guidelines for practitioners interested in designing free-hand gestures for interactive TV scenarios involving similar gesture acquisition technology. We release our dataset consisting in 378 Leap Motion gestures described by fingertips position, direction, and velocity coordinates to foster further studies in the community. This first exploration of viewers’ preferences for fine-grained resolution free-hand gestures for TV control represents one more step toward designing low-effort gesture interfaces for lean-back interaction with the TV set.