Gesture Kinematics Research Articles

Sensitivity and specificity of the action observation network to kinematics, target object, and gesture meaning.

Hierarchical models have been proposed to explain how the brain encodes actions, whereby different areas represent different features, such as gesture kinematics, target object, action goal, and meaning. The visual processing of action-related information is distributed over a well-known network of brain regions spanning separate anatomical areas, attuned to specific stimulus properties, and referred to as action observation network (AON). To determine the brain organization of these features, we measured representational geometries during the observation of a large set of transitive and intransitive gestures in two independent functional magnetic resonance imaging experiments. We provided evidence for a partial dissociation between kinematics, object characteristics, and action meaning in the occipito-parietal, ventro-temporal, and lateral occipito-temporal cortex, respectively. Importantly, most of the AON showed low specificity to all the explored features, and representational spaces sharing similar information content were spread across the cortex without being anatomically adjacent. Overall, our results support the notion that the AON relies on overlapping and distributed coding and may act as a unique representational space instead of mapping features in a modular and segregated manner.

Identification of movement phenotypes from occupational gesture kinematics: Advancing individual ergonomic exposure classification and personalized training

The identification of personalized preventive strategies plays a major role in contrasting the occurrence of work-related musculoskeletal disorders. This requires the identification of distinct movement patterns within large samples and the attribution of a proper risk level to each identified movement phenotype. We assessed the feasibility of this approach by exploiting wearable inertial measurement units to estimate the whole-body kinematics of 43 healthy participants performing 18 reach-to-manipulate movements, which differed based on the object's position in the space and the type of manipulation required. Through unsupervised clustering, we identified multiple movement phenotypes graded by ergonomic performance. Furthermore, we determined which joints mostly contributed to instantiating the ergonomic differences across clusters, emphasizing the importance of monitoring this aspect during occupational gestures. Overall, our analysis suggests that movement phenotypes can be identified within occupational motor repertoires. Assigning individual performance to specific phenotypes has the potential to inform the development of more effective and tailored interventions.

Articulatory overlap as a function of stiffness in German, English and Spanish word-initial stop-lateral clusters

Recently, Roon, Hoole, Zeroual, Du, and Gafos (2021) have offered evidence that an empirical estimate of stiffness, a parameter in the dynamics hypothesized to control the kinematics of gestures, modulates overlap in consonant clusters of Moroccan Arabic. Here, we extend the Roon et al. (2021) study in two ways. First, empirically, we broaden the assessment of the role of stiffness in overlap by examining data from three languages, largely confirming the Roon et al. results with electromagnetic articulography recordings from stop-lateral clusters spoken by German, English, and Spanish participants. Second, we examine for the first time the extent to which C1 and C2 stiffness individually, as opposed to the combined measure of stiffness difference in earlier work, contributes to overlap with the main finding being that the significant effect of stiffness difference is mainly due to C2 stiffness; C1 stiffness is not as robust as C2 stiffness in modulating overlap. These results are placed in the context of other work on articulatory and perceptual influences on inter-consonantal timing as well as in the broader context of work on the coupling between perception and action in other domains of skilled action.

Gesture-vocal coupling in Karnatak music performance: A neuro-bodily distributed aesthetic entanglement.

In many musical styles, vocalists manually gesture while they sing. Coupling between gesture kinematics and vocalization has been examined in speech contexts, but it is an open question how these couple in music making. We examine this in a corpus of South Indian, Karnatak vocal music that includes motion-capture data. Through peak magnitude analysis (linear mixed regression) and continuous time-series analyses (generalized additive modeling), we assessed whether vocal trajectories around peaks in vertical velocity, speed, or acceleration were coupling with changes in vocal acoustics (namely, F0 and amplitude). Kinematic coupling was stronger for F0 change versus amplitude, pointing to F0's musical significance. Acceleration was the most predictive for F0 change and had the most reliable magnitude coupling, showing a one-third power relation. That acceleration, rather than other kinematics, is maximally predictive for vocalization is interesting because acceleration entails force transfers onto the body. As a theoretical contribution, we argue that gesturing in musical contexts should be understood in relation to the physical connections between gesturing and vocal production that are brought into harmony with the vocalists' (enculturated) performance goals. Gesture-vocal coupling should, therefore, be viewed as a neuro-bodily distributed aesthetic entanglement.

Electromyographic and kinematic evaluation of bench press exercise: a case report study on athletes with different impairments and expertise

PurposeWith an increase in the number of adapted sports, the need to monitor sports performance in people with different abilities has grown. Indeed, a thorough evaluation of the sports gesture could prevent the occurrence of injuries, enable a continuous performance assessment, and allow to verify the compliance of the requirements for the competitions. Gesture kinematics provides an assessment of performance, while the muscle activities reveal the underlying strategies adopted by each athlete. In this context, we propose an instrumented evaluation to assess performance in Para-powerlifting. Our goal is to define and test a setup and a protocol to quantitatively assess the execution of bench press exercise in athletes with different abilities.MethodsWe recruited an unimpaired athlete and three Paralympic athletes. They were requested to execute the bench press exercise while we recorded muscle activity and kinematic data from the upper body. We investigated the sport gesture by extracting parameters describing coordination, symmetry, and synchronism between arms, and motor variability while repeating the gesture.ResultsParalympic athletes performed the gestures with higher coordination between arms and low variability across repetitions compared to the unimpaired athlete, who was not at the Olympic level. All participants obtained similar kinematic performance by adopting different muscle strategies.ConclusionsThis study is a proof of concept that the instrumented evaluation proposed here can allow to conduct a complete assessment of the bench press exercise, in terms of kinematics, muscle activity and performance in athletes with different abilities.

Kinematic-Based Classification of Social Gestures and Grasping by Humans and Machine Learning Techniques.

The affective motion of humans conveys messages that other humans perceive and understand without conventional linguistic processing. This ability to classify human movement into meaningful gestures or segments plays also a critical role in creating social interaction between humans and robots. In the research presented here, grasping and social gesture recognition by humans and four machine learning techniques (k-Nearest Neighbor, Locality-Sensitive Hashing Forest, Random Forest and Support Vector Machine) is assessed by using human classification data as a reference for evaluating the classification performance of machine learning techniques for thirty hand/arm gestures. The gestures are rated according to the extent of grasping motion on one task and the extent to which the same gestures are perceived as social according to another task. The results indicate that humans clearly rate differently according to the two different tasks. The machine learning techniques provide a similar classification of the actions according to grasping kinematics and social quality. Furthermore, there is a strong association between gesture kinematics and judgments of grasping and the social quality of the hand/arm gestures. Our results support previous research on intention-from-movement understanding that demonstrates the reliance on kinematic information for perceiving the social aspects and intentions in different grasping actions as well as communicative point-light actions.

Differences in the production and perception of communicative kinematics in autism.

In human communication, social intentions and meaning are often revealed in the way we move. In this study, we investigate the flexibility of human communication in terms of kinematic modulation in a clinical population, namely, autistic individuals. The aim of this study was twofold: to assess (a) whether communicatively relevant kinematic features of gestures differ between autistic and neurotypical individuals, and (b) if autistic individuals use communicative kinematic modulation to support gesture recognition. We tested autistic and neurotypical individuals on a silent gesture production task and a gesture comprehension task. We measured movement during the gesture production task using a Kinect motion tracking device in order to determine if autistic individuals differed from neurotypical individuals in their gesture kinematics. For the gesture comprehension task, we assessed whether autistic individuals used communicatively relevant kinematic cues to support recognition. This was done by using stick‐light figures as stimuli and testing for a correlation between the kinematics of these videos and recognition performance. We found that (a) silent gestures produced by autistic and neurotypical individuals differ in communicatively relevant kinematic features, such as the number of meaningful holds between movements, and (b) while autistic individuals are overall unimpaired at recognizing gestures, they processed repetition and complexity, measured as the amount of submovements perceived, differently than neurotypicals do. These findings highlight how subtle aspects of neurotypical behavior can be experienced differently by autistic individuals. They further demonstrate the relationship between movement kinematics and social interaction in high‐functioning autistic individuals.Lay SummaryHand gestures are an important part of how we communicate, and the way that we move when gesturing can influence how easy a gesture is to understand. We studied how autistic and typical individuals produce and recognize hand gestures, and how this relates to movement characteristics. We found that autistic individuals moved differently when gesturing compared to typical individuals. In addition, while autistic individuals were not worse at recognizing gestures, they differed from typical individuals in how they interpreted certain movement characteristics.

Speakers exhibit a multimodal Lombard effect in noise

In everyday conversation, we are often challenged with communicating in non-ideal settings, such as in noise. Increased speech intensity and larger mouth movements are used to overcome noise in constrained settings (the Lombard effect). How we adapt to noise in face-to-face interaction, the natural environment of human language use, where manual gestures are ubiquitous, is currently unknown. We asked Dutch adults to wear headphones with varying levels of multi-talker babble while attempting to communicate action verbs to one another. Using quantitative motion capture and acoustic analyses, we found that (1) noise is associated with increased speech intensity and enhanced gesture kinematics and mouth movements, and (2) acoustic modulation only occurs when gestures are not present, while kinematic modulation occurs regardless of co-occurring speech. Thus, in face-to-face encounters the Lombard effect is not constrained to speech but is a multimodal phenomenon where the visual channel carries most of the communicative burden.

A Systematic Investigation of Gesture Kinematics in Evolving Manual Languages in the Lab.

Silent gestures consist of complex multi‐articulatory movements but are now primarily studied through categorical coding of the referential gesture content. The relation of categorical linguistic content with continuous kinematics is therefore poorly understood. Here, we reanalyzed the video data from a gestural evolution experiment (Motamedi, Schouwstra, Smith, Culbertson, & Kirby, 2019), which showed increases in the systematicity of gesture content over time. We applied computer vision techniques to quantify the kinematics of the original data. Our kinematic analyses demonstrated that gestures become more efficient and less complex in their kinematics over generations of learners. We further detect the systematicity of gesture form on the level of thegesture kinematic interrelations, which directly scales with the systematicity obtained on semantic coding of the gestures. Thus, from continuous kinematics alone, we can tap into linguistic aspects that were previously only approachable through categorical coding of meaning. Finally, going beyond issues of systematicity, we show how unique gesture kinematic dialects emerged over generations as isolated chains of participants gradually diverged over iterations from other chains. We, thereby, conclude that gestures can come to embody the linguistic system at the level of interrelationships between communicative tokens, which should calibrate our theories about form and linguistic content.

Biomechanical analysis of practitioner's gesture for peripheral venous catheter insertion

Peripheral venous catheter insertion (PVCI) is one of the most common procedures performed by healthcare professionals but remains technically difficult. To develop new medical simulators with better representativeness of the human forearm, an experimental study was performed to collect data related to the puncturing of human skin and a vein in the antebrachial area. A total of 31 volunteers participated in this study. Force sensors and digital image correlation were used to measure the force during the palpation and puncturing of the vein and to retrieve the kinematics of the practitioner's gesture. The in vivo skin rupture load, vein rupture load, and friction loads for skin only and for both the skin and vein were (mean ± standard deviation) 0.85 ± 0.34 N, 1.25 ± 0.37 N, –0.49 ± 0.19 N, and –0.51 ± 0.16 N, respectively. The results of this study can be used to develop realistic skin and vein substitutes and mechanically assess them by reproducing the practitioner's gesture in a controlled fashion.

Electrophysiological indicators of gesture perception.

Electroencephalography (EEG) activity in the mu frequency band (8-13Hz) is suppressed during both gesture performance and observation. However, it is not clear if or how particular characteristics within the kinematic execution of gestures map onto dynamic changes in mu activity. Mapping the time course of gesture kinematics onto that of mu activity could help understand which aspects of gestures capture attention and aid in the classification of communicative intent. In this work, we test whether the timing of inflection points within gesture kinematics predicts the occurrence of oscillatory mu activity during passive gesture observation. The timing for salient features of performed gestures in video stimuli was determined by isolating inflection points in the hands' motion trajectories. Participants passively viewed the gesture videos while continuous EEG data was collected. We used wavelet analysis to extract mu oscillations at 11Hz and at central electrodes and occipital electrodes. We used linear regression to test for associations between the timing of inflection points in motion trajectories and mu oscillations that generalized across gesture stimuli. Separately, we also tested whether inflection point occurrences evoked mu/alpha responses that generalized across participants. Across all gestures and inflection points, and pooled across participants, peaks in 11Hz EEG waveforms were detected 465 and 535ms after inflection points at occipital and central electrodes, respectively. A regression model showed that inflection points in the motion trajectories strongly predicted subsequent mu oscillations ([Formula: see text]<0.01); effects were weaker and non-significant for low (17Hz) and high (21Hz) beta activity. When segmented by inflection point occurrence rather than stimulus onset and testing participants as a random effect, inflection points evoked mu and beta activity from 308 to 364ms at central electrodes, and broad activity from 226 to 800ms at occipital electrodes. The results suggest that inflection points in gesture trajectories elicit coordinated activity in the visual and motor cortices, with prominent activity in the mu/alpha frequency band and extending into the beta frequency band. The time course of activity indicates that visual processing drives subsequent activity in the motor cortex during gesture processing, with a lag of approximately 80ms.

Gesture Networks: Introducing Dynamic Time Warping and Network Analysis for the Kinematic Study of Gesture Ensembles

ABSTRACTWe introduce applications of established methods in time-series and network analysis that we jointly apply here for the kinematic study of gesture ensembles. We define a gesture ensemble as the set of gestures produced during discourse by a single person or a group of persons. Here we are interested in how gestures kinematically relate to one another. We use a bivariate time-series analysis called dynamic time warping to assess how similar each gesture is to other gestures in the ensemble in terms of their velocity profiles (as well as studying multivariate cases with gesture velocity and speech amplitude envelope profiles). By relating each gesture event to all other gesture events produced in the ensemble, we obtain a weighted matrix that essentially represents a network of similarity relationships. We can therefore apply network analysis that can gauge, for example, how diverse or coherent certain gestures are with respect to the gesture ensemble. We believe these analyses promise to be of great value for gesture studies, as we can come to understand how low-level gesture features (kinematics of gesture) relate to the higher-order organizational structures present at the level of discourse.

The quantification of gesture\u2013speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

There is increasing evidence that hand gestures and speech synchronize their activity on multiple dimensions and timescales. For example, gesture’s kinematic peaks (e.g., maximum speed) are coupled with prosodic markers in speech. Such coupling operates on very short timescales at the level of syllables (200 ms), and therefore requires high-resolution measurement of gesture kinematics and speech acoustics. High-resolution speech analysis is common for gesture studies, given that field’s classic ties with (psycho)linguistics. However, the field has lagged behind in the objective study of gesture kinematics (e.g., as compared to research on instrumental action). Often kinematic peaks in gesture are measured by eye, where a “moment of maximum effort” is determined by several raters. In the present article, we provide a tutorial on more efficient methods to quantify the temporal properties of gesture kinematics, in which we focus on common challenges and possible solutions that come with the complexities of studying multimodal language. We further introduce and compare, using an actual gesture dataset (392 gesture events), the performance of two video-based motion-tracking methods (deep learning vs. pixel change) against a high-performance wired motion-tracking system (Polhemus Liberty). We show that the videography methods perform well in the temporal estimation of kinematic peaks, and thus provide a cheap alternative to expensive motion-tracking systems. We hope that the present article incites gesture researchers to embark on the widespread objective study of gesture kinematics and their relation to speech.

The communicative advantage: how kinematic signaling supports semantic comprehension

Humans are unique in their ability to communicate information through representational gestures which visually simulate an action (eg. moving hands as if opening a jar). Previous research indicates that the intention to communicate modulates the kinematics (e.g., velocity, size) of such gestures. If and how this modulation influences addressees’ comprehension of gestures have not been investigated. Here we ask whether communicative kinematic modulation enhances semantic comprehension (i.e., identification) of gestures. We additionally investigate whether any comprehension advantage is due to enhanced early identification or late identification. Participants (n = 20) watched videos of representational gestures produced in a more- (n = 60) or less-communicative (n = 60) context and performed a forced-choice recognition task. We tested the isolated role of kinematics by removing visibility of actor’s faces in Experiment I, and by reducing the stimuli to stick-light figures in Experiment II. Three video lengths were used to disentangle early identification from late identification. Accuracy and response time quantified main effects. Kinematic modulation was tested for correlations with task performance. We found higher gesture identification performance in more- compared to less-communicative gestures. However, early identification was only enhanced within a full visual context, while late identification occurred even when viewing isolated kinematics. Additionally, temporally segmented acts with more post-stroke holds were associated with higher accuracy. Our results demonstrate that communicative signaling, interacting with other visual cues, generally supports gesture identification, while kinematic modulation specifically enhances late identification in the absence of other cues. Results provide insights into mutual understanding processes as well as creating artificial communicative agents.

Quantitative analysis of multimodal speech data

This study presents techniques for quantitatively analyzing coordination and kinematics in multimodal speech using video, audio and electromagnetic articulography (EMA) data. Multimodal speech research has flourished due to recent improvements in technology, yet gesture detection/annotation strategies vary widely, leading to difficulty in generalizing across studies and in advancing this field of research. We describe how FlowAnalyzer software can be used to extract kinematic signals from basic video recordings; and we apply a technique, derived from speech kinematic research, to detect bodily gestures in these kinematic signals. We investigate whether kinematic characteristics of multimodal speech differ dependent on communicative context, and we find that these contexts can be distinguished quantitatively, suggesting a way to improve and standardize existing gesture identification/annotation strategy. We also discuss a method, Correlation Map Analysis (CMA), for quantifying the relationship between speech and bodily gesture kinematics over time. We describe potential applications of CMA to multimodal speech research, such as describing characteristics of speech-gesture coordination in different communicative contexts. The use of the techniques presented here can improve and advance multimodal speech and gesture research by applying quantitative methods in the detection and description of multimodal speech.

Communicative intent modulates production and comprehension of actions and gestures: A Kinect study

Actions may be used to directly act on the world around us, or as a means of communication. Effective communication requires the addressee to recognize the act as being communicative. Humans are sensitive to ostensive communicative cues, such as direct eye gaze (Csibra & Gergely, 2009). However, there may be additional cues present in the action or gesture itself. Here we investigate features that characterize the initiation of a communicative interaction in both production and comprehension.We asked 40 participants to perform 31 pairs of object-directed actions and representational gestures in more- or less- communicative contexts. Data were collected using motion capture technology for kinematics and video recording for eye-gaze. With these data, we focused on two issues. First, if and how actions and gestures are systematically modulated when performed in a communicative context. Second, if observers exploit such kinematic information to classify an act as communicative.Our study showed that during production the communicative context modulates space–time dimensions of kinematics and elicits an increase in addressee-directed eye-gaze. Naïve participants detected communicative intent in actions and gestures preferentially using eye-gaze information, only utilizing kinematic information when eye-gaze was unavailable.Our study highlights the general communicative modulation of action and gesture kinematics during production but also shows that addressees only exploit this modulation to recognize communicative intention in the absence of eye-gaze. We discuss these findings in terms of distinctive but potentially overlapping functions of addressee directed eye-gaze and kinematic modulations within the wider context of human communication and learning.

Coupling relations underlying the production of speech articulator movements and their invariance to speech rate.

Since the seminal works of Bernstein (The coordination and regulation of movements. Pergamon Press, Oxford, 1967) several authors have supported the idea that, to produce a goal-oriented movement in general, and a movement of the organs responsible for the production of speech sounds in particular, individuals activate a set of coupling relations that coordinate the behavior of the elements of the motor system involved in the production of the target movement or sound. In order to characterize the configurations of the coupling relations underlying speech production articulator movements, we introduce an original method based on recurrence analysis. The method is validated through the analysis of simulated dynamical systems adapted to reproduce the features of speech gesture kinematics and it is applied to the analysis of speech articulator movements recorded in five German speakers during the production of labial and coronal plosive and fricative consonants at variable speech rates. We were able to show that the underlying coupling relations change systematically between labial and coronal consonants, but are not affected by speech rate, despite the presence of qualitative changes observed in the trajectory of the jaw at fast speech rate.

Communication for coordination: gesture kinematics and conventionality affect synchronization success in piano duos

Ensemble musicians often exchange visual cues in the form of body gestures (e.g., rhythmic head nods) to help coordinate piece entrances. These cues must communicate beats clearly, especially if the piece requires interperformer synchronization of the first chord. This study aimed to (1) replicate prior findings suggesting that points of peak acceleration in head gestures communicate beat position and (2) identify the kinematic features of head gestures that encourage successful synchronization. It was expected that increased precision of the alignment between leaders’ head gestures and first note onsets, increased gesture smoothness, magnitude, and prototypicality, and increased leader ensemble/conducting experience would improve gesture synchronizability. Audio/MIDI and motion capture recordings were made of piano duos performing short musical passages under assigned leader/follower conditions. The leader of each trial listened to a particular tempo over headphones, then cued their partner in at the given tempo, without speaking. A subset of motion capture recordings were then presented as point-light videos with corresponding audio to a sample of musicians who tapped in synchrony with the beat. Musicians were found to align their first taps with the period of deceleration following acceleration peaks in leaders’ head gestures, suggesting that acceleration patterns communicate beat position. Musicians’ synchronization with leaders’ first onsets improved as cueing gesture smoothness and magnitude increased and prototypicality decreased. Synchronization was also more successful with more experienced leaders’ gestures. These results might be applied to interactive systems using gesture recognition or reproduction for music-making tasks (e.g., intelligent accompaniment systems).

Recognition of the semantics and kinematics of gestures: Neural responses to “what” and “how”?

The extensive use of gestures for human-human communication, independently of culture and language, suggests an underlying universal neural mechanism for gesture recognition. The mirror neuron system (MNS) is known to respond to observed human actions, and overlaps with self-action. The minimal cues needed for activation of the MNS for gesture recognition, facial expressions and bodily dynamics, is not yet defined. Using LED-point representations of gestures, we compared two types of brain activations: 1) in response to human recognizable vs non-recognizable motion and 2) in response to human vs non-human motion. Our preliminary results show that parts of the MNS respond only to human kinematics, and not to nonhuman kinematics, suggesting that the brain has a mechanism of discriminating human from nonhuman motion, even if the pattern of motion is meaningless, but still follows biological motion patterns. This implies that mechanisms of learning-mimicking, empathy and emotional communication, are possibly constrained by biological motion patterns. We then suggest a two-tier-model of human-bodily-communication: (1) recognition of human biological kinematics; (2) recognition of meaning. Implications are both theoretical (understanding the underlying mechanism for action recognition) and applicative (in digital graphical social representations, motion should be reasonably biological to generate the same emotional and mimicking automatic mechanisms as in face-to-face social interactions).

Toward the Autism Motor Signature: Gesture patterns during smart tablet gameplay identify children with autism.

Autism is a developmental disorder evident from infancy. Yet, its clinical identification requires expert diagnostic training. New evidence indicates disruption to motor timing and integration may underpin the disorder, providing a potential new computational marker for its early identification. In this study, we employed smart tablet computers with touch-sensitive screens and embedded inertial movement sensors to record the movement kinematics and gesture forces made by 37 children 3–6 years old with autism and 45 age- and gender-matched children developing typically. Machine learning analysis of the children’s motor patterns identified autism with up to 93% accuracy. Analysis revealed these patterns consisted of greater forces at contact and with a different distribution of forces within a gesture, and gesture kinematics were faster and larger, with more distal use of space. These data support the notion disruption to movement is core feature of autism, and demonstrate autism can be computationally assessed by fun, smart device gameplay.