Complex Gestures Research Articles

Grammatical aspect, gesture, and conceptualization: Using co-speech gesture to reveal event representations

AbstractGrammatical aspect is a pervasive linguistic device that, according to linguistic analyses, allows speakers to encode different ways of construing events. For instance, the progressive (I was writing a book) is thought to reflect increased focus on the internal details of an event, as contrasted with the perfect (I had written a book). However, experimental evidence that speakers describing events using progressive versus non-progressive aspect are in fact thinking about the same events differently is lacking. We used co-speech gesture as a means to investigate what speakers' event representations are like when they produce progressive versus non-progressive utterances. We found that progressive event descriptions were accompanied by longer-lasting and more complex gestures, but only when participants described events originally presented to them in the progressive. This evidence suggests that people are actually construing events differently when they use different grammatical aspects, but that the aspect originally used to encode the events plays a role as well.

Avoiding Conflicting Requirements: Development of Home Healthcare Devices Through an Interdisciplinary Approach

<i>Avoiding Conflicting Requirements:</i> Development of Home Healthcare Devices Through an Interdisciplinary Approach

Gesture comprehension, knowledge and production in Alzheimer’s disease

Although apraxia is a typical consequence of Alzheimer's disease (AD), the profile of apraxic impairments is still subject to debate. Here, we analysed apraxia components in patients with AD with mild-to-moderate or moderately severe dementia. Thirty-one patients were included. We first evaluated simple gestures, that is, the imitation of finger and hand configurations, symbolic gestures (recognition, production on verbal command and imitation), pantomimes (recognition, production on verbal command, imitation and production with the object), general knowledge and complex gestures (tool-object association, function-tool association, production of complex actions and knowledge about action sequences). Tests for dementia (Mini Mental State Examination and the Dementia Rating Scale), language disorders, visual agnosia and executive function were also administered. Compared with controls, patients showed significant difficulties (P ≤ 0.01) in subtests relating to simple gestures (except for the recognition and imitation of symbolic gestures). General knowledge about tools, objects and action sequences was less severely impaired. Performance was frequently correlated with the severity of dementia. Multiple-case analyses revealed that (i) the frequency of apraxia depended on the definition used, (ii) ideomotor apraxia was more frequent than ideational apraxia, (iii) conceptual difficulties were slightly more frequent than production difficulties in the early stage of AD and (iv) difficulties in gesture recognition were frequent (especially for pantomimes). Patients with AD can clearly show gesture apraxia from the mild-moderate stage of dementia onwards. Recognition and imitation disorders are relatively frequent (especially for pantomimes). We did not find conceptual difficulties to be the main problem in early-stage AD.

A Simultaneous Gesture of Proximity and Distance: W.G. Sebald's Empathic Narrative Persona

W.G. Sebald’s prose narratives exist at the borderline of the novel form. Their self-conscious hybridity, combining memoir, historical account, travelogue, and fiction, may be seen as pushing the boundaries of genre. But Sebald’s use of a narrator-figure with some biographical correspondence to the author, who takes part in the action, enables an even greater crossing of borders: those between past and present, memory and history, and current and previous generations. This insertion of what I call an “empathic narrative persona” between author and subject helps enable an approach to the past in which proximity and distance occur simultaneously in a complex gesture of empathy. By situating these narratives in the context of the Sebaldian persona, I show how the author self-reflexively foregrounds the process of LaCaprian “empathic unsettlement” by establishing close personal connection with the victims of history while simultaneously inserting several layers of structural and epistemological distance.

Estimation of independent and dependent components of non-invasive EMG using fast ICA: validation in recognising complex gestures

The identification of a number of active muscles during complex actions is the useful information to identify different gestures. Biosignals such as surface electromyogram (sEMG) are a result of the summation of electrical activity of a number of sources. The complexity of the anatomy and actions makes it difficult in identifying the number of active sources from the multiple channel recordings. This paper addresses two applications of independent component analysis (ICA) on sEMG: the first one is to evaluate the use of ICA for the separation of bioelectric signals when the number of active sources may not be known. The second application is to identify complex hand gestures using decomposed sEMG. The theoretical analysis and experimental results demonstrate that the ICA is suitable for the separation of myoelectric signals. The results identify the usage of ICA for identifying complex gestures.

Accelerometer Based Gesture Recognition Using Fusion Features and SVM

In this paper, a gesture recognition system based on single tri-axis accelerometer mounted on a cell phone is proposed. We present a novel human computer interaction for cell phone through recognizing seventeen complex gestures. A new feature fusion method for gesture recognition based on time-domain and frequency-domain is proposed. First of all, we extract the time-domain features from acceleration data, that is short-time energy. Secondly, we extract the hybrid features which combine Wavelet Packet Decomposition with Fast Fourier Transform. Finally, we fuse these two categories features together and employ the principal component analysis to reduce dimension of fusion features. The Classifier we used is Multi-class Support Vector Machine. The average recognition results of seventeen complex gestures using the proposed fusion feature are 89.89%, which better than previous works. The performance of experimental results show that gesture-based interaction can be used as a novel human computer interaction for mobile device and consumer electronics.

Estimation and filtering of physiological tremor for real‐time compensation in surgical robotics applications

Physiological tremor is the main cause of imprecision in microsurgical procedures/robotics applications. Existing methods, such as weighted-frequency Fourier linear combiner (WFLC), rely on estimating the tremor under the assumption that it has a single dominant frequency. This paper focuses on developing a new algorithm for accurate tremor filtering in real time. A study conducted on several novice subjects and microsurgeons showed the tremor to contain several dominant frequencies in a band, rather than a single dominant frequency. Based on the tremor characteristics, a new algorithm band-limited multiple Fourier linear combiner (BMFLC) has been developed to estimate a band of signals with multiple dominant frequencies. A separation procedure to separate the intended motion/drift from the tremor portion is also discussed. A simulation study was first conducted to validate the theoretical development on recorded tremor data. The experimental set-up was designed to study the real-time performance of the proposed algorithm. Tremor sensing using accelerometers is also discussed, with the proposed algorithm. Our experiments showed that the developed BMFLC algorithm had an average tremor compensation of 64% compared to 43% for the WFLC algorithm in real-time for one degree of freedom (1-DOF) cancellation of tremor. The BMFLC algorithm can be applied for the three axes separately and 3-DOF cancellation of tremor can be achieved. Further research is required to deal with complex gestures involved during microsurgery.

Rhythmic muscular activation pattern for fast figure-eight movement

To address the question of how the CNS generates muscle activation patterns for complex gestures, we have chosen to study a figure-eight movement. We hypothesized that the well defined rhythmic aspect of this figure will provide further insights into the temporal features of multi-muscular commands. Subjects performed, as fast as possible, figure-eights initiated in the center of the figure with 4 different initial directions and 2 positions of the shoulder. We extracted the temporal modulation of the EMG patterns by calculating conjugate cross-correlation functions. (1) The muscular command was tuned with respect to the rotational direction of the figure-eight, (2) two sets of synergistic muscles acted in a reciprocal mode, and (3) these reciprocal commands presented an invariant temporal correlation with the spatial component of the velocity having the highest frequency. Our results suggest that the rhythmic features of certain drawing movements favor the partitioning of the muscles into synergistic groups acting in a reciprocal mode. The inclusion of an individual muscle in one group or the other takes into account the expected number of changes of direction in the movement as a whole. Muscular temporal synergies may depend on the rhythmic features of the trajectory.

When knowing can replace seeing in audiovisual integration of actions

The ability to predict the effects of actions is necessary to behave properly in our physical and social world. Here, we describe how the ability to predict the consequence of complex gestures can change the way we integrate sight and sound when relevant visual information is missing. Six drummers and six novices were asked to judge audiovisual synchrony for drumming point-light displays where the visual information was manipulated to eliminate or include the drumstick–drumhead impact point. In the condition with only the arm information novices were unable to detect asynchrony whereas drummers were able to. Additionally, in the conditions that included the impact point drummers perceived the best alignment when the sight preceded the sound, while in the arm only condition they perceived the best alignment when the sound occurred together with or preceded the sight, as it would be expected if they were predicting the beat occurrence. Taken together these findings suggest that humans can acquire, through practice, internal models of action which can be used to replace missing information when integrating multisensory signals from the environment.

Neural Dynamics of Learning Sound—Action Associations

A motor component is pre-requisite to any communicative act as one must inherently move to communicate. To learn to make a communicative act, the brain must be able to dynamically associate arbitrary percepts to the neural substrate underlying the pre-requisite motor activity. We aimed to investigate whether brain regions involved in complex gestures (ventral pre-motor cortex, Brodmann Area 44) were involved in mediating association between novel abstract auditory stimuli and novel gestural movements. In a functional resonance imaging (fMRI) study we asked participants to learn associations between previously unrelated novel sounds and meaningless gestures inside the scanner. We use functional connectivity analysis to eliminate the often present confound of ‘strategic covert naming’ when dealing with BA44 and to rule out effects of non-specific reductions in signal. Brodmann Area 44, a region incorporating Broca's region showed strong, bilateral, negative correlation of BOLD (blood oxygen level dependent) response with learning of sound-action associations during data acquisition. Left-inferior-parietal-lobule (l-IPL) and bilateral loci in and around visual area V5, right-orbital-frontal-gyrus, right-hippocampus, left-para-hippocampus, right-head-of-caudate, right-insula and left-lingual-gyrus also showed decreases in BOLD response with learning. Concurrent with these decreases in BOLD response, an increasing connectivity between areas of the imaged network as well as the right-middle-frontal-gyrus with rising learning performance was revealed by a psychophysiological interaction (PPI) analysis. The increasing connectivity therefore occurs within an increasingly energy efficient network as learning proceeds. Strongest learning related connectivity between regions was found when analysing BA44 and l-IPL seeds. The results clearly show that BA44 and l-IPL is dynamically involved in linking gesture and sound and therefore provides evidence that one of the mechanisms required for the evolution of human communication is found within these motor regions.

Recognition of Complex Dynamic Gesture Based on HMM-FNN Model

复杂动态手势识别是利用视频手势进行人机交互的关键问题.提出一种HMM-FNN模型结构.它整合了隐马尔可夫模型对时序数据的建模能力与模糊神经网络的模糊规则构建与推理能力,并将其应用到复杂动态手势的识别中.复杂动态手势具备两大特点:运动特征的可分解性与定义描述的模糊性.针对这两种特性,复杂手势被分解为手形变化、2D平面运动与Z轴方向运动3个子部分,分别利用HMM进行建模,HMM模型对观察子序列的似然概率被作为FNN的模糊隶属度,通过模糊规则推理,最终得到手势的分类类别.HMM-FNN方法将高维手势特征分解为低;Recognition of complex dynamic gesture is a key issue for visual gesture-based human-computer interaction. In this paper, an HMM-FNN model is proposed for gesture recognition, which combines ability of HMM model for temporal data modeling with that of fuzzy neural network for fuzzy rule modeling and fuzzy inference. Complex dynamic gesture has two important properties: Its motion can be decomposed and usually being defined in a fuzzy way. By HMM-FNN, complex gesture is firstly decomposed into three components: Posture changing, movement in 2D plane and movement in Z-axis direction, each of which is modeled by HMM. The likelihood of each HMM to observation sequence is considered as membership value of FNN, and gesture is classified through fuzzy inference of FNN. In this proposed method, high-dimensional gesture feature is transformed into several low-dimensional features, as a result, computational complexity is reduced. Furthermore, human's experience or prior knowledge can be used to build and optimize model structure. Experimental results show that the proposed method is an effective method for recognition of complex dynamic gesture, and is superior to conventional HMM method.

LIMITATIONS AND APPLICATIONS OF ICA FOR SURFACE ELECTROMYOGRAM FOR IDENTIFYING HAND GESTURES

Surface electromyogram (SEMG) has numerous applications, but the presence of artifacts and cross talk especially at low level of muscle activity makes the recordings unreliable. Spectral and temporal overlap can make the removal of artifacts and noise, or separation of relevant signals from other bioelectric signals extremely difficult. Identification of hand gestures using low level of SEMG is one application that has a number of applications but the presence of high level of cross talk makes such an application highly unreliable. Individual muscles may be considered as independent at the local level and this makes an argument for separating the signals using independent component analysis (ICA). In the recent past, due to the easy availability of ICA tools, a number of researchers have attempted to use ICA for this application. This paper reports research conducted to evaluate the use of ICA for the separation of muscle activity and removal of the artifacts from SEMG. It discusses some of the conditions that could affect the reliability of the separation and evaluates issues related to the properties of the signals and a number of sources. This paper also identifies the lack of suitable measure of quality of separation for bioelectric signals and it recommends and tests a more robust measure of separation. This paper also proposes semi-blind ICA approach with the combination of prior knowledge of SEMG sources with ICA to identify hand gestures using low level of SEMG recordings. The theoretical analysis and experimental results demonstrate that ICA is suitable for SEMG signals. The results demonstrate the limitations of such applications due to the inability of the system to identify the correct order and magnitude of the signals. This paper determines the suitability of the use of error between estimated and actual mixing matrix as a mean for identifying the quality of separation of the output. This work also demonstrates that semi-blind ICA can accurately identify complex hand gestures from the low-level SEMG recordings.

The performance of hand postures in front- and back-of-device interaction for mobile computing

Three studies of different mobile-device hand postures are presented. The first study measures the performance of postures in Fitts’ law tasks using one and two hands, thumbs and index fingers, horizontal and vertical movements, and front- and back-of-device interaction. Results indicate that the index finger performs well on both the front and the back of the device, and that thumb performance on the front of the device is generally worse. Fitts’ law models are created and serve as a basis for comparisons. The second study examines the orientation of shapes on the front and back of a mobile device. It shows that participants’ expectations of visual feedback for finger movements on the back of a device reverse the direction of their finger movements to favor a “transparent device” orientation. The third study examines letter-like gestures made on the front and back of a device. It confirms the performance of the index finger on the front of the device, while showing limitations in the ability for the index finger on the back to perform complex gestures. Taken together, these results provide an empirical foundation upon which new mobile interaction designs can be based. A set of design implications and recommendations are given based directly on the findings presented.

Ensemble Musicians’ Synchronization With Conductors’ Gestures: An Automated Feature-Extraction Analysis

Previous work suggests that the perception of a visual beat in conductors’ gestures is related to certain physical characteristics of the movements they produce, most notably to periods of negative acceleration, and low position in the vertical axis. These findings are based on studies that have presented participants with somewhat simple gestures, and in which participants have been required to simply tap in time with the beat. Thus, it is not clear how generalizable these findings are to real-world conducting situations, in which a conductor uses considerably more complex gestures to direct an ensemble of musicians playing actual instruments. The aims of the present study were to examine the features of conductors’ gestures with which ensemble musicians synchronize their performance in an ecologically valid setting and to develop automatic feature extraction methods for the analysis of audio and movement data. An optical motion capture system was used to record the gestures of an expert conductor directing an ensemble of expert musicians over a 20-minute period. A simultaneous audio recording of the performance of the ensemble was also made and synchronized with the motion capture data. Four short excerpts were selected for analysis, two in which the conductor communicated the beat with high clarity, and two in which the beat was communicated with low clarity. Twelve movement variables were computationally extracted from the movement data and cross-correlated with the pulse of the ensemble’s performance, the latter based on the spectral flux of the audio signal. Results of the analysis indicated that the ensemble’s performance tended to be most highly synchronized with periods of maximal deceleration along the trajectory, followed by periods of high vertical velocity (a higher correlation than deceleration but a longer delay).

Limb apraxia and cognitive impairment in progressive supranuclear palsy

Limb apraxia occurs in progressive supranuclear palsy (PSP); however, its characteristics and relation to dementia are unclear. We compared 25 probable PSP patients on cognitive and ideomotor apraxia tests with 19 healthy controls. Eleven PSP patients were demented (D-PSP) and 14 not (ND-PSP). D-PSP patients did worse than controls and ND-PSP patients in all cognitive tests; and ND-PSP patients did worse than controls in all except the memory test. In the apraxia test, PSP patients committed mainly sequence errors and executed complex gestures badly. Apraxia correlated strongly with dementia. In PSP, limb apraxia seems mainly due to frontal deafferentation and lesions, but only when these are severe enough to cause dementia.

Book Review

Book Review

Gesture during speech in first and second language

Many of the hand-arm gestures that accompany speech — especially wide and complex (‘ideational’) gestures — seem to be associated with speech dysfluencies and, it has been suggested, may facilitate word retrieval. Such facilitation may originate in either semantic or phonological processes. The present study has investigated this issue by looking at ideational gestures in five subjects who had Hebrew as first (L1) and English as second (L2) language. Their performance was monitored during speech in L1 and L2, as well as during a-vista translation from L1 to L2 and vice versa. It has been assumed that speech in L2 involves more lexical search than speech in L1 on both semantic and phonological levels. By contrast, translation between languages probably puts a heavier load on phonological processes than on semantic processes. This implies that speech in L2 should generate the highest rates of ideational gestures; translation into L2 should show the lowest gesture rates, while speech in L1 should show intermediate rates. These predictions were borne out by our results.

In a complex sequential movement, what component of the motor program is improved with intensive practice, sequence timing or ensemble motor learning?

We investigated whether the higher maximum speed of complex athletic movements attained by experts compared with novices might solely be ascribed to the acquisition of a more efficient timing of the initiation of the simple movements that comprise complex sporting gestures. Six novices in fencing and five experts in fencing performed three experimental series: 7 trials of touche, 7 trials of lunge (control series) and 50 trials of lunge + touche (test series), where the touche and lunge were initiated with different chronological sequences imposed by the experimenter. The lunge and the touche can be assimilated to a fast forward step and to a pointing task, respectively. We compared the maximum speed of touche between the two groups, recorded by an accelerometer fixed to the hand guard of the foil, and the speed of the centre of mass obtained from a force plate. The speed of the centre of mass was not statistically different between the two groups in the control and in the test series. The maximum speed of touche was not statistically different between the two groups in the control series. In contrast, in the test series, the maximum speed of touche was higher in the fencers' than in novices' groups, while the timing of initiation of the lunge and the touche was similar. The results of the test series show that the higher maximum speed of touche of fencers compared with novices cannot solely be ascribed to the acquisition of a more efficient initiation timing in motor programming. In a complex fencing gesture, the higher maximum speed of touche following intensive practice is discussed with regard to the inhibition of negative effects linked to the "refractory period" associated with the initiation of two closely spaced motor programs.

Apraxia ideomotriz y habilidades visuoconstructivas

To study alterations in copying behaviour in Alzheimer's disease (AD) and its relationship with other cognitive parameters. As secondary objectives, in this paper we describe the evolution of concepts, definitions and examination of ideas-movement and constructive apraxia, together with its involvement in AD and review the investigations of the Uruguayan school on the development of behaviour copying and how this is affected in patients with cerebral lesions. We made a study of 82 consecutive patients with probable AD at stages 3 to 5 on the overall deterioration scale (GDS), registered in the Department of Neuropsychology, and a group of healthy persons, paired for age, sex and educational level. We used an extensive protocol for cognitive evaluation. We evaluated the idea-movement apraxia by means of imitation of increasingly complex gestures, and constructive apraxia using the protocol described by Mendilaharsu et al. At stages 4 and 5 of the GDS scale, the idea-movement apraxia was significantly correlated with digital agnosia and the anomias. Constructive apraxia was found in GDS stage 3. We found 'homogeneous cognitive deterioration' in advanced patients (GDS 5) who showed closing-in when copying figures. The closing-in correlated significantly with the presence of anomias, idea apraxia and digital agnosia. In the early stages, EA shows varied praxis profiles, but above stage 5 of the GDS scale there was a tendency to homogeneity.

Perception of self-generated movement following left parietal lesion.

Three apraxic patients with lesions in the left parietal cortex were required to execute finger movements with either hand, while the visual feedback they received about the movement was manipulated systematically. We used a device which allowed us to present on a video monitor either the patient's hand or the examiner's hand simultaneously performing an identical or a different movement. In each trial, patients were required to decide whether the hand shown on the screen was their own or not. Hand movements produced in response to verbal command included simple (single-finger extension) and complex gestures (multi-finger extension). Ownership judgements were analysed and compared with those produced by six normal controls and two non-apraxic neurological patients. Apraxic patients and controls accurately recognized their own hand on the screen (own movement condition) and correctly identified the viewed hand as the examiner's when it performed a movement different from their own movement (incongruent movement condition). However, when the viewed hand was the examiner's hand executing their own movement (congruent movement condition), apraxic patients were significantly more impaired than controls. When the results were analysed as a function of gesture type, the number of correct responses was significantly lower for apraxic patients with respect to controls only for complex gestures. Interestingly, when patients executed the finger gestures inaccurately, they still failed to recognize the examiner's hand as alien, and claimed that the correct movement presented on the screen was their own. These results confirm that parietal lesions alter the representational aspects of gestures, and suggest a failure in evaluating and comparing internal and external feedback about movement. We conclude that the parietal cortex plays an important role in generating and maintaining a kinaesthetic model of ongoing movements.