Gestural Activity Research Articles

Hand Gestures Recognition Based on One-Channel Surface EMG Signal

This paper presents an experiment using OPENBCI to collect data of two hand gestures and decoding the signal to distinguish gestures. The signal was extracted with three electrodes on the subject’s forearm and transferred in one channel. After utilizing a Butterworth bandpass filter, we chose a novel way to detect gesture action segment. Instead of using moving average algorithm, which is based on the calculation of energy, We developed an algorithm based on the Hilbert transform to find a dynamic threshold and identified the action segment. Four features have been extracted from each activity section, generating feature vectors for classification. During the process of classification, we made a comparison between K-nearest-neighbors (KNN) and support vector machine (SVM), based on a relatively small amount of samples. Most common experiments are based on a large quantity of data to pursue a highly fitted model. But there are certain circumstances where we cannot obtain enough training data, so it makes the exploration of best method to do classification under small sample data imperative. Though KNN is known for its simplicity and practicability, it is a relatively time-consuming method. On the other hand, SVM has a better performance in terms of time requirement and recognition accuracy, due to its application of different Risk Minimization Principle. Experimental results show an average recognition rate for the SVM algorithm that is 1.25% higher than for KNN while SVM is 2.031 s shorter than that KNN.

Research on Recognition of Nine Kinds of Fine Gestures Based on Adaptive AdaBoost Algorithm and Multi-Feature Combination

Accurate recognition of gestures based on surface EMG signals is of very importance in the study of human prosthetic interaction. In this paper, multi-feature combination and adaptive AdaBoost algorithm are used to identify nine kinds of fine gestures. First, in order to adaptively set the threshold for extracting the active segment, the algorithm for detecting the active segment of the surface myoelectric signal is improved, and the effective active segment value is obtained by rejecting the invalid active segment. Second, a fine gesture classification method based on multi-feature combination is proposed. In this paper, the recursive feature elimination algorithm based on cyclic feature selection is used to screen out the feature set with the top classification effect, and the feature set selected by eight experimental objects is counted, and the optimal feature combination is obtained according to the experimental analysis. Finally, nine fine gesture actions are identified through a weak classifier constructed by combining the AdaBoost algorithm and the single-layer decision tree. The experimental results show that the proposed algorithm can achieve an average recognition rate of 98.56 (±1.44)% for nine fine gestures.

A preference for non-invasive touch in caregiving contexts

This article analyses how a professional caregiver uses touch as a key resource when instructing and guiding a person with Parkinson’s disease. Touch is shown to have both facilitating and controlling functions in the accomplishment of everyday tasks in residential care. We find an orientation to touch as a sensitive action, invading the patient’s intimacy and right to self-determination. First, the semiotic resources occur in a successive order, where touch often occurs only when a verbal or gestural action has failed. Second, less invasive kinds of touch, such as patting, precede more invasive kinds, such as holding and shoving.

The flow of language: Respiration and the rhythmic motor control of speech

Speech rhythm can be described as the temporal patterning by which sequences of vocalic and gestural actions unfold, within and between interlocutors. Despite efforts to quantify and model speech rhythm across languages, it remains a scientifically enigmatic aspect of prosody. For example, the existence and/or form of a basic speech rhythmic unit is hotly contested. One challenge is that the primary means of speech rhythm research has been the analysis of the acoustic signal. As speech is multimodal and motoric, investigations of speech rhythm will likely benefit from a greater range of complementary measures, including physiological recordings. The current experiment explores respiratory effort as a contributing factor in speech rhythm production. Undergoing simultaneous inductive plethysmography and acoustic recording, participants produce speech by reading novel prosaic texts aloud in single and dyadic conditions. Speech-associated breathing patterns are analysed in conjunction with traditional acoustic-phonological approaches to speech rhythm. It is hypothesized that, like spoken utterances, speech breathing is subject to rhythmic constraints. Preliminary results are therefore interpreted with a focus on the temporal relationship between inhalation events and previously proposed phonological rhythmic units (e.g., inter-stress, p-centre, syllable), underscoring breathing as a necessary, yet often overlooked, component in speech rhythm planning and production.

Advances in description of 3D human motion

This paper aims to provide a comprehensive reference source on depth-based human motion descriptors. Motion description is a challenging problem which became popular with recent advances in 3D computer vision. Our purpose is twofold. First, we introduce the main trends in human 3D motion descriptor design and evaluation. Second, we present a review of recent methods belonging to three different application categories: action recognition, gesture recognition and gait assessment. Selected categories have different specifics, which allow us to highlight aspects of a motion descriptor construction. A comparison of different methods by their main characteristics is provided. Finally, possible directions and recommendations for future research in 3D motion description are outlined.

Emoji and communicative action: The semiotics, sequence and gestural actions of ‘face covering hand’

This paper uses conversation analysis to explore the communicative functions of one emoji in a mobile reading community in China. In contrast to semiotic approaches to emoji that focus on their cultural signification, or that treat them as reflections of users’ inner intensions, we analyse emoji as communication phenomena by exploring their relation to other textual actions in the production of text-talk. The emoji analysed here functioned as a laughter token, and performed specific interactional work related to laughter. We conclude that conversation analysis offers an important corrective to abstracted semiotic analysis and a useful resource for exploring the demonstrable meaning of emoji for interlocutors. However, we also emphasise the importance of capturing the process of composing messages, the challenges of dealing with the variety of forms that emoji take and their relation to gestural and other actions in face to face communication.

Approaching the architectural native: a graphical transcription method to capture sketching and gesture activity

As a research team without architectural training, we approached a total of 22 experienced architects and designers in order to investigate architectural design practice with an emphasis on human wayfinding. For the main study, eight informers developed circulation systems for a number of case studies. Our graphical transcription method allowed us to systematically investigate the sketching and gesture activity as is typical in design practice. Our analysis reveals the early design decisions of our informers and links them to priorities and the resulting design solutions. Regarding our method we can conclude that it is possible to gain insight into the concepts and work practices of a visual domain or profession, even for researchers untrained in that domain.

The effect of directive words on integrated comprehension of speech and iconic gestures for actions in young children

AbstractWe investigated the effect of age and directive words on children's ability to semantically integrate speech and iconic gesture. The participants were 132 children aged 4 to 6 years old. In the experiment, we presented messages of actions in 3 conditions (Verbal‐only, Gesture‐only, and Verbal‐Gesture) in a serial manner to assess multimodal gain as the ability and had participants select 1 answer corresponding to each message from among 4 options. Participants were divided according to whether the messages of various speech and gestures used directive words or not (e.g., “one is throwing like this” vs. “one is throwing”). The results showed that participants in the directive‐word‐present condition were more likely to select the correct answer than the participants in the directive‐word‐absent condition were. Our results suggest that directive words promote young children's ability to semantically integrate speech and gesture, and therefore are useful for promoting communication.Highlights We examined whether the presence or absence of directive words influence young children's comprehension of speech and iconic gestures to communicate actions. One hundred thirty‐two 4 to 6‐year‐old children were presented with eight messages using speech and gesture either with or without including directive words. The results showed that participants in the directive‐word‐present condition were more likely to select the correct answer than the participants in the directive‐word‐absent condition were. We concluded directive words promote young children's ability to semantically integrate speech and gesture, and therefore are useful for promoting communication.

Gestural Learning in Orchestra Conducting Through Self-observation

The Orchestra Conducting training has traditionally been based on the imitation of the gesture of experienced conductors. It has only recently begun to work on other methodological alternatives for the teaching of gesture in the orchestral conducting. In this research, gestural learning in this field has been approached by means of self-observation through video with 28 students from first course in Conducting at Higher Music Education Conservatoire of the Basque Country – Musikene. The results have shown that with this methodology the participants’ gestural activity has improved, especially in those competencies more difficult to acquire, such as facial expression, visual contact and the use of the left arm. Significant differences have also been revealed in the improvements produced over time and in the assessments of the different types of judges involved. Gender differences have not been significant, but it is worth mentioning that women in this study have scored lower (themselves and others) than men.

El aprendizaje de la gestualidad en Dirección de Orquesta mediante la auto-observación

The Orchestra Conducting training has traditionally been based on the imitation of the gesture of experienced conductors. It has only recently begun to work on other methodological alternatives for the teaching of gesture in the orchestral conducting. In this research, gestural learning in this field has been approached through self-observation through video with 28 students from first course in Conducting at Higher Music Education Conservatoire of the Basque Country — Musikene. The results have shown that with this methodology the participants’ gestural activity has improved, especially in those competencies more difficult to acquire, such as facial expression, visual contact and the use of the left arm. Significant differences have also been revealed in the improvements produced over time and in the assessments of the different types of judges involved. Gender differences have not been significant, but it is worth mentioning that women in this study have scored lower (themselves and others) than men.

Unsupervised Hierarchical Dynamic Parsing and Encoding for Action Recognition.

Generally, the evolution of an action is not uniform across the video, but exhibits quite complex rhythms and non-stationary dynamics. To model such non-uniform temporal dynamics, in this paper, we describe a novel hierarchical dynamic parsing and encoding method to capture both the locally smooth dynamics and globally drastic dynamic changes. It parses the dynamics of an action into different layers and encodes such multi-layer temporal information into a joint representation for action recognition. At the first layer, the action sequence is parsed in an unsupervised manner into several smooth-changing stages corresponding to different key poses or temporal structures by temporal clustering. The dynamics within each stage are encoded by mean-pooling or rank-pooling. At the second layer, the temporal information of the ordered dynamics extracted from the previous layer is encoded again by rank-pooling to form the overall representation. Extensive experiments on a gesture action data set (Chalearn Gesture) and three generic action data sets (Olympic Sports, Hollywood2, and UCF101) have demonstrated the effectiveness of the proposed method.

Graph theoretical analysis of functional network for comprehension of sign language

Signed languages are natural human languages using the visual-motor modality. Previous neuroimaging studies based on univariate activation analysis show that a widely overlapped cortical network is recruited regardless whether the sign language is comprehended (for signers) or not (for non-signers). Here we move beyond previous studies by examining whether the functional connectivity profiles and the underlying organizational structure of the overlapped neural network may differ between signers and non-signers when watching sign language. Using graph theoretical analysis (GTA) and fMRI, we compared the large-scale functional network organization in hearing signers with non-signers during the observation of sentences in Chinese Sign Language. We found that signed sentences elicited highly similar cortical activations in the two groups of participants, with slightly larger responses within the left frontal and left temporal gyrus in signers than in non-signers. Crucially, further GTA revealed substantial group differences in the topologies of this activation network. Globally, the network engaged by signers showed higher local efficiency (t(24)=2.379, p=0.026), small-worldness (t(24)=2.604, p=0.016) and modularity (t(24)=3.513, p=0.002), and exhibited different modular structures, compared to the network engaged by non-signers. Locally, the left ventral pars opercularis served as a network hub in the signer group but not in the non-signer group. These findings suggest that, despite overlap in cortical activation, the neural substrates underlying sign language comprehension are distinguishable at the network level from those for the processing of gestural action.

A Two-Tone Radar Sensor for Concurrent Detection of Absolute Distance and Relative Movement for Gesture Sensing

This letter presents a two-tone radar sensor system that can concurrently detect both absolute distance and relative movement. In contrast to the traditional two-tone radar, the proposed radar sensor can measure both distance and vibration. Moreover, it is more accurate in predicting the phase shift by leveraging the accuracy of nonlinear phase demodulation in the digital domain. Unlike the frequency-modulated continuous wave radar, the proposed radar system has simpler system architecture and does not require broadband signals. With a tone difference of 40MHz, the radar system can detect a distance up to 1.875m without ambiguity. Experiments show that the proposed radar system was able to not only capture the delaying movement of a Newton's cradle but to measure the distance with mm accuracy as well. It successfully detected the gesture motion of virtual slider, making it promising for sensing action gestures for human-computer interaction.

T004 Cognitive processes of motor behavior revealed by tDCS

Transcranial direct current stimulation (tDCS) is a non-invasive tool, which effectively modulates behavior, and related brain activity. When applied to the primary motor cortex (M1), tDCS affects motor function, enhancing or decreasing motor learning and action execution in both healthy participants and brain-damaged patients. Evidence in both humans and macaques has shown that multiple cortical areas beyond M1 are involved in controlling and guiding movement, including the posterior parietal cortex (PPC) and the premotor cortex (PMC). Recent evidence in healthy humans and brain-damaged showed that the modulation of cortical excitability of PPC and PMC can be used to improve action-related cognitive processes, highlighting interesting hemispheric asymmetries. Indeed, the anodal tDCS of the PPC of the left hemisphere can be used to selectively facilitate action planning and gesture recognition, while excitability shifts within the PM of the right hemisphere affects action monitoring and awareness. These results indicate that a left parietal system contributes to the activation of motor engrams (such as the pattern and sequence of movements needed to execute an action), while left premotor areas are mostly involved in the operation of the motor monitoring, namely detecting the mismatch between the actual motor output and intended action goal. The final relay is M1, which converts the motor program into the desired action. In conclusion, this evidence offers a functional and anatomical framework for understanding how selective neuromodulatory effects on different higher-level (action planning and monitoring) stages of motor processing can be induced by tDCS in order to optimize motor control and learning. This knowledge opens up novel perspectives in the neurorehabilitation of stroke patients with apraxic and motor awareness disorders, which represents a considerable burden for motor and cognitive rehabilitation.

A two-sided collaborative transparent display supporting workspace awareness

Transparent displays naturally support workspace awareness during face-to-face interactions. Viewers see another person’s actions through the display: gestures, gaze, body movements, and what one is manipulating on the display. Yet we can design even better collaborative transparent displays. First, collaborators on either side should be able to directly interact with workspace objects. Second, and more controversially, both sides should be capable of presenting different content. This affords: reversal of images/text in place (so that people on both sides see objects correctly); personal and private territories aligned atop each other; and GUI objects that provide different visuals for feedthrough vs. feedback. Third, the display should visually enhance the gestural actions of the person on the other side to better support workspace awareness. We show how our FacingBoard-2 design supports these collaborative requirements, and confirm via a controlled study that visually enhancing gestures is effective under a range of deteriorating transparency conditions.

A smart segmentation technique towards improved infrequent non-speech gestural activity recognition model

Infrequent Non-Speech Gestural Activities (IGAs) such as coughing, deglutition and yawning help identify fine-grained physiological symptoms and chronic psychological conditions which are not directly observable from traditional daily activities. We propose a new wearable smart earring which is capable of differentiating IGAs in daily environment with single integrated accelerometer sensor signal processing. Our prior framework, GeSmart, shows significant improvement in IGAs recognition based on smart earring which necessitates users to replace the earring batteries frequently due to its energy hungry requirement (high sampling frequency) towards fine-grained IGAs recognition. In this improved work, we propose a new segmentation technique along with GeSmart which takes the advantages of change-point detection algorithm to segment sensor data streams, feature extraction and classification thus any machine learning technique can perform significantly well in low sampling rate. We also implement a baseline traditional graphical model based gesture recognition techniques and compare their performances with our model in terms of accuracy, energy consumption and degradation of sampling rate scenarios. Experimental results based on real data traces demonstrate that our approach improves the performances significantly compared to previously proposed solutions. We also apply our segmentation technique on two benchmark datasets to prove the superiority of our technique in low sampling rate scenario.

A third-person perspective on co-speech action gestures in Parkinson's disease

A combination of impaired motor and cognitive function in Parkinson's disease (PD) can impact on language and communication, with patients exhibiting a particular difficulty processing action verbs. Co-speech gestures embody a link between action and language and contribute significantly to communication in healthy people. Here, we investigated how co-speech gestures depicting actions are affected in PD, in particular with respect to the visual perspective—or the viewpoint – they depict. Gestures are closely related to mental imagery and motor simulations, but people with PD may be impaired in the way they simulate actions from a first-person perspective and may compensate for this by relying more on third-person visual features. We analysed the action-depicting gestures produced by mild-moderate PD patients and age-matched controls on an action description task and examined the relationship between gesture viewpoint, action naming, and performance on an action observation task (weight judgement). Healthy controls produced the majority of their action gestures from a first-person perspective, whereas PD patients produced a greater proportion of gestures produced from a third-person perspective. We propose that this reflects a compensatory reliance on third-person visual features in the simulation of actions in PD. Performance was also impaired in action naming and weight judgement, although this was unrelated to gesture viewpoint. Our findings provide a more comprehensive understanding of how action-language impairments in PD impact on action communication, on the cognitive underpinnings of this impairment, as well as elucidating the role of action simulation in gesture production.

Gestural Performance of Electronic Music—A “NIME” Practice as Research

The practice of gestural electronic music performance provides a valid context for artistic or practice-based investigations in the field of ’NIME.’ To this end, the material and conceptual conditions for the development of performance pieces using gestural actions need to be explored. The use of digital musical instruments and concepts for the expressive performance with digital sounds leads to questions of perception—by the musician and by the audience—of movements and actions, the body, the instruments, and of their affordances. When considering this performance mode as a topic for investigation, it becomes evident that in order to be based on practice, research in this field needs a definition and differentiation that helps to identify the specific perspectives that are only made possible through application in an actual artistic practice.

Crossover learning of gestures in two ideomotor apraxia patients: A single case experimental design study

ABSTRACTCrossover learning may aid rehabilitation in patients with neurological disorders. Ideomotor apraxia (IMA) is a common sequela of left-brain damage that comprises a deficit in the ability to perform gestures to verbal commands or by imitation. This study elucidated whether crossover learning occurred in two post-stroke IMA patients without motor paralysis after gesture training approximately 2 months after stroke onset. We quantitatively analysed the therapeutic intervention history and investigated whether revised action occurred during gesture production. Treatment intervention was to examine how to influence improvement and generalisation of the ability to produce the gesture. This study used an alternating treatments single-subject design, and the intervention method was errorless learning. Results indicated crossover learning in both patients. Qualitative analysis indicated that revised action occurred during the gesture-production process in one patient and that there were two types of post-revised action gestures: correct and incorrect gestures. We also discovered that even when a comparably short time had elapsed since stroke onset, generalisation was difficult. Information transfer between the left and right hemispheres of the brain via commissural fibres is important in crossover learning. In conclusion, improvements in gesture-production skill should be made with reference to the left cerebral hemisphere disconnection hypothesis.

A Structure and Framework for Sign Language Interaction

Objective:The goal of this thesis is to design the interaction structure and framework of system to recognize sign language. Background: The sign language of meaningful individual gestures is combined to construct a sentence, so it is difficult to interpret and recognize the meaning of hand gesture for system, because of the sequence of continuous gestures. This being so, in order to interpret the meaning of individual gesture correctly, the interaction structure and framework are needed so that they can segment the indication of individual gesture. Method: We analyze 700 sign language words to structuralize the sign language gesture interaction. First of all, we analyze the transformational patterns of the hand gesture. Second, we analyze the movement of the transformational patterns of the hand gesture. Third, we analyze the type of other gestures except hands. Based on this, we design a framework for sign language interaction. Results: We elicited 8 patterns of hand gesture on the basis of the fact on whether the gesture has a change from starting point to ending point. And then, we analyzed the hand movement based on 3 elements: patterns of movement, direction, and whether hand movement is repeating or not. Moreover, we defined 11 movements of other gestures except hands and classified 8 types of interaction. The framework for sign language interaction, which was designed based on this mentioned above, applies to more than 700 individual gestures of the sign language, and can be classified as an individual gesture in spite of situation which has continuous gestures. Conclusion: This study has structuralized in 3 aspects defined to analyze the transformational patterns of the starting point and the ending point of hand shape, hand movement, and other gestures except hands for sign language interaction. Based on this, we designed the framework that can recognize the individual gestures and interpret the meaning more accurately, when meaningful individual gesture is input sequence of continuous gestures. Application: When we develop the system of sign language recognition, we can apply interaction framework to it. Structuralized gesture can be used for using database of sign language, inventing an automatic recognition system, and studying on the action gestures in other areas.