Gesture Video Research Articles

The influence of gestures and visuospatial ability during learning about movements with dynamic visualizations – An fNIRS study

There is increasing evidence that learning manual tasks from dynamic visualizations (e.g., origami folding) is facilitated when human hands are shown or gestures can be observed in the visualizations. This study examined whether observing and making gestures improves learning about non-human biological movements (i.e., fish locomotion) and whether gestures that correspond to the to-be-learned movement are superior to non-corresponding gestures. Moreover, learners’ visuospatial ability was assessed as a possible moderator. Regarding underlying neurophysiological processes, functional near-infrared spectroscopy (fNIRS) was used to investigate whether gestures activate the human mirror-neuron system (hMNS) and whether this activation mediates the facilitation of learning. During learning, participants viewed animations that were supplemented with either a self-gesturing instruction (yes/no) and/or a gesture video (corresponding/non-corresponding/no gesture) resulting in six conditions (2x3-between-subjects design). Results showed that higher-visuospatial-ability learners benefitted from learning with non-corresponding gestures, whereas those gestures were detrimental for lower-visuospatial-ability learners. Furthermore, activation of the inferior frontal cortex (part of hMNS) tended to predict better learning outcomes. Making gestures did not influence learning, but participants observing corresponding gestures showed higher inferior frontal cortex activation if they self-gestured than when they did not self-gesture. Implications of the results for the design of instructional materials are discussed.

How does hand gestures in videos impact social media engagement - Insights based on deep learning

With the fast improvement and development in deep learning and computer vision, the interaction between humans and computers is becoming immense. This research work aims at recognizing hand gestures performed in TEDx videos and analyse the relationship between gestures and user engagement. Here we propose a technique based on deep learning, Convolutional Neural Network (CNN), to recognize hand gestures from a video or image input. ResNeXt-101 model is used for the classification of hand gestures. Here we also used images from the Twenty Bn dataset for some gestures while we also collected gesture images from TEDx videos. Our experiments obtained high accuracy for gesture recognition on training as 95% to 99% and 94.35% for testing. Firstly a gesture is identified in each frame and then classified. We also created a count function that helped us count the number of times a gesture was performed and help us analyse these talks in a better way. Two experimental studies were carried out to analyse viewer engagement: one, the effect of suitable gestures on the viewer count, and secondly, suitable gestures on the sentiment of the viewer. Interesting results are observed that suitable gestures from talkers have an impact on increasing positive review and viewer count.

A hierarchical model for learning to understand head gesture videos

Head gesture videos recorded of a person bear rich information about the individual. Automatically understanding these videos can empower many useful human-centered applications in areas such as smart health, education, work safety and security. To understand a video’s content, low-level head gesture signals carried in the video that capture characteristics of both human postures and motions need to be translated into high-level semantic labels. To meet this aim, we propose a hierarchical model for learning to understand head gesture videos. Given a head gesture video of an arbitrary length, the model first segments the full-length video into multiple short clips for clip-based feature extraction. Multiple base feature extraction procedures are then independently tuned via a set of peripheral learning tasks without consuming any labels of the goal task. These independently derived base features are subsequently aggregated through a multi-task learning framework, coupled with a feature dimensionality reduction module, to optimally learn to accomplish the end video understanding task in an weakly supervised manner, utilizing the limited amount of video labels available of the goal task. Experimental results show that the hierarchical model is superior to multiple state-of-the-art peer methods in tackling versatile video understanding tasks.

Deep Multi-Stage Approach For Emotional Body Gesture Recognition In Job Interview

Abstract Affective computing is a key research topic in artificial intelligence which is applied to psychology and machines. It consists of the estimation and measurement of human emotions. A person’s body language is one of the most significant sources of information during job interview, and it reflects a deep psychological state that is often missing from other data sources. In our work, we combine two tasks of pose estimation and emotion classification for emotional body gesture recognition to propose a deep multi-stage architecture that is able to deal with both tasks. Our deep pose decoding method detects and tracks the candidate’s skeleton in a video using a combination of depthwise convolutional network and detection-based method for 2D pose reconstruction. Moreover, we propose a representation technique based on the superposition of skeletons to generate for each video sequence a single image synthesizing the different poses of the subject. We call this image: ‘history pose image’, and it is used as input to the convolutional neural network model based on the Visual Geometry Group architecture. We demonstrate the effectiveness of our method in comparison with other methods in the state of the art on the standard Common Object in Context keypoint dataset and Face and Body gesture video database.

Semantic Deep Learning to Translate Dynamic Sign Language

Dynamic Sign Language Recognition aims to recognize hand gestures of any person. Dynamic Sign Language Recognition systems have challenges in recognizing the semantic of hand gestures. These challenges come from the personal differences in hand signs from one person to another. Real-life video gesture frames couldn’t be treated as frame-level as a static sign. This research proposes a semantic translation system for dynamic hand gestures using deep learning and ontology. We used the proposed MSLO (Multi Sign Language Ontology) in the semantic translation step. Also, any user can retrain the system to be a personal one. We used Three-dimensional Convolutional Neural Networks followed by Convolutional long short-term memory to improve the recognition accuracy in Dynamic sign language recognition. We applied the proposed system on three dynamic gesture datasets from color videos. The recognition accuracy average was 97.4%. We did all the training and testing processes on the Graphics Processing Unit with the support of Google Colab. Using "Google Colab" in the training process decreases the average run time by about 87.9%. In addition to adding semantic in dynamic sign language translation, the proposed system achieves good results compared to some dynamic sign language recognition systems.

An Indian Corpus of Gestures for Nouns and Verbs: A Preliminary Study

The study aimed to develop a corpus of comprehensive gestures for a set of nouns and verbs in the Indian context. The study was conducted in the following three phases. Phase 1: Gestural nouns and verbs were selected from the existing word lists. Phase 2: A set of 221 nouns and 122 verbs were enacted and recorded by a professional actor and a cameraman respectively. Phase 3: Two groups of participants i.e., 15 young (18-39 years) and 15 old (40-69 years) neuro-typical adults participated in the two tasks namely (a) naming and (b) rating tasks. The analyses of the responses revealed maximum uniformity of correct responses between the participants at 85% agreement level for both noun and verb gestures. Thus, a set of 86 noun and 60 verb gesture videos entered the final gesture corpus which satisfied the criteria. Further, the varied responses among the participants in interpreting the gestures and type of gestures have been discussed and documented. The corpus developed can serve as stimulus in various gesture studies. Although the corpus comes with few limitations, the data provided would help the researchers to carefully select the gesture videos based on their purpose of the study.

A Crowdsourcing-Based Framework for the Development and Validation of Machine Readable Parallel Corpus for Sign Languages

Sign languages are used by the deaf and mute community of the world. These are gesture based languages where the subjects use hands and facial expressions to perform different gestures. There are hundreds of different sign languages in the world. Furthermore, like natural languages, there exist different dialects for many sign languages. In order to facilitate the deaf community several different repositories of video gestures are available for many sign languages of the world. These video based repositories do not support the development of an automated language translation systems. This research aims to investigate the idea of engaging the deaf community for the development and validation of a parallel corpus for a sign language and its dialects. As a principal contribution, this research presents a framework for building a parallel corpus for sign languages by harnessing the powers of crowdsourcing with editorial manager, thus it engages a diversified set of stakeholders for building and validating a repository in a quality controlled manner. It further presents processes to develop a word-level parallel corpus for different dialects of a sign language; and a process to develop sentence-level translation corpus comprising of source and translated sentences. The proposed framework has been successfully implemented and involved different stakeholders to build corpus. As a result, a word-level parallel corpus comprising of the gestures of almost 700 words of Pakistan Sign Language (PSL) has been developed. While, a sentence-level translation corpus comprising of more than 8000 sentences for different tenses has also been developed for PSL. This sentence-level corpus can be used in developing and evaluating machine translation models for natural to sign language translation and vice-versa. While the machine-readable word level parallel corpus will help in generating avatar based videos for the translated sentences in different dialects of a sign language.

Video Summarization Based on Mutual Information and Entropy Sliding Window Method.

This paper proposes a video summarization algorithm called the Mutual Information and Entropy based adaptive Sliding Window (MIESW) method, which is specifically for the static summary of gesture videos. Considering that gesture videos usually have uncertain transition postures and unclear movement boundaries or inexplicable frames, we propose a three-step method where the first step involves browsing a video, the second step applies the MIESW method to select candidate key frames, and the third step removes most redundant key frames. In detail, the first step is to convert the video into a sequence of frames and adjust the size of the frames. In the second step, a key frame extraction algorithm named MIESW is executed. The inter-frame mutual information value is used as a metric to adaptively adjust the size of the sliding window to group similar content of the video. Then, based on the entropy value of the frame and the average mutual information value of the frame group, the threshold method is applied to optimize the grouping, and the key frames are extracted. In the third step, speeded up robust features (SURF) analysis is performed to eliminate redundant frames in these candidate key frames. The calculation of Precision, Recall, and F are optimized from the perspective of practicality and feasibility. Experiments demonstrate that key frames extracted using our method provide high-quality video summaries and basically cover the main content of the gesture video.

Dynamic Gesture Recognition Based on iCPM and RNN

Aim to solve the problem that continuous complex actions are difficult to be recognized by computer vision technology in actual production, this paper collects gesture video from single-and double-viewpoint, constructs gesture joint point coordinate recognition network model by improved Convolutional Pose Machine(iCPM) model, obtains gesture Gauss heat map and joint point coordinates in each frame of video, and then inputs joint point coordinates with time series into gesture sequence recognition network. Finally, the gesture sequence is obtained. The experimental results show that the recognition accuracy of gesture recognition model based on double-viewpoint is 4.18% ∼ 6.92% higher than that of the single-viewpoint gesture recognition model, which verifies that the gesture recognition model based on dual-viewpoint has better recognition effect. In addition, the video samples of Tin-Lead Soldering including continuous and complex actions verify that the model has ideal recognition ability for dynamic gesture in actual production process.

Feature Fusion Based Hand Gesture Recognition Method for Automotive Interfaces

Hand gesture recognition on the depth videos is a promising approach for automotive interfaces because it is less sensitive to light variation and more accurate than other traditional methods. However, video gestures recognition is still a challenging task since lots of interferences are induced by the uncorrelated gesture factors. Considering that if the displays are more relevant, the results will more accurate, so ResNext, a kind of compact and efficient neural network, is firstly used as feature extractor, then an improved weighted frame unification method is adopted to obtain the key frame samples, finally the Discriminant correlation analysis (DCA) is employed to fuse features for static data and dynamic data after conducting Feature embedding branch (FEB) on static data. The public dataset named Depth based gesture recognition database (DGRD) is used in this paper, but the dataset is a little small and the class distribution is largely imbalance, and we find the performance of ResNext degrades badly in the condition of imbalance problem although it achieves excellent result at sufficient training data. In order to conquer the disadvantages of limited dataset, a special loss function scheme combining the softmax loss and dice loss is proposed. Evaluation of the algorithm performances in comparison with other state-of-the-art methods indicates that the proposed method is more practical for gesture recognition and may be widely adopted by automotive interfaces.

Fast and robust key frame extraction method for gesture video based on high-level feature representation

In gesture video, the inner-frame difference is too subtle to be projected via low-level features, and the gesture frames, expressing semantic information, are distributed only among the tiny part of the whole video frame. This paper introduces a fast and robust key frame extraction method for gesture video, founded upon high-level feature representation to extract the gesture key frame precisely without affecting the semantic information. Firstly, a gesture video segmentation model is designed by employing SSD, which classify gesture video into the semantic scene and the static scene. And then, the 2D-DWT-based perceptual hash algorithm is studied to extract candidate static key frames. Afterward, the multi-channel gradient magnitude frequency histogram (HGMF-MC) based on improved VGG16 is developed as a new image descriptor. Finally, a key frame extraction mechanism based on HGMF-MC is proposed to generate gesture video summary of two scenes, respectively. Experiments consistently show the superiority of the proposed method on Chinese sign language, Cambridge, ChaLearn and CVRR-Hands gesture datasets. The results demonstrate that the method proposed is effective, which improves the video compression ratio and outperforms the state-of-the-art methods.

Retracted: Gesture recognition algorithm based on multi‐scale feature fusion in RGB‐D images

With the rapid development of sensor technology and artificial intelligence, the video gesture recognition technology under the background of big data makes human–computer interaction more natural and flexible, bringing the richer interactive experience to teaching, on-board control, electronic games etc. To perform robust recognition under the conditions of illumination change, background clutter, rapid movement, and partial occlusion, an algorithm based on multi-level feature fusion of two-stream convolutional neural network is proposed, which includes three main steps. Firstly, the Kinect sensor obtains red–green–blue-depth (RGB-D) images to establish a gesture database. At the same time, data enhancement is performed on the training set and test set. Then, a model of multi-level feature fusion of a two-stream convolutional neural network is established and trained. Experiments show that the proposed network model can robustly track and recognise gestures under complex backgrounds (such as similar complexion, illumination changes, and occlusion), and compared with the single-channel model, the average detection accuracy is improved by 1.08%, and mean average precision is improved by 3.56%.

A New Dataset for Facial Motion Analysis in Individuals With Neurological Disorders.

We present the first public dataset with videos of oro-facial gestures performed by individuals with oro-facial impairment due to neurological disorders, such as amyotrophic lateral sclerosis (ALS) and stroke. Perceptual clinical scores from trained clinicians are provided as metadata. Manual annotation of facial landmarks is also provided for a subset of over 3300 frames. Through extensive experiments with multiple facial landmark detection algorithms, including state-of-the-art convolutional neural network (CNN) models, we demonstrated the presence of bias in the landmark localization accuracy of pre-trained face alignment approaches in our participant groups. The pre-trained models produced higher errors in the two clinical groups compared to age-matched healthy control subjects. We also investigated how this bias changes when the existing models are fine-tuned using data from the target population. The release of this dataset aims to propel the development of face alignment algorithms robust to the presence of oro-facial impairment, support the automatic analysis and recognition of oro-facial gestures, enhance the automatic identification of neurological diseases, as well as the estimation of disease severity from videos and images.

Gesture's body orientation modulates the N400 for visual sentences primed by gestures.

Body orientation of gesture entails social‐communicative intention, and may thus influence how gestures are perceived and comprehended together with auditory speech during face‐to‐face communication. To date, despite the emergence of neuroscientific literature on the role of body orientation on hand action perception, limited studies have directly investigated the role of body orientation in the interaction between gesture and language. To address this research question, we carried out an electroencephalography (EEG) experiment presenting to participants (n = 21) videos of frontal and lateral communicative hand gestures of 5 s (e.g., raising a hand), followed by visually presented sentences that are either congruent or incongruent with the gesture (e.g., “the mountain is high/low…”). Participants underwent a semantic probe task, judging whether a target word is related or unrelated to the gesture‐sentence event. EEG results suggest that, during the perception phase of handgestures, while both frontal and lateral gestures elicited a power decrease in both the alpha (8–12 Hz) and the beta (16–24 Hz) bands, lateral versus frontal gestures elicited reduced power decrease in the beta band, source‐located to the medial prefrontal cortex. For sentence comprehension, at the critical word whose meaning is congruent/incongruent with the gesture prime, frontal gestures elicited an N400 effect for gesture‐sentence incongruency. More importantly, this incongruency effect was significantly reduced for lateral gestures. These findings suggest that body orientation plays an important role in gesture perception, and that its inferred social‐communicative intention may influence gesture‐language interaction at semantic level.

Convert Gestures of Arabic Words into Voice

Gestures are one of the best ways of communication between dumbs and other people using the expression of signs language. In this paper, we suggest an algorithm for recognizing hand gestures of Arabic words (اتمنى لك حیاة سعیدة-اقتباس) to by using dumb (through signs) and convert the sings into voice corresponding to sings words. The proposed algorithm for Convert Gestures of Arabic Words into Voice, record video of gesture (of the dumb person) then convert the video into frames (images), preprocessing for the resulted image must done by remove the noise, resize the images and increase the contrast, then calculate the distance to clustering the words by using (C4.5, k-mean, k- medoid and artificial neural network), calculate the distance (or features) by using Euclidean distance and slope where, there are eighteen features (eight features from Euclidean distance, eight features from slop, Area, and perimeter). The results in the training stage were (C4.5 gave 100%, k-mean gave 95.2% k-medoid gave 91.9% and ANN gave 91.27%). While in the testing stage we used three classifiers (Euclidian Distance, Modify of the Standardize Euclidian Distance and Correlation) and the results show that (Euclidian Distance gave 94.4%, Modify of the Standardize Euclidian Distance gave 100% and Correlation gave 94.4%) We create our database (three videos with 250 frames) for training and one video for testing.

Human Mirror Neuron System Based Alarms in the Cockpit: A Neuroergonomic Evaluation.

Controlled Flight Into Terrain (CFIT) events still remain among the deadliest accidents in aviation. When facing the possible occurrence of such an event, pilots have to immediately react to the ground proximity alarm ("Pull Up" alarm) in order to avoid the impending collision. However, the pilots' reaction to this alarm is not always optimal. This may be at least partly due to the low visual saliency of the current alarm and the deleterious effects of stress that alleviate the pilot's reactions. In the present study, two experiments (in a laboratory and in a flight simulator) were conducted to (1)investigate whether hand gesture videos (a hand pulling back the sidestick) can trigger brainwave frequencies related to the mirror neuron system; (2)determine whether enhancing the visual characteristics of the "Pull Up" alarm could improve pilots' response times. Electrophysiological results suggest that hand gesture videos attracted more participants' attention (greater alpha desynchronization in the parieto-occipital area) and possibly triggered greater activity of the mirror neuron system (greater mu and beta desynchronizations at central electrodes). Results obtained in theflight simulator revealed that enhancing the visual characteristics of the original "Pull Up" alarm improved the pilots' reaction times. However, no significant difference in reaction times between an enlarged "Pull Up" inscription and the hand gesture video was found. Further work is needed to determine whether mirror neuron system based alarms could bring benefits for flight safety, in particular, these alarms should be assessed during a high stress context.

Hand Gesture Recognition from RGB-D Data using 2D and 3D Convolutional Neural Networks: a comparative study

Despite considerable enhances in recognizing hand gestures from still images, there are still many challenges in the classification of hand gestures in videos. The latter comes with more challenges, including higher computational complexity and arduous task of representing temporal features. Hand movement dynamics, represented by temporal features, have to be extracted by analyzing the total frames of a video. So far, both 2D and 3D convolutional neural networks have been used to manipulate the temporal dynamics of the video frames. 3D CNNs can extract the changes in the consecutive frames and tend to be more suitable for the video classification task, however, they usually need more time. On the other hand, by using techniques like tiling it is possible to aggregate all the frames in a single matrix and preserve the temporal and spatial features. This way, using 2D CNNs, which are inherently simpler than 3D CNNs can be used to classify the video instances. In this paper, we compared the application of 2D and 3D CNNs for representing temporal features and classifying hand gesture sequences. Additionally, providing a two-stage two-stream architecture, we efficiently combined color and depth modalities and 2D and 3D CNN predictions. The effect of different types of augmentation techniques is also investigated. Our results confirm that appropriate usage of 2D CNNs outperforms a 3D CNN implementation in this task.

Electrophysiological indicators of gesture perception.

Electroencephalography (EEG) activity in the mu frequency band (8-13Hz) is suppressed during both gesture performance and observation. However, it is not clear if or how particular characteristics within the kinematic execution of gestures map onto dynamic changes in mu activity. Mapping the time course of gesture kinematics onto that of mu activity could help understand which aspects of gestures capture attention and aid in the classification of communicative intent. In this work, we test whether the timing of inflection points within gesture kinematics predicts the occurrence of oscillatory mu activity during passive gesture observation. The timing for salient features of performed gestures in video stimuli was determined by isolating inflection points in the hands' motion trajectories. Participants passively viewed the gesture videos while continuous EEG data was collected. We used wavelet analysis to extract mu oscillations at 11Hz and at central electrodes and occipital electrodes. We used linear regression to test for associations between the timing of inflection points in motion trajectories and mu oscillations that generalized across gesture stimuli. Separately, we also tested whether inflection point occurrences evoked mu/alpha responses that generalized across participants. Across all gestures and inflection points, and pooled across participants, peaks in 11Hz EEG waveforms were detected 465 and 535ms after inflection points at occipital and central electrodes, respectively. A regression model showed that inflection points in the motion trajectories strongly predicted subsequent mu oscillations ([Formula: see text]<0.01); effects were weaker and non-significant for low (17Hz) and high (21Hz) beta activity. When segmented by inflection point occurrence rather than stimulus onset and testing participants as a random effect, inflection points evoked mu and beta activity from 308 to 364ms at central electrodes, and broad activity from 226 to 800ms at occipital electrodes. The results suggest that inflection points in gesture trajectories elicit coordinated activity in the visual and motor cortices, with prominent activity in the mu/alpha frequency band and extending into the beta frequency band. The time course of activity indicates that visual processing drives subsequent activity in the motor cortex during gesture processing, with a lag of approximately 80ms.

RETRACTED: Learning spatiotemporal features with 3D DenseNet and attention for gesture recognition

Gesture recognition aims at understanding dynamic gestures of the human body and is one of the most important ways of human–computer interaction; to extract more effective spatiotemporal features in gesture videos for more accurate gesture classification, a novel feature extractor network, spatiotemporal attention 3D DenseNet is proposed in this study. We extend DenseNet with 3D kernels and Refined Temporal Transition Layer based on Temporal Transition Layer, and we also explore attention mechanism in 3D ConvNets. We embed the Refined Temporal Transition Layer and attention mechanism in DenseNet3D, named the proposed network “spatiotemporal attention 3D DenseNet.” Our experiments show that our Refined Temporal Transition Layer performs better than Temporal Transition Layer and the proposed spatiotemporal attention 3D DenseNet in each modality outperforms the current state-of-the-art methods on the ChaLearn LAP Large-Scale Isolated gesture dataset. The code and pretrained model are released in https://github.com/dzf19927/STA3D .

Deep Gesture Video Generation With Learning on Regions of Interest

Generating videos with semantic meaning, such as gestures in sign language, is a challenging problem. The model should not only learn to generate videos with realistic appearance, but also take notice of crucial details in frames to convey precise information. In this paper, we focus on the problem of generating long-term gesture videos containing precise and complete semantic meanings. We develop a novel architecture to learn the temporal and spatial transforms in regions of interest, i.e., gesticulating hands or face in our case. We adopt a hierarchical approach for generating gesture videos, by first making predictions on future pose configurations, and then using the encoder-decoder architecture to synthesize future frames based on the predicted pose structures. We develop the scheme of action progress in our architecture to represent how far the action has been performed during its expected execution, and to instruct our model to synthesize actions with various paces. Our approach is evaluated on two challenging datasets for the task of gesture video generation. Experimental results show that our method can produce gesture videos with more realistic appearance and precise meaning than the state-of-the-art video generation approaches.