Gesture Sequences Research Articles

Multi-modal Anaphora and Broadcasting of Information by Gestural Post-holds

This paper deals with three interrelated topics, linguistic anaphora, multi-modal anaphora and the top-down broadcasting of information using gestural post-holds in multimodal dialogue. Initially, a new solution for definite, pronominal and pro-adverbial anaphora is given based on the idea that an existentially quantified general term may output a definite reference. This approach is extended to multimodal anaphora, where part or all of an anaphor’s meaning is contributed by some sequence of iconic or deictic gestures. Anaphora exploit the semantic potential of their antecedents, they work, as tradition has it, “bottom-up”. An inverse relation, more general than cataphora, and investigated here for the first time, is “broadcasting”, where information is freely distributed top down and input to receiving sites (ports). Anaphora are modelled with the same top-down mechanism and the same applies for coherence relations in dialogue which generally show an anaphora-like behaviour. “Broadcasting” can be used in the context of anaphors, for example, to provide their gestural meaning parts but also for a verb’s multi-modal arguments for referring to a location, a direction or an area. As to multi-modal data, broadcasting is shown to be frequently tied up with gestural post-holds, the holding of a gesture’s stroke information independently of semantically alignable speech. This leads to considering post-holds from a new perspective, stressing their speech-independent function and their relevance for indicating topic-continuity. We show that multi-modal anaphora and especially broadcasting cross single contributions and turns. The data which let us develop these perspectives come from the SaGA (Speech and Gesture Alignment) corpus, a set of route-description dialogues generated in a VR-setting incorporating marker-based eye-tracking facilities. The calculus used to model the anaphora and broadcasting dynamics is the concurrent λΨ-calculus, a recently developed two-tiered machinery using a Ψ-calculus for input-output, data transport and broadcasting. The data transported are in a typed λ-calculus format incorporating Neo-Davidsonian representations; these data can be linguistic, gestural only or multi-modal. Multi-modal informational chunks are modelled as communicating agents sending and receiving information via input-output-channels. They are introduced incrementally on an empirically motivated construction or gesture-plus-construction or gesture only basis. The λΨ-calculus is also used for the multi-modal fusion component unifying gestural and linguistic information; hence, the paper is also a contribution to multi-modal fusion of linguistic and gestural input. Finally, it is shown how the presented algorithm can capture multi-modal coherence relations or a multi-modal anaphora resolution based on PTT ideas.

Empowering sign language communication: Integrating sentiment and semantics for facial expression synthesis

Translating written sentences from oral languages to a sequence of manual and non-manual gestures plays a crucial role in building a more inclusive society for deaf and hard-of-hearing people. Facial expressions (non-manual), in particular, are responsible for encoding the grammar of the sentence to be spoken, applying punctuation, pronouns, or emphasizing signs. These non-manual gestures are closely related to the semantics of the sentence being spoken and also to the utterance of the speaker’s emotions. However, most Sign Language Production (SLP) approaches are centered on synthesizing manual gestures and do not focus on modeling the speaker’s expression. This paper introduces a new method focused in synthesizing facial expressions for sign language. Our goal is to improve sign language production by integrating sentiment information in facial expression generation. The approach leverages a sentence’s sentiment and semantic features to sample from a meaningful representation space, integrating the bias of the non-manual components into the sign language production process. To evaluate our method, we extend the Fréchet gesture distance (FGD) and propose a new metric called Fréchet Expression Distance (FED) and apply an extensive set of metrics to assess the quality of specific regions of the face. The experimental results showed that our method achieved state of the art, being superior to the competitors on How2Sign and PHOENIX14T datasets. Moreover, our architecture is based on a carefully designed graph pyramid that makes it simpler, easier to train, and capable of leveraging emotions to produce facial expressions. Our code and pretrained models will be available at: https://github.com/verlab/empowering-sign-language.

Computer Interactive Gesture Recognition Model Based on Improved YOLOv5 Algorithm

The expansion of virtual and augmented reality, intelligent assistance technology, and other fields has led to an increased demand for human–computer interaction methods that are more natural and intuitive. Gesture recognition has become an important research direction. Traditional gesture recognition methods are mainly based on image processing and pattern recognition techniques. However, there are certain challenges to its accuracy and robustness in complex backgrounds. In addition, the temporal correlation and spatial information in gesture sequences have not been fully utilized, which limits the performance of gesture recognition systems. In response to the above issues, this study first utilizes the Ghost module for feature extraction based on the You Only Look Once version 5 (YOLOv5) algorithm. Then drawing inspiration from the idea of densely connected networks, feature map stitching is carried out, and a human–machine interactive gesture recognition algorithm is designed by combining it with a hybrid attention mechanism. The experimental results showed that the average accuracy of the algorithm tended to converge after 160 iterations, and the final MAP value converged to 92.19%. Compared to the standard YOLOv5 algorithm, its iteration speed had been improved by 12.5%, and the MAP value had been improved by 4.63%. The designed human–computer interaction gesture recognition algorithm has higher accuracy and smaller error, and has certain application potential in the field of machine vision.

Gloss-driven Conditional Diffusion Models for Sign Language Production

Sign Language Production (SLP) aims to convert text or audio sentences into sign language videos corresponding to their semantics, which is challenging due to the diversity and complexity of sign languages, and cross-modal semantic mapping issues. In this work, we propose a Gloss-driven Conditional Diffusion Model (GCDM) for SLP. The core of the GCDM is a diffusion model architecture, in which the sign gloss sequence is encoded by a Transformer-based encoder and input into the diffusion model as a semantic prior condition. In the process of sign pose generation, the textual semantic priors carried in the encoded gloss features are integrated into the embedded Gaussian noise via cross-attention. Subsequently, the model converts the fused features into sign language pose sequences through T-round denoising steps. During the training process, the model uses the ground-truth labels of sign poses as the starting point, generates Gaussian noise through T rounds of noise, and then performs T rounds of denoising to approximate the real sign language gestures. The entire process is constrained by the MAE loss function to ensure that the generated sign language gestures are as close as possible to the real labels. In the inference phase, the model directly randomly samples a set of Gaussian noise, generates multiple sign language gesture sequence hypotheses under the guidance of the gloss sequence, and outputs a high-confidence sign language gesture video by averaging multiple hypotheses. Experimental results on the Phoenix2014T dataset show that the proposed GCDM method achieves competitiveness in both quantitative performance and qualitative visualization.

Research on wearable sensor gesture recognition based on CNN_GRU and attention mechanism

Gesture recognition based on wearable sensors has received extensive attention in recent years. This paper proposes a gesture recognition model (CGR_ATT) based on Convolutional Neural Network (CNN) and Gated Recurrent Unit (GRU) fused attention mechanism to improve accuracy rate of wearable sensors. First, CNN serves as a feature extractor, learning features automatically from sensor data by performing multiple layers of convolution and pooling operations, capturing spatial features of gestures. Furthermore, a temporal modeling unit GRU is introduced to capture the temporal dynamics in gesture sequences. By controlling the information flow through gate mechanisms, it effectively handles the temporal relationships in sensor data. Finally, an attention mechanism is introduced to assign different weights to the hidden state of the GRU. By calculating the attention weights for each time period, the model automatically selects key time periods related to gesture movements. The GR-dataset proposed in this paper involves 910 sets of training parameters. The model achieves an ultimate accuracy of 97.57% . In compare with CLA-net, CLT-net, CGR, GRU, LSTM and CNN, the experimental results demonstrate that the proposed method has superior accuracy.

An Angle-Oriented Approach to Transferring Speech to Gesture for Highly Anthropomorphized Embodied Conversational Agents

Realistic co-speech gestures are important to anthropomorphize ECAs, as nonverbal behavior improves expressiveness of their speech greatly. However, the existing approaches to generating co-speech gestures with sufficient details (including fingers, etc.) in 3D scenarios are indeed rare. Additionally, they hardly address the problem of abnormal gestures, temporal–spatial coherence and diversity of gesture sequences comprehensively. To handle abnormal gesture issues, we put forward an angle conversion method to remove body part length from the original in-the-wild video dataset via transferring coordinates of human upper body key points into relative deflection angles and pitch angles. We also propose a neural network called HARP with encoder–decoder architecture to transfer MFCC featured speech audio into aforementioned angles on the basis of CNN and LSTM. The angles then can be rendered as corresponding co-speech gestures. Compared with the other latest approaches, the co-speech gestures generated by HARP are proved to be almost as good as the real person, i.e., they have strong temporal–spatial coherence, diversity, persuasiveness and credibility. Our approach puts finer control on co-speech gestures than most of the existing works by handling all key points of the human upper body. It is more feasible for industrial application, since HARP can be adaptive to any human upper body model. All related code and evidence videos of HARP can be accessed at https://github.com/drrobincroft/HARP .

Gesture combinations during collaborative decision-making at wall displays

Abstract This paper describes an empirical user study with 24 participants during collaborative decision-making at large wall displays. The main objective of the user study is to analyze combinations of mid-air pointing gestures with other gestures or gaze. Particularly, we investigate gesture sequences (having pointing gestures as an initiator gesture) and gaze-pointing gesture misalignments. Our results show that most pointing gestures are part of gesture sequences and more precise gestures lead to touch gestures on the wall display, likely because they are associated with precise concepts. Regarding combinations of pointing gestures and gaze, misalignments often happen when users touch the display to make a change and want to observe the effect of that change on another display. The analyses conducted as part of this study clarify which natural awareness cues are more frequent in face-to-face collaboration, so that appropriate choices can be made regarding the transmission of equivalent cues to a remote location.

Speech-Driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference.

Speech-driven gesture generation is an emerging field within virtual human creation. However, a significant challenge lies in accurately determining and processing the multitude of input features (such as acoustic, semantic, emotional, personality, and even subtle unknown features). Traditional approaches, reliant on various explicit feature inputs and complex multimodal processing, constrain the expressiveness of resulting gestures and limit their applicability. To address these challenges, we present Persona-Gestor, a novel end-to-end generative model designed to generate highly personalized 3D full-body gestures solely relying on raw speech audio. The model combines a fuzzy feature extractor and a non-autoregressive Adaptive Layer Normalization (AdaLN) transformer diffusion architecture (DiTs-based). The fuzzy feature extractor harnesses a fuzzy inference strategy that automatically infers implicit, continuous fuzzy features. These fuzzy features, represented as a unified latent feature, are fed into the AdaLN transformer. The AdaLN transformer introduces a conditional mechanism that applies a uniform function across all tokens, thereby effectively modeling the correlation between the fuzzy features and the gesture sequence. This module ensures a high level of gesture-speech synchronization while preserving naturalness. Finally, we employ the diffusion model to train and infer various gestures. Extensive subjective and objective evaluations on the Trinity, ZEGGS, and BEAT datasets confirm our model's superior performance to the current state-of-the-art approaches. Persona-Gestor improves the system's usability and generalization capabilities, setting a new benchmark in speech-driven gesture synthesis and broadening the horizon for virtual human technology.

Gesture Recognition Technology of VR Piano Playing Teaching Game based on Hidden Markov Model

Virtual Reality (VR) is a kind of simulation environment generated by computer simulation. It is a kind of isolation of users from the physical environment, and users are fully engaged in the simulation environment. In music teaching, especially in the learning of piano performance, VR technology can simulate the real playing experience without physical piano, so that learners can learn and practice piano anywhere. At present, the traditional way of learning piano often needs the guidance of professional piano teachers, and learners must study and practice beside the piano, which undoubtedly brings limitations to the time and space of learners. VR technology provides a more convenient way for learners to learn and practice without a physical piano, making learning more flexible and free. Firstly, a piano playing teaching game platform based on VR technology is constructed to provide a virtual piano playing scene. Then, by collecting a large number of piano players' gesture data, they are accurately marked. Finally, the Hidden Markov Model (HMM) is used to train and model the gesture data, and the key features of the gesture sequence are extracted to realize the recognition and real-time feedback of the piano player's gesture. According to the findings, the recognition accuracy of the Principal Component Analysis-HMM (PCA-HMM) without playing correction is very low after playing correction. Compared to the logarithmic likelihood ratio test, the recognition rate of the quadratic statistical model is only 15%. The results show that this method has good iterative convergence and convergence performance, which can effectively avoid falling into local optima. Compared with the traditional hidden Markov algorithm, it reduces 96.05% and 89.48% respectively. Compared with the fuzzy hidden Markov algorithm, it reduces 90.20% and 73.35% respectively. The average time and standard deviation for real-time positioning and map construction are 0.251s and 0.152s, respectively. The fitness of HMM based on fuzzy Particle Swarm Optimization (PSO) is 6.8% and 2.6% higher than that of the hidden Markov algorithm and the fuzzy hidden Markov algorithm, respectively. At the same time, in the comparison between the research model and other algorithm models, the accuracy and other parameter performance of the research model are better, with the highest error value of only 3.5, which is 5.9 lower than the lowest Convolutional Neural Network (CNN) algorithm model. It can be seen that the error value of the model used in the study is lower and the performance is better. Learners can receive real-time guidance and a more immersive playing experience through this technology. This provides useful reference for further exploring the combination of music teaching and VR technology

Deceptive indicators

Correctly identifying when an individual is lying based on their nonverbal communication has been a central aim of researchers and practitioners. Traditionally, research has focused on individual ‘tells’ or indicators of deception. More recently, researchers have focused on clusters and sequences of nonverbal gestures; however, these methods can lead to complex outputs that are difficult to interpret. The current research uses a newly developed temporal method, Indicator Waves, to analyse the concurrent and sequential nonverbal communication of individuals caught verbally lying on tape. The analyses show the complexity of nonverbal behaviours in simplified wave diagrams and the importance of using linguistics to code nonverbal deceptive behaviours accurately. Statements made by individuals were analysed using Statement Analysis and nonverbal gestures related to lips, head, eyes and upper body were shown to change between deceptive and truthful individuals, both in the lead-up to lying and post-lying. This research is the first to use Indicator Waves in deception detection and provides a new simplified approach for understanding complex patterns of nonverbal communication in lying.

Cross-Domain Gesture Sequence Recognition for Two-Player Exergames using COTS mmWave Radar

Wireless-based gesture recognition provides an effective input method for exergames. However, previous works in wireless-based gesture recognition systems mainly recognize one primary user's gestures. In the multi-player scenario, the mutual interference between users makes it difficult to predict multiple players' gestures individually. To address this challenge, we propose a flexible FMCW-radar-based system, RFDual, which enables real-time cross-domain gesture sequence recognition for two players. To eliminate the mutual interference between users, we extract a new feature type, biased range-velocity spectrum (BRVS), which only depends on a target user. We then propose customized preprocessing methods (cropping and stationary component removal) to produce environment-independent and position-independent inputs. To enhance RFDual's resistance to unseen users and articulating speeds, we design effective data augmentation methods, sequence concatenating, and randomizing. RFDual is evaluated with a dataset containing only unseen gesture sequences and achieves a gesture error rate of 1.41%. Extensive experimental results show the impressive robustness of RFDual for data in new domains, including new users, articulating speeds, positions, and environments. These results demonstrate the great potential of RFDual in practical applications like two-player exergames and gesture/activity recognition for drivers and passengers in the cab.

Music conditioned 2D hand gesture dance generation with HGS

AbstractIn recent years, the short video industry is booming. However, there are still many difficulties in the action generation of virtual characters. We observed that on the short video social platform, “hand gesture dance” is a very popular short video form. However, its development is limited by the professionalism of choreography. In order to solve these problems, we propose an intelligent choreography framework, which can generate new gesture sequences for unseen audio based on pairing data in the database. Our framework adopts multimodal method and obtains excellent results. In additional, we collected and produced the first and largest pair labeled hand gesture dance data set. Various experiments showed that our results not only generate smooth and rich action sequences, but also collect some semantic information contained in the audio.

PD01-10 USING SURGICAL GESTURES TO BUILD EXPLAINABLE ARTIFICIAL INTELLIGENCE FOR SURGICAL SKILLS ASSESSMENT

You have accessJournal of UrologyCME1 Apr 2023PD01-10 USING SURGICAL GESTURES TO BUILD EXPLAINABLE ARTIFICIAL INTELLIGENCE FOR SURGICAL SKILLS ASSESSMENT Mitchell G Goldenberg, Runzhuo Ma, Elyssa Y. Wong, Timothy N. Chu, Christian Wagner, and Andrew J. Hung Mitchell G GoldenbergMitchell G Goldenberg More articles by this author , Runzhuo MaRunzhuo Ma More articles by this author , Elyssa Y. WongElyssa Y. Wong More articles by this author , Timothy N. ChuTimothy N. Chu More articles by this author , Christian WagnerChristian Wagner More articles by this author , and Andrew J. HungAndrew J. Hung More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003218.10AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Traditional assessment of surgical performance is based on the evaluation of human experts. Artificial intelligence (AI)-based assessments enhance the efficiency and scalability of these assessments but lack transparency in how these scores are calculated. Deconstructing surgical actions into their most basic movements may improve the objectivity and explainability of these assessments. We sought to understand whether a surgical gestures-trained AI algorithm can predict validated measures of technical skill, as a first step toward transparency in automated surgical skills assessment. METHODS: Data was prospectively collected from two international institutions. Videos of the nerve-spare (NS) step of Robotic-assisted radical prostatectomy (RARP) cases were blindly analyzed by trained human raters using the validated Dissection Assessment for Robotic Technique (DART) tool, and surgical gestures based on a previously published classification were separately annotated (Figure 1a). Surgeon and patient demographics for each case were abstracted. Spearman’s rank correlation was used to identify significant associations between DART skill domains and the proportion of surgical gestures used. The cohort was divided 80:20 for training and validation, and an interpretable recurrent neural network (IMV-LSTM) was used to extract information from gesture sequences to predict DART domains. RESULTS: Eighty cases were included in the analysis from 21 surgeons. Median prior experience was 450 cases (IQR 230-2000). A median of 438 discrete gestures (IQR 254-559) was identified per NS case. Grouping gestures by DART skill domains we found multiple positive and negative correlations (Figure 1a), with total DART score significantly associated with proportion of hook and clip gestures (p<.001) (Figure 1b). The neural network was able to predict DART domains, and AUCs for predicting tissue handling, tissue retraction, and efficiency were 0.64, 0.66, and 0.64 respectively (Figure 1c). CONCLUSIONS: Surgical gestures may provide a link between the objectivity and scalability of AI-based technical skill assessments, and the explainability and familiarity of human expert-based ones. As metrics of surgical performance continue to diversify, these data support a multifaceted approach in evaluation of surgical performance. Source of Funding: Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA273031. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health © 2023 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 209Issue Supplement 4April 2023Page: e67 Advertisement Copyright & Permissions© 2023 by American Urological Association Education and Research, Inc.MetricsAuthor Information Mitchell G Goldenberg More articles by this author Runzhuo Ma More articles by this author Elyssa Y. Wong More articles by this author Timothy N. Chu More articles by this author Christian Wagner More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement PDF downloadLoading ...

SequenceSense: A Tool for Designing Usable Foot-Based Gestures Using a Sequence-Based Gesture Recognizer

Foot-based gestures enable people to interact with mobile and wearable devices when their hands are unavailable for interaction. For the foot gestures to be truly usable, the gestures should be recognizable by the system without being confused by daily activities and still be easy to perform. However, designing such gestures often requires multiple iterations of gesture design, model training, and evaluation. In this paper, we present SequenceSense, a tool developed to help designers efficiently design a usable gesture set using inertial sensors, which eliminates the need for multiple data collection studies to evaluate the gestures’ usability through gesture modification by sequencing atomic actions and instant false positive analysis, and instead requires only the initial gesture sample collection. Unlike gesture recognizers using complete gestures to train a model, SequenceSense segments gesture into a sequence of atomic actions. For example, a foot tap to the right may have (1) lift the foot, (2) move the foot to the right, and (3) land the foot. SequenceSense also compares the gesture sequence with the sequence database created from the daily activities to identify possible conflicts. This allows gesture designers to build easily usable foot-based gestures without the need for recollecting and evaluating gestures. We validated SequenceSense’s efficacy in designing usable gestures with low false positives through a user study with nine gesture designers.

GesturalOrigins: A bottom-up framework for establishing systematic gesture data across ape species

Current methodologies present significant hurdles to understanding patterns in the gestural communication of individuals, populations, and species. To address this issue, we present a bottom-up data collection framework for the study of gesture: GesturalOrigins. By “bottom-up”, we mean that we minimise a priori structural choices, allowing researchers to define larger concepts (such as ‘gesture types’, ‘response latencies’, or ‘gesture sequences’) flexibly once coding is complete. Data can easily be re-organised to provide replication of, and comparison with, a wide range of datasets in published and planned analyses. We present packages, templates, and instructions for the complete data collection and coding process. We illustrate the flexibility that our methodological tool offers with worked examples of (great ape) gestural communication, demonstrating differences in the duration of action phases across distinct gesture action types and showing how species variation in the latency to respond to gestural requests may be revealed or masked by methodological choices. While GesturalOrigins is built from an ape-centred perspective, the basic framework can be adapted across a range of species and potentially to other communication systems. By making our gesture coding methods transparent and open access, we hope to enable a more direct comparison of findings across research groups, improve collaborations, and advance the field to tackle some of the long-standing questions in comparative gesture research.

Sign Language Gesture to Speech Conversion Using Convolutional Neural Network

A genuine disability prevents a person from speaking. There are numerous ways for people with this condition to communicate with others, including sign language, which is one of the more widely used forms of communication. Human body language can be used to communicate with one another using sign language, where each word is represented by a specific sequence of gestures. The goal of the paper is to translate human sign language into speech that can interpret human gestures. Through a deep convolution neural network, we first construct the data-set, save the hand gestures in the database, and then use an appropriate model on these hand gesture visuals to test and train the system. When a user launches the application, it then detects the gestures that are saved inthe database and displays the corresponding results. By employing this system, it is possible to assist those who are hard of hearing while simultaneously making communication with them simpler for everyone else.

Interaction with a Hand Rehabilitation Exoskeleton in EMG-Driven Bilateral Therapy: Influence of Visual Biofeedback on the Users' Performance.

The effectiveness of EMG biofeedback with neurorehabilitation robotic platforms has not been previously addressed. The present work evaluates the influence of an EMG-based visual biofeedback on the user performance when performing EMG-driven bilateral exercises with a robotic hand exoskeleton. Eighteen healthy subjects were asked to perform 1-min randomly generated sequences of hand gestures (rest, open and close) in four different conditions resulting from the combination of using or not (1) EMG-based visual biofeedback and (2) kinesthetic feedback from the exoskeleton movement. The user performance in each test was measured by computing similarity between the target gestures and the recognized user gestures using the L2 distance. Statistically significant differences in the subject performance were found in the type of provided feedback (p-value 0.0124). Pairwise comparisons showed that the L2 distance was statistically significantly lower when only EMG-based visual feedback was present (2.89 ± 0.71) than with the presence of the kinesthetic feedback alone (3.43 ± 0.75, p-value = 0.0412) or the combination of both (3.39 ± 0.70, p-value = 0.0497). Hence, EMG-based visual feedback enables subjects to increase their control over the movement of the robotic platform by assessing their muscle activation in real time. This type of feedback could benefit patients in learning more quickly how to activate robot functions, increasing their motivation towards rehabilitation.

UltrasonicGS: A Highly Robust Gesture and Sign Language Recognition Method Based on Ultrasonic Signals.

With the global spread of the novel coronavirus, avoiding human-to-human contact has become an effective way to cut off the spread of the virus. Therefore, contactless gesture recognition becomes an effective means to reduce the risk of contact infection in outbreak prevention and control. However, the recognition of everyday behavioral sign language of a certain population of deaf people presents a challenge to sensing technology. Ubiquitous acoustics offer new ideas on how to perceive everyday behavior. The advantages of a low sampling rate, slow propagation speed, and easy access to the equipment have led to the widespread use of acoustic signal-based gesture recognition sensing technology. Therefore, this paper proposed a contactless gesture and sign language behavior sensing method based on ultrasonic signals-UltrasonicGS. The method used Generative Adversarial Network (GAN)-based data augmentation techniques to expand the dataset without human intervention and improve the performance of the behavior recognition model. In addition, to solve the problem of inconsistent length and difficult alignment of input and output sequences of continuous gestures and sign language gestures, we added the Connectionist Temporal Classification (CTC) algorithm after the CRNN network. Additionally, the architecture can achieve better recognition of sign language behaviors of certain people, filling the gap of acoustic-based perception of Chinese sign language. We have conducted extensive experiments and evaluations of UltrasonicGS in a variety of real scenarios. The experimental results showed that UltrasonicGS achieved a combined recognition rate of 98.8% for 15 single gestures and an average correct recognition rate of 92.4% and 86.3% for six sets of continuous gestures and sign language gestures, respectively. As a result, our proposed method provided a low-cost and highly robust solution for avoiding human-to-human contact.

Describing movement learning using metric learning.

Analysing movement learning can rely on human evaluation, e.g. annotating video recordings, or on computing means in applying metrics on behavioural data. However, it remains challenging to relate human perception of movement similarity to computational measures that aim at modelling such similarity. In this paper, we propose a metric learning method bridging the gap between human ratings of movement similarity in a motor learning task and computational metric evaluation on the same task. It applies metric learning on a Dynamic Time Warping algorithm to derive an optimal set of movement features that best explain human ratings. We evaluated this method on an existing movement dataset, which comprises videos of participants practising a complex gesture sequence toward a target template, as well as the collected data that describes the movements. We show that it is possible to establish a linear relationship between human ratings and our learned computational metric. This learned metric can be used to describe the most salient temporal moments implicitly used by annotators, as well as movement parameters that correlate with motor improvements in the dataset. We conclude with possibilities to generalise this method for designing computational tools dedicated to movement annotation and evaluation of skill learning.

Sign Language Recognition using Deep Neural Network

Abstract—Sign language recognition is an attractive research field with a wide range of applications including telesurgery techniques. Another important application of hand gesture recognition is the translation of sign language, which is a complicated structured form of hand gestures. In sign language, the fingers’ configuration, the hand’s orientation, and the hand’s relative position to the body are the basis of structured expressions. The importance of hand gesture recognition has increased due to the prevalence of touchless applications and the rapid growth of the hearing-impaired population. However, developing an efficient recognition system needs to overcome the challenges of hand segmentation, local hand shape representation, global body configuration representation, and gesture sequence modeling. In this paper, a novel system is proposed for dynamic hand gesture recognition using multiple deep learning architectures for hand segmentation, local and global feature representations, and sequence feature globalization and recognition. The proposed system is evaluated on a very challenging dataset, which consists of dynamic hand gestures performed by subjects in an uncontrolled environment. The results show that the proposed system outperforms state-of- the-art approaches, demonstrating its effectiveness.