Two-handed Gestures Research Articles

A Multimodal Analysis of Teachers Two-handed Gestures in English Vocabulary Teaching

This paper selected a video from an English vocabulary teaching course and analyzed the teachers two-handed gestures, class discourse, and teaching procedures based on a multimodal discourse analysis framework. The researcher tried to explore the use of two-handed gestures in English vocabulary teaching classrooms and the relationship between gestures, teaching procedures, and discourse. It was concluded that beats are the most frequently-used gesture in this class which play an important communicative role in teaching as well as supports the teachers discourse. It appears frequently when the teachers discourse switches between two languages. This study enriches the corpus of pedagogical research applying multimodality and can provide a reference for the development of multimodal teaching.

Abstract No. 486 Augmented reality registration of three-dimensional models: a multicenter assessment using the next-generation HoloLens 2

Comparison of registration methods on HoloLens 2 for aligning 3D holographic models onto a CT grid amongst two academic centers to assess efficacy and reproducibility. A 3D holographic model of an abdominal CT phantom (CIRS 071B) with an overlying CT grid (Beekley Medical 117) was segmented using ITK-SNAP and projected using Microsoft HoloLens 2. Custom code was developed in Unity and C# for model registration using four methods: 1) one-handed gestures, 2) optional two-handed gestures, 3) Xbox controller, and 4) computer vision with automated image detection using Vuforia SDK. Code was shared and deployed across institutions. Registration times for aligning the model to the CT grid were recorded until visual satisfaction and alignment errors were < 1 cm in the x-, y-, and z-planes. Eighteen users from two academic centers (3 attendings, 3 trainees, and 3 medical students at each center) attempted each registration method using HoloLens 2 three times. Comparisons were made with paired t-tests within each center, F-tests, and unpaired t-tests between centers. Center 1 had mean manual registration times of 22.6s, 25.2s, and 77.2s, and Center 2 had times of 32.7s, 23.4s, 68.5s for one-handed gestures, two-handed gestures, and Xbox controller, respectively. Both centers showed faster registration times using one- and two-handed hand gestures compared to controller (P < 0.001 at both centers). Between centers, there was no significant difference in registration times for two-handed gestures and controller, but one-handed gestures were faster at Center 1 compared to Center 2 by 31% (P < 0.05). Both centers had fastest registration times with automated computer vision. Center 2’s mean automatic registration time was 3.4s versus Center 1’s time of 7.2s (52% reduction, P < 0.001). Registration times demonstrated similar trends within each center but showed variability in certain registration methods between centers. Prior exposure with the HoloLens may impact hologram interaction and ambient lighting conditions may affect computer vision detection. These findings encourage cross-institutional development and reinforce multicenter evaluation for new technologies, like augmented reality, in the IR suite.

SEMG-Based Neural Network Prediction Model Selection of Gesture Fatigue and Dataset Optimization.

The fatigue energy consumption of independent gestures can be obtained by calculating the power spectrum of surface electromyography (sEMG) signals. The existing research studies focus on the fatigue of independent gestures, while the research studies on integrated gestures are few. However, the actual gesture operation mode is usually integrated by multiple independent gestures, so the fatigue degree of integrated gestures can be predicted by training neural network of independent gestures. Three natural gestures including browsing information, playing games, and typing are divided into nine independent gestures in this paper, and the predicted model is established and trained by calculating the energy consumption of independent gestures. The artificial neural networks (ANNs) including backpropagation (BP) neural network, recurrent neural network (RNN), and long short-term memory (LSTM) are used to predict the fatigue of gesture. The support vector machine (SVM) is used to assist verification. Mean square error (MSE), root mean square error (RMSE), and mean absolute error (MAE) are utilized to evaluate the optimal prediction model. Furthermore, the different datasets of the processed sEMG signal and its decomposed wavelet coefficients are trained, respectively, and the changes of error functions of them are compared. The experimental results show that LSTM model is more suitable for gesture fatigue prediction. The processed sEMG signals are appropriate for using as the training set the fatigue degree of one-handed gesture. It is better to use wavelet decomposition coefficients as datasets to predict the high-dimensional sEMG signals of two-handed gestures. The experimental results can be applied to predict the fatigue degree of complex human-machine interactive gestures, help to avoid unreasonable gestures, and improve the user's interactive experience.

Beyond remote control: Exploring natural gesture inputs for smart TV systems

This paper reports on work that explores natural gesture inputs from end-users for television use in a typical living room setting. First, we derive a set of 19 user-defined freehand gestures for regular TV control tasks. This study helps us to determine the gestures preferred by TV viewers and red uce the risk of developing an unusable, ineffective system. Then, based on this user-defined gesture set, we propose a unified framework to address specific problems in a complex real-world TV viewing environment, including 1) the automatic exclusion of many meaningless daily actions by TV viewers, 2) the capability to recognize both static and dynamic gestures simultaneously, as well as one- and two-handed gestures simultaneously, and 3) the continuous recognition of multiple dynamic gestures in the air (e.g., a channel switching gesture for channel 127). Experimental results show that our approach allows users to interact with TV-based applications more flexibly and effectively, with improved user experience and user satisfaction. Finally, we highlight the implications of our work for the design and development of related freehand gesture applications.

User-defined gestures for controlling primitive motions of an end effector

In designing and developing a gesture recognition system, it is crucial to know the characteristics of a gesture selected to control, for example, an end effector of a robot arm. We conducted an experiment to collect a set of user-defined gestures and investigate characteristics of the gestures for controlling primitive motions of an end effector in human–robot collaboration. We recorded 152 gestures from 19 volunteers by presenting virtual robotic arm movements to the participants, and then asked the participants to think about and perform gestures that would cause the motions. It was found that the hands were the parts of the body used most often for gesture articulation even when the participants were holding tools and objects with both hands: a number of participants used one- and two-handed gestures interchangeably, gestures were consistently performed by the participants across all pairs of reversible gestures, and the participants expected better recognition performance for gestures that were easy to think of and perform. These findings are expected to be useful as guidelines in creating a gesture set for controlling robotic arms according to natural user behaviors.

The parallel development of the form and meaning of two-handed gestures and linguistic information packaging within a clause in narrative

AbstractWe examined how two-handed gestures and speech with equivalent contents that are used in narrative develop during childhood. The participants were 40 native speakers of English consisting of four different age groups: 3-, 5-, 9-year-olds, and adults. A set of 10 video clips depicting motion events were used to elicit speech and gesture. There are two findings. First, two types of two-handed gestures showed different developmental changes: those with a single-handed stroke with a simultaneous hold increased with age, while those with a two handed-stroke decreased with age. Second, representational gesture and speech developed in parallel at the discourse level. More specifically, the ways in which information is packaged in a gesture and in a clause are similar for a given age group; that is, gesture and speech develop hand-in-hand.

Design and evaluation of finger-count interaction: Combining multitouch gestures and menus

Selecting commands on multi-touch displays is still a challenging problem. While a number of gestural vocabularies have been proposed, these are generally restricted to one or two fingers or can be difficult to learn. We introduce Finger-Count gestures, a coherent set of multi-finger and two-handed gestures. Finger-Count gestures are simple, robust, expressive and fast to perform. In order to make these gestures self-revealing and easy to learn, we propose the Finger-Count menu, a menu technique and teaching method for implicitly learning Finger-Count gestures. We discuss the properties, advantages and limitations of Finger-Count interaction from the gesture and menu technique perspectives as well as its integration into three applications. We present alternative designs to increase the number of commands and to enable multi-user scenarios. Following a study which shows that Finger-Count is as easy to learn as radial menus, we report the results of an evaluation investigating which gestures are easier to learn and which finger chords people prefer. Finally, we present Finger-Count for in-the-air gestures. Thereby, the same gesture set can be used from a distance as well as when touching the surface.

Gesture imitation in musicians and non-musicians

Imitation plays a crucial role in the learning of many complex motor skills. Recent behavioral and neuroimaging evidence suggests that the ability to imitate is influenced by past experience, such as musical training. To investigate the impact of musical training on motor imitation, musicians and non-musicians were tested on their ability to imitate videoclips of simple and complex two-handed gestures taken from American Sign Language. Participants viewed a set of 30 gestures, one at a time, and imitated them immediately after presentation. Participants' imitations were videotaped and scored off-line by raters blind to participant group. Imitation performance was assessed by a rating of performance accuracy, where the arm, hand, and finger components of the gestures were rated separately on a 5-point scale (1 = unrecognizable; 5 = exact imitation). A global accuracy score (PAglobal) was calculated by summing the three components. Response duration compared to the model (%MTdiff), and reaction time (RT) were also assessed. Results indicated that musicians were able to imitate more accurately than non-musicians, reflected by significantly higher PAglobal and lower %MTdiff scores. Furthermore, the greatest difference in performance was for the fine-motor (finger) gesture component. These findings support the view that the ability to imitate is influenced by experience. This is consistent with generalist theories of motor imitation, which explain imitation in terms of links between perceptual and motor action representations that become strengthened through experience. It is also likely that musical training contributed to the ability to imitate manual gestures by influencing the personal action repertoire of musicians.

Explaining educational experience: On one- and two-handed gestures as semiotic entities and the flexibility of their use

This article studies hand gestures and specifies four different possible uses: (1) one-handed gestures; (2) symmetrical two-handed gestures; (3) symmetrical two-handed gestures in which the hands alternately perform the same movement ; (4) asymmetrical two-handed gestures. The paper shows how a speaker employs various hand gestures to explain her teaching experiences and views the concurrence ofgestures with speech, body posture, shrugs, gaze, and facial expressions as reciprocal actions to an interlocutor's responses. The final section of the paper discusses the flexibility of hand gestures and the different ways in which it is revealed.

A comparative study of two state-of-the-art sequence processing techniques for hand gesture recognition

In this paper, we address the problem of the recognition of isolated, complex, dynamic hand gestures. The goal of this paper is to provide an empirical comparison of two state-of-the-art techniques for temporal event modeling combined with specific features on two different databases. The models proposed are the Hidden Markov Model (HMM) and Input/Output Hidden Markov Model (IOHMM), implemented within the framework of an open source machine learning library (www.torch.ch). There are very few hand gesture databases available to the research community; consequently, most of the algorithms and features proposed for hand gesture recognition are not evaluated on common data. We thus propose to use two publicly available databases for our comparison of hand gesture recognition techniques. The first database contains both one- and two-handed gestures, and the second only two-handed gestures.

Automatic visual recognition of gestures made by motor-impaired computer users

The use of gesture recognition as a means of Human Computer Interaction for physically disabled users is discussed. The ability of motor-impaired computer users to make distinct, recognisable gestures is not exploited by current assistive technologies. A real-time computer vision system for recognition of one- and two-handed gestures defined by such users is described. An investigation into the feasibility of real-time, unencumbered recognition of gestures defined by motor-impaired users by means of Hidden Markov Models, trained with relatively few examples is performed and reported. Different feature vectors are compared and the trade-off between accuracy and training set size is explored: an important issue for such interactively trained systems.

Multimodal human discourse

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-accurate videoplayer to correlate the speech with the gestural entities. We compare the results of both analyses to identify the cues accessible in the gestural and audio data that correlate well with the expert psycholinguistic analysis. We show that "handedness" and the kind of symmetry in two-handed gestures provide effective supersegmental discourse cues.