Gesture Instructions Research Articles

From Hands to Mind: How Gesture, Emotional Valence, and Individual Differences Impact Narrative Recall

Narrative recall and comprehension are important lifelong skills. While gesturing may improve recall by alleviating cognitive load, it may be differentially beneficial, depending on task and individual characteristics. While research on gesture’s effects on a variety of task modalities is burgeoning, effects on recall of narratives read aloud are under-examined. Further, emotional valence and individual differences in verbal memory may affect recall, through effects on task difficulty. If gesturing lightens cognitive load, it may be more beneficial for harder tasks, namely for narratives lacking emotional content and for individuals with poorer verbal memory. Across two studies, impacts of gesture production, emotional valence, and individual differences on narrative recall were evaluated. In Study 1, participants (N = 100) read aloud three emotive narratives (positive, negative, neutral) while either instructed to gesture or receiving no gesture instructions. Gesture production hindered recall, particularly for those with higher verbal memory. Emotion benefited recall, with enhanced recall of the negative narrative and impaired recall for the neutral narrative. In Study 2, following a measure of individual propensity to gesture, participants (N = 98) similarly read aloud three emotive narratives. Instructions to gesture hindered recall for participants with a lower propensity to gesture, and emotional narratives again saw enhanced recall relative to the neutral narrative. Propensity to gesture and verbal memory were positively associated with narrative comprehension. Results suggest instructions to produce gestures may for some individuals hinder recall for self-guided learners when studying written texts, while emotional content benefits recall.

Metaverse‐based remote support system with smooth combination of free viewpoint observation and hand gesture instruction

AbstractIn field operations with large machines, onsite large task spaces have been typically required. In these cases, remote skilled workers have to provide a step‐by‐step guide by observing the onsite situation around the large task spaces in three dimensions, instructing onsite unskilled workers on how to perform the tasks with the correct hand gestures at the right position, and switching between observation and instruction frequently during the remote support. In this study, we developed a Metaverse‐based remote support system with a seamless user interface for switching between free viewpoint observation and hand gesture instruction. The proposed system enables remote skilled workers to observe the onsite field from any viewpoint, transfer hand gesture instruction to onsite workers, and seamlessly switch between free viewpoint observation and free hand gesture instruction without having to change devices. We compared the time efficiency of the proposed system and a conventional system through experiments with 28 users and found that our system improved the observation time efficiency by 65.7%, the instruction time efficiency by 27.9%, and the switching time efficiency between observation and instruction by 14.6%. These results indicate that the proposed system enables remote skilled workers to support onsite workers quickly and efficiently.

Learners' Spontaneous Gesture Before a Math Lesson Predicts the Efficacy of Seeing Versus Doing Gesture During the Lesson.

Gestures-hand movements that accompany speech and express ideas-can help children learn how to solve problems, flexibly generalize learning to novel problem-solving contexts, and retain what they have learned. But does it matter who is doing the gesturing? We know that producing gesture leads to better comprehension of a message than watching someone else produce gesture. But we do not know how producing versus observing gesture impacts deeper learning outcomes such as generalization and retention across time. Moreover, not all children benefit equally from gesture instruction, suggesting that there are individual differences that may play a role in who learns from gesture. Here, we consider two factors that might impact whether gesture leads to learning, generalization, and retention after mathematical instruction: (1) whether children see gesture or do gesture and (2) whether a child spontaneously gestures before instruction when explaining their problem-solving reasoning. For children who spontaneously gestured before instruction, both doing and seeing gesture led to better generalization and retention of the knowledge gained than a comparison manipulative action. For children who did not spontaneously gesture before instruction, doing gesture was less effective than the comparison action for learning, generalization, and retention. Importantly, this learning deficit was specific to gesture, as these children did benefit from doing the comparison manipulative action. Our findings are the first evidence that a child's use of a particular representational format for communication (gesture) directly predicts that child's propensity to learn from using the same representational format.

Individual differences in working memory predict the efficacy of experimenter-manipulated gestures in first-grade children.

Why is instructional gesture ineffective in some contexts? And what is it about learners that predicts whether they will learn from gestures? This between-subjects linear measurement training study compares gesture instruction to two controls-operant action and transient action-in a diverse sample of first-grade students (N = 174, Mage = 7.01 years; Nfemale = 84; Nmale = 90, 10% Latinx-identified; 70% White; 6% Black; 6% Asian; 18% multiple racial categories, Mincome = $59,750, SDincome ≈ $25,000; data collected 03/16-03/19). Results show that instructor-manipulated gestures may be less effective than demonstrative actions in part because they are iterative and do not leave a lasting trace. Verbal working memory, but not spatial, positively predicted an ability to learn from gesture and transient action in children with the lowest context knowledge.

Neuro-Symbolic Reasoning for Multimodal Referring Expression Comprehension in HMI Systems

AbstractConventional Human–Machine Interaction (HMI) interfaces have predominantly relied on GUI and voice commands. However, natural human communication also consists of non-verbal communication, including hand gestures like pointing. Thus, recent works in HMI systems have tried to incorporate pointing gestures as an input, making significant progress in recognizing and integrating them with voice commands. However, existing approaches often treat these input modalities independently, limiting their capacity to handle complex multimodal instructions requiring intricate reasoning of language and gestures. On the other hand, multimodal tasks requiring complex reasoning are being challenged in the language and vision domain, but these typically do not include gestures like pointing. To bridge this gap, we explore one of the challenging multimodal tasks, called Referring Expression Comprehension (REC), within multimodal HMI systems incorporating pointing gestures. We present a virtual setup in which a robot shares an environment with a user and is tasked with identifying objects based on the user’s language and gestural instructions. Furthermore, to address this challenge, we propose a hybrid neuro-symbolic model combining deep learning’s versatility with symbolic reasoning’s interpretability. Our contributions include a challenging multimodal REC dataset for HMI systems, an interpretable neuro-symbolic model, and an assessment of its ability to generalize the reasoning to unseen environments, complemented by an in-depth qualitative analysis of the model’s inner workings.

How Prior Knowledge, Gesture Instruction, and Interference After Instruction Interact to Influence Learning of Mathematical Equivalence.

Although children learn more when teachers gesture, it is not clear how gesture supports learning. Here, we sought to investigate the nature of the memory processes that underlie the observed benefits of gesture on lasting learning. We hypothesized that instruction with gesture might create memory representations that are particularly resistant to interference. We investigated this possibility in a classroom study with 402 second- and third-grade children. Participants received classroom-level instruction in mathematical equivalence using videos with or without accompanying gesture. After instruction, children solved problems that were either visually similar to the problems that were taught, and consistent with an operational interpretation of the equal sign (interference), or visually distinct from equivalence problems and without an equal sign (control) in order to assess the role of gesture in resisting interference after learning. Gesture facilitated learning, but the effects of gesture and interference varied depending on type of problem being solved and the strategies that children used to solve problems prior to instruction. Some children benefitted from gesture, while others did not. These findings have implications for understanding the mechanisms underlying the beneficial effect of gesture on mathematical learning, revealing that gesture does not work via a general mechanism like enhancing attention or engagement that would apply to children with all forms of prior knowledge.

Exploring Individual Differences: A Case for Measuring Children's Spontaneous Gesture Production as a Predictor of Learning From Gesture Instruction.

Decades of research have established that learners benefit when instruction includes hand gestures. This benefit is seen when learners watch an instructor gesture, as well as when they are taught or encouraged to gesture themselves. However, there is substantial individual variability with respect to this phenomenon-not all individuals benefit equally from gesture instruction. In the current paper, we explore the sources of this variability. First, we review the existing research on individual differences that do or do not predict learning from gesture instruction, including differences that are either context-dependent (linked to the particular task at hand) or context-independent (linked to the learner across multiple tasks). Next, we focus on one understudied measure of individual difference: the learner's own spontaneous gesture rate. We present data showing rates of "non-gesturers" across a number of studies and we provide theoretical motivation for why this is a fruitful area for future research. We end by suggesting ways in which research on individual differences will help gesture researchers to further refine existing theories and develop specific predictions about targeted gesture intervention for all kinds of learners.

Guiding Technology for Faster Work by Controlling the Presentation Speed of Hand Gesture Instruction in Augmented Reality

Guiding Technology for Faster Work by Controlling the Presentation Speed of Hand Gesture Instruction in Augmented Reality

Unlocking the Power of Gesture: Using Movement-Based Instruction to Improve First Grade Children's Spatial Unit Misconceptions.

Gestures are hand movements that are produced simultaneously with spoken language and can supplement it by representing semantic information, emphasizing important points, or showing spatial locations and relations. Gestures' specific features make them a promising tool to improve spatial thinking. Yet, there is recent work showing that not all learners benefit equally from gesture instruction and that this may be driven, in part, by children's difficulty understanding what an instructor's gesture is intended to represent. The current study directly compares instruction with gestures to instruction with plastic unit chips (Action) in a linear measurement learning paradigm aimed at teaching children the concept of spatial units. Some children performed only one type of movement, and some children performed both: Action-then-Gesture [AG] or Gesture-then-Action [GA]. Children learned most from the Gesture-then-Action [GA] and Action only [A] training conditions. After controlling for initial differences in learning, the gesture-then-action condition outperformed all three other training conditions on a transfer task. While gesture is cognitively challenging for some learners, that challenge may be desirable-immediately following gesture with a concrete representation to clarify that gesture's meaning is an especially effective way to unlock the power of this spatial tool and lead to deep, generalizable learning.

Classroom gesture instruction on second language learners' academic presentations: Evidence from Chinese intermediate English learners

Gesture has been recognized as an important aspect of second language (L2) acquisition, playing a vital role in L2 communication. Previous research has shown that L2 learners' communicative efficacy can be facilitated through gesture mediation. In particular, L2 learners may find gesture a useful tool to deliver their subject-related academic presentations. However, whether classroom gesture instruction could boost L2 learners' gesture use in academic speaking context and their overall presentation performance has rarely been explored. Drawing on activity theory, this study aims to investigate the potential role of gesture instruction in intermediate L2 English learners' gesture use and their academic presentation performance. Our study found that L2 learners who received classroom gesture instructions showed significant improvement in their use of gestures, and outperformed the control group in a post-test presentation session, showing improvement in all gesture types among which beats were significantly increased. These findings confirm the teachability of gesture to enhance Chinese L2 English learners' academic presentation skills, proving gesture as an indispensable instructional component in English for Academic Purposes (EAP) classroom.

Development of a Remote Field Operation Support System with Functions of Free Viewpoint Observation and Free Hand Gesture Instruction via Metaverse

Development of a Remote Field Operation Support System with Functions of Free Viewpoint Observation and Free Hand Gesture Instruction via Metaverse

Adoption of Artificial Intelligence Along with Gesture Interactive Robot in Musical Perception Education Based on Deep Learning Method

The present work aims to improve students’ interest in music teaching and promote modern teaching. A distributed application system of artificial intelligence gesture interactive robot is designed through deep learning technology and applied to music perception education. First, the user’s gesture instruction data is collected through the double channel convolution neural network (DCCNN). It uses the double-size convolution kernel to extract feature information in the image and collect the video frame’s gesture instruction. Secondly, a two-stream convolutional neural network (two-stream CNN) recognizes the collected gesture instruction data. The spatial and temporal information is extracted from RGB color mode (RGB) images and optical flow images and input into the two-stream CNN to fuse the prediction results of each network as the final detection result. Then, the distributed system used by the interactive robot is introduced. This structure can improve the stability of the interactive systems and reduce the requirements for local hardware performance. Finally, experiments are conducted to test the gesture command acquisition and recognition network, and the performance of the gesture interactive robot in practice. The results indicate that combining convolution kernels of [Formula: see text] and [Formula: see text] can increase the recognition accuracy of DCCNN to 98% and effectively collect gesture instruction data. The gesture recognition accuracy of two-stream CNN after training reaches 90%, higher than the mainstream dynamic gesture recognition algorithm trained with the same data set. Finally, the recognition test of gesture instructions is carried out on the gesture interactive robot reported here. The results show that the recognition accuracy of the gesture interactive robots is more than 90%, meeting the routine interaction needs. Therefore, the interactive gesture robot has good reliability and stability and is applicable to music perception teaching. The research reported here has guiding significance for establishing music teaching with multiple perception modes.

What parents bring to preschool children? Parental instructive speech and gestures, children’s learning and cultural differences

Instructions via speech and gestures provided by caregivers can facilitate children’s learning. However, caregivers with different cultural backgrounds demonstrate diverse parental behaviors, which exert influences on children’s learning in different ways. The current study aimed to explore the cultural differences between Chinese and American caregivers in regard to the way they use speech and gestures to instruct their children. The potential cross-cultural differences in the facilitating effects of parental instructions on children’s learning were also examined. Two groups of children aged 3–4 years and their caregivers from China and the U.S. were invited to participate in our study (34 from China and 35 from the U.S.). The caregivers assisted their children in working with a difficult puzzle task. The caregivers’ speech and gestural instructions from both cultural groups, as well as the children’s learning performance, were coded and analyzed. The results showed that relative to the caregivers in the U.S., those from China provided more instructions to their children. Cultural background was a significant moderator in the relationship between parental instructions and children’s learning performance, namely, speech instructions provided by Chinese caregivers had a more effective facilitating influence on their children’s learning than did those provided by American caregivers.

Exploring how visual attention, inhibitory control, and co-speech gesture instruction contribute to children’s analogical reasoning ability

Analogical reasoning is an important, but difficult skill to develop. Here, we consider (1) whether children’s difficulty with analogical reasoning can be understood by measuring visual attention during analogy problem solving and a measure of inhibitory control, which researchers have suggested plays an integral role in analogical reasoning, and (2) whether analogical reasoning can be supported with co-speech gesture instruction. Children (4–5 year-olds) solved scene analogies before and after speech-alone or speech + gesture instruction. Results suggest relations exist between some behavioral performance measures, inhibitory control, and visual attention, but some of these relations change after instruction. During instruction, gesture led to more mature looking patterns, although this did not predict performance: All children showed gains from pretest to posttest. We consider how these findings extend prior gesture and analogical reasoning literature, call for replication of these findings, and identify future directions of research.

Assessing Collaborative Physical Tasks Via Gestural Analysis

Recent studies have shown that gestures are useful indicators of understanding, learning, and memory retention. However, and specially in collaborative settings, current metrics that estimate task understanding often neglect the information expressed through gestures. This work introduces the physical instruction assimilation (PIA) metric, a novel approach to estimate task understanding by analyzing the way in which collaborators use gestures to convey, assimilate, and execute physical instructions. PIA estimates task understanding by inspecting the number of necessary gestures required to complete a shared task. PIA is calculated based on the multiagent gestural instruction comparer (MAGIC) architecture, a previously proposed framework to represent, assess, and compare gestures. To evaluate our metric, we collected gestures from collaborators remotely completing the following three tasks: block assembly, origami, and ultrasound training. The PIA scores of these individuals are compared against two other metrics used to estimate task understanding: number of errors and amount of idle time during the task. Statistically significant correlations between PIA and these metrics are found. Additionally, a Taguchi design is used to evaluate PIA's sensitivity to changes in the MAGIC architecture. The factors evaluated the effect of changes in time, order, and motion trajectories of the collaborators' gestures. PIA is shown to be robust to these changes, having an average mean change of 0.45. These results hint that gestures, in the form of the assimilation of physical instructions, can reveal insights of task understanding and complement other commonly used metrics.

Teaching Analogical Reasoning With Co-speech Gesture Shows Children Where to Look, but Only Boosts Learning for Some.

In general, we know that gesture accompanying spoken instruction can help children learn. The present study was conducted to better understand how gesture can support children’s comprehension of spoken instruction and whether the benefit of teaching though speech and gesture over spoken instruction alone depends on differences in cognitive profile – prior knowledge children have that is related to a to-be-learned concept. To answer this question, we explored the impact of gesture instruction on children’s analogical reasoning ability. Children between the ages of 4 and 11 years solved scene analogy problems before and after speech alone or speech and gesture instruction while their visual attention was monitored. Our behavioral results suggest a marginal benefit of gesture instruction over speech alone, but only 5-year-old children showed a distinct advantage from speech + gesture instruction when solving the post-instruction trial, suggesting that at this age, children have the cognitive profile in place to utilize the added support of gesture. Furthermore, while speech + gesture instruction facilitated effective visual attention during instruction, directing attention away from featural matches and toward relational information was pivotal for younger children’s success post instruction. We consider how these results contribute to the gesture-for-learning literature and consider how the nuanced impact of gesture is informative for educators teaching tasks of analogy in the classroom.

An Underwater Human–Robot Interaction Using Hand Gestures for Fuzzy Control

Autonomous underwater vehicle (AUV) plays an important role in ocean research and exploration. The underwater environment has a great influence on AUV control and human–robot interaction, since underwater environment is highly dynamic with unpredictable fluctuation of water flow, high pressure and light attenuation. The traditional control model contains a large number of parameters, which is not effective and produces errors frequently. The proposal of fuzzy control addressed this issue to a certain extent. It applies fuzzy variables to the controller, which replace the values in an interval. In addition to the controller, underwater human–robot interaction is also difficult. Divers cannot speak or show any facial expressions underwater. The buttons on the AUV also need to overcome the huge water pressure. In this paper, we proposed a method to recognize the gesture instructions and apply it to the fuzzy control of AUV. Our contribution is the gesture recognition framework for the human–robot interaction, including the gesture detection network and the algorithm for the control of AUV. The experiment result shows the efficiency of the proposed method.

Encouraging kids to beat: Children's beat gesture production boosts their narrative performance.

Gesture is an integral part of language development. While recent evidence shows that observing a speaker who is simultaneously producing beat gestures helps preschoolers remember and understand information and also improves the production of oral narratives, little is known about the potential value of encouraging children to produce beat gestures-as opposed to merely observing them. In this between-subjects pretest-posttest training study we examine whether encouraging children to produce beats can boost their narrative performance. A total of 47 5- to 6-year-old children were divided into two groups and exposed to a training session in which a total of six stories were presented under one of two experimental conditions: (a) the children merely observed video-recordings of a storyteller who used beat gestures and were then asked to retell the narratives; or (b) the children observed the same video-recordings and then retold the narratives but were encouraged to simultaneously use their hands in the same way the storytellers did. Pretests and posttests consisting of children's narrations of short animated cartoons were analysed for narrative structure and fluency. A comparison of scores showed that children in the group that had been encouraged to use beat gestures in the training phase performed better in both narrative structure and fluency than the group of children who were simply asked to retell the story without gesture instruction. These findings suggest that linguistically relevant body movements serve to boost language development and that embodied storytelling can be of help in narrative training.

3DCNN-Based Real-Time Driver Fatigue Behavior Detection in Urban Rail Transit

With the rapid development of urban rail transit, traffic safety has become the focus of attention and people are paying increasing attention to the prevention of fatigue driving. “Gesture and oral instructions of urban rail traffic drivers” is operational actions of drivers written in the Chinese metro operation specification. It is a method to prevent drivers from fatigue driving and ensure safety. However, there is a lack of scientific detection methods. We combine the standard traffic operational actions with fatigue action to construct a fatigue detection system that is suitable for the urban rail transit industry. The system includes a dynamic tracking model for the large-scale operation of rail transit drivers and a dual-input action discrimination model based on a three-dimensional convolutional neural network (3DCNN). The model sets the skipping frame and continuous frame as two inputs of the model, and extracts five channels of information from the two inputs. Dual-input multi-channel information enables the model to learn not only the spatial and temporal information of the entire action, but also the subtle changes of the action. First, we trained and validated the dual-input model based on a 3DCNN using the open dataset KTH, which contains several variations. Then, the model trained on KTH was migrated to our data using the transfer learning method, which saved training time and achieves an accuracy of 98.41%. This transfer learning scheme can also be applied when new categories are encountered in practice. Finally, we discussed and envisaged the future optimization of the system.

Gesture helps learners learn, but not merely by guiding their visual attention.

Teaching a new concept through gestures-hand movements that accompany speech-facilitates learning above-and-beyond instruction through speech alone (e.g., Singer & Goldin-Meadow, ). However, the mechanisms underlying this phenomenon are still under investigation. Here, we use eye tracking to explore one often proposed mechanism-gesture's ability to direct visual attention. Behaviorally, we replicate previous findings: Children perform significantly better on a posttest after learning through Speech+Gesture instruction than through Speech Alone instruction. Using eye tracking measures, we show that children who watch a math lesson with gesture do allocate their visual attention differently from children who watch a math lesson without gesture-they look more to the problem being explained, less to the instructor, and are more likely to synchronize their visual attention with information presented in the instructor's speech (i.e., follow along with speech) than children who watch the no-gesture lesson. The striking finding is that, even though these looking patterns positively predict learning outcomes, the patterns do not mediate the effects of training condition (Speech Alone vs. Speech+Gesture) on posttest success. We find instead a complex relation between gesture and visual attention in which gesture moderates the impact of visual looking patterns on learning-following along with speech predicts learning for children in the Speech+Gesture condition, but not for children in the Speech Alone condition. Gesture's beneficial effects on learning thus come not merely from its ability to guide visual attention, but also from its ability to synchronize with speech and affect what learners glean from that speech.