Gesture Semantics Research Articles

Design and Optimization of Human-Computer Interaction System for Education Management Based on Artificial Intelligence

The continuous improvement and refinement of artificial intelligence (AI) technology has facilitated the broader application of human-computer interaction in the field of education management. The construction of an educational management human-computer interaction system based on AI technology can optimize and improve key parameters of educational management human-computer interaction scenarios, thereby creating a more comprehensive mobile learning (m-learning) application system. This paper is based on AI technology, analyzing gesture semantics and speech semantics, and combining fusion algorithms to construct an education management human interaction system. The performance changes of the system were compared with real experimental operations and the NOBOOK platform analysis. The results show that the education management human-computer interaction system constructed in this article can enhance the m-learning experience of participants. It ensures high recognition accuracy, leading to higher scores in all dimensions of indicator evaluation. Therefore, as one of the crucial forms of m-learning, the human-computer interaction system for education management based on AI can establish a foundation for the further enhancement and development of education management.

THROW

This paper provides a detailed description of the distribution of an utterance-accompanying or utterance-replacing throwing away gesture (see Bressem & Müller 2014, 2017), THROW, and proposes a formal analysis of its contribution. We argue that this gesture conveys dismissal, which we model as the marking of the question addressed by a preceding discourse move as unimportant. This work extends the growing body of linguistic work on formal gesture semantics to discourse-management gestures; moreover, we find that the dismissal meaning encoded by THROW is unlike other discourse-management operators in being unable to operate on propositional content that it accompanies.

Rhythmic Gesticulator

Automatic synthesis of realistic co-speech gestures is an increasingly important yet challenging task in artificial embodied agent creation. Previous systems mainly focus on generating gestures in an end-to-end manner, which leads to difficulties in mining the clear rhythm and semantics due to the complex yet subtle harmony between speech and gestures. We present a novel co-speech gesture synthesis method that achieves convincing results both on the rhythm and semantics. For the rhythm, our system contains a robust rhythm-based segmentation pipeline to ensure the temporal coherence between the vocalization and gestures explicitly. For the gesture semantics, we devise a mechanism to effectively disentangle both low- and high-level neural embeddings of speech and motion based on linguistic theory. The high-level embedding corresponds to semantics, while the low-level embedding relates to subtle variations. Lastly, we build correspondence between the hierarchical embeddings of the speech and the motion, resulting in rhythm- and semantics-aware gesture synthesis. Evaluations with existing objective metrics, a newly proposed rhythmic metric, and human feedback show that our method outperforms state-of-the-art systems by a clear margin.

TMS Reveals Dynamic Interaction between Inferior Frontal Gyrus and Posterior Middle Temporal Gyrus in Gesture-Speech Semantic Integration.

Semantic processing is an amodal process with modality-specific information integrated in supramodal "convergence zones" or "semantic hub" with executive mechanisms that tailor semantic representation in a task-appropriate way. One unsolved question is how frontal control region dynamically interacts with temporal representation region in semantic integration. The present study addressed this issue by using inhibitory double-pulse transcranial magnetic stimulation over the left inferior frontal gyrus (IFG) or left posterior middle temporal gyrus (pMTG) in one of eight 40 ms time windows (TWs) (3 TWs before and 5 TWs after the identification point of speech), when human participants (12 females, 14 males) were presented with semantically congruent or incongruent gesture-speech pairs but merely identified the gender of speech. We found a TW-selective disruption of gesture-speech integration, indexed by the semantic congruency effect (i.e., a cost of reaction time because of semantic conflict), when stimulating the left pMTG in TW1, TW2, and TW7 but when stimulating the left IFG in TW3 and TW6. Based on the timing relationship, we hypothesize a two-stage gesture-speech integration circuit with a pMTG-to-IFG sequential involvement in the prelexical stage for activating gesture semantics and top-down constraining the phonological processing of speech. In the postlexical stage, an IFG-to-pMTG feedback signal might be implicated for the control of goal-directed representations and multimodal semantic unification. Our findings provide new insights into the dynamic brain network of multimodal semantic processing by causally revealing the temporal dynamics of frontal control and temporal representation regions.SIGNIFICANCE STATEMENT Previous research has identified differential functions of left inferior frontal gyrus (IFG) and posterior middle temporal gyrus (pMTG) in semantic control and semantic representation, respectively, and a causal contribution of both regions in gesture-speech integration. However, it remains largely unclear how the two regions dynamically interact in semantic processing. By using double-pulse transcranial magnetic stimulation to disrupt regional activity at specific time, this study for the first time revealed critical time windows when the two areas were causally involved in integrating gesture and speech semantics. Findings suggest a pMTG-IFG-pMTG neurocircuit loop in gesture-speech integration, which deepens current knowledge and inspires future investigation of the temporal dynamics and cognitive processes of the amodal semantic network.

Discourse anaphoricity and first-person indexicality in emoji resolution

This paper proposes a formal semantic classification of emoji-text combinations, focusing on two core sets of emoji: face emoji and activity emoji. Based on different data sources (introspective intuitions, naturalistic Twitter examples, and experimental evidence), we argue that activity emoji (case study I) are essentially event descriptions that serve as separate discourse units (similar to free adjuncts) and connect to the accompanying text by virtue of suitable discourse relations. By contrast, face emoji (case study II) are expressive elements that are anchored to an attitude holder and comment on a proposition provided by the accompanying text. We conclude by revisiting emoji semantics from the perspective of formal gesture semantics: we probe interactions of emoji and texts that contain clausal negation, and conclude that emoji generally do not scope under negation; however, the appearance of such a scope relation arises when activity emoji are connected to the accompanying text by virtue of an Explanation discourse relation.

Non-Verbal Signs and Secret Communication as Universal Signs of Intercultural Communication

The article is devoted to the study of the national cognitive nature, the hidden essence of non-verbal techniques. The main purpose of the study is to prove that, based on materials on nonverbal uses in language, they are used to convey a confidential, hidden message, to show that in international communication there is a change in the semantics of nonverbal actions depending on the culture and individual knowledge of each nation. Methods In human life, in parallel with verbal communication, an experiment was conducted on nonverbal techniques, sometimes used in pure form and used to convey a secret and hidden message. The article presents an analysis based on a statistical method compared with some kinemes used in a common language. In the studies on non-verbal techniques, it is noted that they perform a main function in the relations between people. In this article, in the course of analyzing the nature of nonverbal techniques, we characterized some kinemes used depending on the worldview of the Kazakh people, and concluded that their background is directly related to the mystery. Application of the study: the results, materials, and conclusions of the work can be used in the history of language, sociolinguistics, and paralinguistic studies, and allow researchers of gesture semantics to explore nonverbal techniques from a new perspective. Scientists-linguists, psychologists have created many works related to the definition of human nature, character, and the place of nonverbal techniques in the speech act. However, it was not said that auxiliary means located outside the language will be used to transmit confidential, non-public information. In this work, we stopped at the place of nonverbal methods of transmitting confidential, hidden information and analyzed the functions in confidential communication.

Can gesture observation help people with aphasia name actions?

It has been suggested that gesture can play a role in the treatment of naming impairments in aphasia, however investigation is still sparse, especially when compared to research on verbal treatments. Critically, previous studies have included either verbal or gesture production in the training. However, while in speakers without language impairment, action naming is facilitated by gesture observation, no study has yet systematically determined whether gesture observation alone influences word retrieval in people with aphasia. This is the aim of the research presented here.In a gesture priming experiment, participants with aphasia named actions that were preceded by the observation of videos of congruent or unrelated gestures or a non-gesture control condition. At the group-level, action naming was facilitated by observation of congruent gestures. However, single-case analyses revealed variability in the extent to which the participants benefited from gesture cueing. The potential mechanisms underlying the effects of gesture observation on action picture naming in people with aphasia were examined by exploring participant-related and item-related predictors of improvement.It is concluded that gesture observation may facilitate verb retrieval at either semantic or lexical levels. In addition, and despite variability across individuals, gesture observation seems more likely to facilitate action naming in people with spared gesture semantics and mild-moderate deficits in lexical-semantic or post-semantic processing.

Many-to-One Gesture-to-Command Flexible Mapping Approach for Smart Teaching Interface Interaction

Mapping between gestures and semantics is inherently challenging, especially in the aspect of defining meaningful gesture semantics that are easy to understand and remember. To address this challenge, we put forward an intent-driven approach in this paper, which is based on users' interaction intent. We implemented a many-to-one flexible mapping between multiple gestures and one semantic object. We also designed and evaluated a domain-specific gestural input technique for classroom use by considering the intent of the user. The main contributions of this work consist of: (1) a mapping between multiple gestures and one semantics and (2) an intelligent and natural interacting interface model for a three-dimensional (3D) user interfaces (UIs). The proposed algorithm is evaluated with five frequently used gestures and applied to the UIs system. The experimental results demonstrate that the gesture-based algorithm performs well and can substantially reduce the memory loads of the user.

FM: Flexible mapping from one gesture to multiple semantics

An algorithm for flexible mapping (FM) from one gesture to multiple semantics in the same situational context is presented for the first time to reduce the cognitive and operating loads of an operator by using gesture commands. First, the foundation of the FM is built with the semantic-oriented difference features of behavior model (SDFBM) of the operator as implicit inputs. Second, the gesture–semantics FM algorithm and its extended process, namely, attribute classification for FM are proposed and implemented. Third, five commonly used gestures are designed to demonstrate the way a gesture is mapped to multiple semantics. Finally, several comparative experimental results are provided to demonstrate the superiority of the proposed methods to the state of the art. The main innovation of this study is that the same gesture in the same context can be mapped to several different semantics through the SDFBM feature recognition. This study provides an intelligent and natural interaction interface model for 3D platforms and key support for gesture-based implicit interaction design. The proposed algorithms are also tested or used in several applications, such as smart teaching interface and onboard vehicular systems and intelligent TV.

Latent Semantic Analysis for Multimodal User Input With Speech and Gestures

This paper describes our work in semantic interpretation of a “multimodal language” with speech and gestures using latent semantic analysis (LSA). Our aim is to infer the domain-specific informational goal of multimodal inputs. The informational goal is characterized by lexical terms used in the spoken modality, partial semantics of gestures in the pen modality, as well as term co-occurrence patterns across modalities, leading to “multimodal terms.” We designed and collected a multimodal corpus of navigational inquiries. We also obtained perfect (i.e. manual) and imperfect (i.e. automatic via recognition) transcriptions for these. We automatically align parsed spoken locative references (SLRs) with their corresponding pen gesture(s) using the Viterbi alignment, according to their numeric and location type features. Then, we characterize each cross-modal integration pattern as a 3-tuple multimodal term with SLR, pen gesture type and their temporal relationship. We propose to use latent semantic analysis (LSA) to derive the latent semantics from manual (i.e. perfect) and automatic (i.e. imperfect) transcriptions of the collected multimodal inputs. In order to achieve this, both multimodal and lexical terms are used to compose an inquiry-term matrix, which is then factorized using singular value decomposition (SVD) to derive the latent semantics automatically. Informational goal inference based on the latent semantics shows that the informational goal inference accuracy of a disjoint test set is 99% and 84% when a perfect and imperfect projection model is used respectively, which performs significantly better than (at least 9.9% absolute) the baseline performance using vector-space model (VSM).