Gesture Framework Research Articles

An Integrated Gesture Framework of Smart Entry Based on Arduino and Random Forest Classifier

Gesture-based systems have emerged as a prominent breakthrough in the field of smart access control, effectively integrating security measures with user comfort. This study presents a novel gesture detection framework for smart entry systems that combines the computational capabilities of a Random Forest Classifier with the practicality of Arduino-based hardware. Central to methodology is the utilization of MediaPipe, an advanced computer vision library, to extract hand motion landmarks from live video streams. The selected landmarks function as a comprehensive dataset for training a Random Forest Classifier, which has been specifically chosen due to its high level of accuracy and efficiency in managing intricate classification jobs. The model exhibits outstanding competence in the categorization of gestures in real-time, attaining high levels of accuracy that are crucial for ensuring dependable entrance control. The Arduino microcontroller plays a vital role in the execution of the entry mechanism as it serves as the intermediary between the gesture detection software and the tangible entry control hardware. The incorporation of gesture recognition technology facilitates a cohesive and prompt user experience, wherein identified motions are directly converted into input commands. The system's practical use is demonstrated through a series of detailed tests, which highlight its dependability and efficiency across diverse climatic circumstances. The findings underscore the system's capacity as a flexible and safe solution for contactless access in many environments, including both private homes and highly protected establishments. Furthermore, the study makes a substantial contribution to the larger domain of human-computer interaction by showcasing the practicality of advanced gesture detection systems in many everyday contexts. The suggested framework presents a novel approach to smart entry systems and also paves the way for further investigation in the domains of smart home automation and interactive systems. In these areas, gesture-based interfaces have the potential to deliver user experiences that are both intuitive and efficient.

Innovative Collaborative Method for Interaction between a Human Operator and Robotic Manipulator Using Pointing Gestures

The concept of “Industry 4.0” relies heavily on the utilization of collaborative robotic applications. As a result, the need for an effective, natural, and ergonomic interface arises, as more workers will be required to work with robots. Designing and implementing natural forms of human–robot interaction (HRI) is key to ensuring efficient and productive collaboration between humans and robots. This paper presents a gestural framework for controlling a collaborative robotic manipulator using pointing gestures. The core principle lies in the ability of the user to send the robot’s end effector to the location towards, which he points to by his hand. The main idea is derived from the concept of so-called “linear HRI”. The framework utilizes a collaborative robotic arm UR5e and the state-of-the-art human body tracking sensor Leap Motion. The user is not required to wear any equipment. The paper describes the overview of the framework’s core method and provides the necessary mathematical background. An experimental evaluation of the method is provided, and the main influencing factors are identified. A unique robotic collaborative workspace called Complex Collaborative HRI Workplace (COCOHRIP) was designed around the gestural framework to evaluate the method and provide the basis for the future development of HRI applications.

Neuromotor Control of Speech and Speechlike Tasks: Implications From Articulatory Gestures

Purpose This study aimed to provide a preliminary examination of the articulatory control of speech and speechlike tasks based on a gestural framework and identify shared and task-specific articulatory factors in speech and speechlike tasks. Method Ten healthy participants performed two speechlike tasks (i.e., alternating motion rate [AMR] and sequential motion rate [SMR]) and three speech tasks (i.e., reading of “clever Kim called the cat clinic” at the regular, fast, and slow rates) that varied in phonological complexity and rate. Articulatory kinematics were recorded using an electromagnetic kinematic tracking system (Wave, Northern Digital Inc.). Based on the gestural framework for articulatory phonology, the gestures of tongue body and lips were derived from the kinematic data. These gestures were subjected to a fine-grained analysis, which extracted (a) four gestural features (i.e., range of magnitude [ROM], frequency [Freq], acceleration time, and maximum speed [maxSpd]) for the tongue body gesture; (b) three intergestural measures including the peak intergestural coherence (InterCOH), frequency at which the peak intergestural coherence occurs (Freq_InterCOH), and the mean absolute relative phase between the tongue body and lip gestures; and (c) three intragestural (i.e., interarticulator) measures including the peak intragestural coherence (IntraCOH), Freq_IntraCOH, and mean absolute relative phase between the tongue body and the jaw, which are the component articulators that underlie the tongue body gesture. In addition, the performance rate for each task was also derived. The effects of task and sex on all the articulatory and behavioral measures were examined using mixed-design analysis of variance followed by post hoc pairwise comparisons across tasks. Results Task had a significant effect on performance rate, ROM, Freq, maxSpd, InterCOH, Freq_InterCOH, IntraCOH, and Freq_IntraCOH. Compared to the speech tasks, the AMR task showed a decrease in ROM and increases in Freq, InterCOH, Freq_InterCOH, IntraCOH, and Freq_IntraCOH. The SMR task showed similar ROM, Freq, maxSpd, InterCOH, and IntraCOH as the fast and regular speech tasks. Conclusions The simple phonological structure and demand for rapid syllable rate for the AMR task may elicit a distinct articulatory control mechanism. Despite being a rapid nonsense syllable repetition task, the relatively complex phonological structure of the SMR task appeared to elicit a similar articulatory control mechanism as that of speech production. Based on these shared and task-specific articulatory features between speech and speechlike tasks, the clinical implications for articulatory assessment were discussed.

Sounding the body: the role of the Valsalva mechanism in the emergence of the linguistic sign

he main aim of this study, conducted within STEELS, a gestural theory of the origins of speech, is to set out a proposal as to the possible role of the Valsalva mechanism in the emergence of the linguistic sign. STEELS posits that in the earliest forms of speech developed by Homo, vocomimetic laryngeal resonances of nonlinguistic origin were integrated into LV (laryngeal + vowel) protosyllables referring back to oro-naso-laryngeal (ONL) actions such as breathing, sneezing and coughing. It further posits that these protosyllables were conceptually mapped to non-ONL bodily actions making use of the Valsalva manoeuvre, such as lifting, birthing, and defecating. This claim, which stems from a submorphemic analysis of certain Proto-Indo-European “body-part” roots projected back, within a gestural framework, to the emergence of speech, suggests that the vocomimetic protosyllables posited would have become (self-)referential through a neurocognitive process of recurrent, somatotopically-driven pattern-extraction.

Understanding the Meaningfulness of Vocal and Instrumental Music Teachers' Hand Gestures Through the Teacher Behavior and Gesture Framework

This article considers the usefulness of the recently proposed ‘Teacher Behaviour and Gesture (TBG) framework’ for understanding the meaningfulness of teachers’ hand gestures from an Enactive Cognition perspective (wherein cognition is fundamentally considered as embodied, embedded, enactive and extended). The framework is based on the main premise that gestures are part of an integral musical communicational process, fully integrated with speech and/or music-making and contextualised within specific teaching behaviours. By considering teachers’ teaching behaviours, it is possible to realise teachers’ pedagogical intentions. This, in turn, enables deeper understandings on teachers’ gestures from the points of view of meaning, function and purpose. Application of the TBG framework across instrumental and vocal music pedagogical contexts (one-to-one, small and large teaching groups) will bring relevant insights on developing practical scaffolding approaches, with direct implications on the quality of teaching and learning, for the benefit of both teachers and students.

Motoric Mechanisms for the Emergence of Non-local Phonological Patterns.

Non-local phonological patterns can be difficult to analyze in the context of speech production models. Some patterns – e.g., vowel harmonies, nasal harmonies – can be readily analyzed to arise from temporal extension of articulatory gestures (i.e., spreading); such patterns can be viewed as articulatorily local. However, there are other patterns – e.g., nasal consonant harmony, laryngeal feature harmony – which cannot be analyzed as spreading; instead these patterns appear to enforce agreement between features of similar segments without affecting intervening segments. Indeed, there are numerous typological differences between spreading harmonies and agreement harmonies, and this suggests that there is a mechanistic difference in the ways that spreading and agreement harmonies arise. This paper argues that in order to properly understand spreading and agreement patterns, the gestural framework of Articulatory Phonology must be enriched with respect to how targets of the vocal tract are controlled in planning and production. Specifically, it is proposed that production models should distinguish between excitatory and inhibitory articulatory gestures, and that gestures which are below a selection threshold can influence the state of the vocal tract, despite not being active. These ideas are motivated by several empirical phenomena, which include anticipatory posturing before production of a word form, and dissimilatory interactions in distractor-target response paradigms. Based on these ideas, a model is developed which provides two distinct mechanisms for the emergence of non-local phonological patterns.

A framework for studying teachers’ hand gestures in instrumental and vocal music contexts

A framework for studying teachers’ hand gestures in instrumental music pedagogy is proposed, focusing on teachers’ teaching behaviours as a context-dependent basis for understanding the meaning and functionality of their gestures. The application of the Teacher Behaviour and Gesture framework across instrumental music pedagogical settings (one-to-one, small and large teaching groups and across singing, woodwind, brass, strings, and other pedagogical contexts) will bring understandings on the role of teachers’ gestures in their pedagogical interactions with students, with implications for student learning and instrumental music teachers’ teaching and education.

A multimodal framework for tracking Sesshomaru's character development in an anime movie – Inuyasha: Swords of an Honourable Ruler – an appraisal and gestural perspective

In this article, I propose an integrated multimodal framework for tracking a Japanese character's development, that of Sesshomaru, in the anime movie, Inuyasha: Swords of an Honourable Ruler. The language part of the framework focuses specifically on the appraisal analysis of the language spoken by the characters. The visual framework builds on Lim's system of gesture by integrating Martin and White's system nodes for appraisal analysis into Lim's gesture framework for a nuanced analysis of the characters' gestures. The framework is then illustrated through a close comparative analysis of two scenes from the anime movie. I explain how the relationship between the language and visual semiotic resources based on appraisal analysis changes from convergence between the language and visuals to divergence, which could be correlated with character development. I conclude by discussing how the narrative context influences the semiotic representation of the character Sesshomaru.

Banquet plate and Renaissance culture: a day in the life

The paper discusses the many functions of banquet plate within the spatial and gestural framework of feasting at the Renaissance court. During the Cinquecento, a golden age of Italian gastronomy, an increasing variety and number of silver and gold vessels were used for formal dining, including basins, ewers, food warmers, drinking cups, sauce boats, confections dishes, wine coolers, forks and other accoutrements. Banquets were conceived as spectacles where plate was displayed on tiered sideboards or credenze and also used as tableware that was handled by servants, patrons and guests. At these events, costly table furnishings would be seen in motion against an ongoing show of culinary extravaganzas punctuated by live performances and musical intermezzi. The Gonzaga court of Mantua (1450–1550) offers a rich context for studying how these precious vessels were handled in public during service, as well as behind the scenes, where their movements were carefully regulated in household ordinances. (pp. 621–633)

Seamless interaction in virtual reality

Jot is a novel research interface for virtual reality modeling. This system seamlessly integrates and applies a variety of virtual and physical tools, each customized for specific tasks. The Jot interface not only moves smoothly from one tool to another but also physically and cognitively matches individual tools to the tasks they perform. In particular, we exploit the notion that gestural interaction is more direct, in many cases, than traditional widget based interaction. We also respect the time tested observation that some operations-even conceptually three dimensional ones-are better performed with 1D or 2D input devices, whereas other operations are more naturally performed using stereoscopic views, higher DOF input devices, or both. Ultimately we strive for a 3D modeling system with an interface as transparent as the interaction afforded by a pencil and a sheet of paper. For example, the system should facilitate the tasks of drawing and erasing and provide an easy transition between the two. Jot emerged from our previous work on a mouse based system, called Sketch, for gesturally creating imprecise 3D models. Jot extends Sketch's functionality to a wider spectrum of modeling, from concept design to detailed feature based parametric parts. Jot also extends the interaction in Sketch to better support individual modeling tasks. We extended Sketch's gestural framework to integrate interface components ranging from traditional desktop interface widgets to context sensitive gestures to direct manipulation techniques originally designed for immersive VR.

Sound Change, Phonological Rules, and Articulatory Phonology

Abstract. The gestural model of Articulatory Phonology currently being developed by Browman and Goldstein provides a new way of modelling both synchronic and diachronic phonetic processes as well as certain types of synchronic phonological rules. Although Browman and Goldstein place stringent restrictions on the model, ruling out categorical deletion and insertion of gestures, as well as gestural permutation not resulting from magnitude and timing changes, Articulatory Phonology can nonetheless provide enlightening accounts of various types of sound change, including historical developments which have previously been analysed as segmental insertions and deletions. The application of Articulatory Phonology to sound changes is beneficial in that it allows the formulation of a change to include some account of its motivation from the point of view of the speaker (or indeed, though less straightforwardly, the hearer). We aim to extend Browman and Goldstein's preliminary applications of their model to sound change, by demonstrating that changes which have been analysed as entirely separate developments in a traditional segmental phonology can be seen instead as part of an integrated complex of interrelated changes within Articulatory Phonology. Focussing on the development of non-rhotic varieties of English, we show that the sound changes producing present-day linking [r], which are typically given as three independent developments of Pre-[r] Breaking, Pre-Schwa Laxing, and /r/-Deletion, can be shown to be interdependent and analysed in an explanatory way using the gestural model. However, we argue that not all the synchronic phonological processes to which such sound changes give rise can be analysed in gestural terms, given the current restrictions on Articulatory Phonology. For instance, in present-day English varieties which exhibit intrusive as well as linking [r], and which seem to be best characterised by an [r]-Insertion analysis, synchronic addition of gestures must be permitted. Insertion processes of this sort may initially seem incompatible with Articulatory Phonology, but there is clear motivation to retain the gestural framework, given its ability to model many sound changes, casual speech processes and phonological rules using the same mechanisms. Consequently, we propose that, to account for English [r] and similar cases, the current constraints on Articulatory Phonology must be relaxed to a limited extent at some level of the grammar. We suggest that this might be achieved by integrating the gestural approach into a model of Lexical Phonology.