Gesture System Research Articles

Developing a 3D Gestural Interface for Anesthesia-Related Human-Computer Interaction Tasks Using Both Experts and Novices.

The purpose of this research was to compare gesture-function mappings for experts and novices using a 3D, vision-based, gestural input system when exposed to the same context of anesthesia tasks in the operating room (OR). 3D, vision-based, gestural input systems can serve as a natural way to interact with computers and are potentially useful in sterile environments (e.g., ORs) to limit the spread of bacteria. Anesthesia providers' hands have been linked to bacterial transfer in the OR, but a gestural input system for anesthetic tasks has not been investigated. A repeated-measures study was conducted with two cohorts: anesthesia providers (i.e., experts) ( N = 16) and students (i.e., novices) ( N = 30). Participants chose gestures for 10 anesthetic functions across three blocks to determine intuitive gesture-function mappings. Reaction time was collected as a complementary measure for understanding the mappings. The two gesture-function mapping sets showed some similarities and differences. The gesture mappings of the anesthesia providers showed a relationship to physical components in the anesthesia environment that were not seen in the students' gestures. The students also exhibited evidence related to longer reaction times compared to the anesthesia providers. Domain expertise is influential when creating gesture-function mappings. However, both experts and novices should be able to use a gesture system intuitively, so development methods need to be refined for considering the needs of different user groups. The development of a touchless interface for perioperative anesthesia may reduce bacterial contamination and eventually offer a reduced risk of infection to patients.

Extraction of Hand Gesture Features for Indian Sign languages using Combined DWT-DCT and Local Binary Pattern

The Hand Gesture system is based on two modes, viz, Enrollment mode and Recognition mode. In the enrollment mode, the Hand features are acquired from the camera and stored in a database along with the Sign languages. In the recognition mode, the hand features are re-acquired from the camera and compared against the stored Indian sign language data to determine the exact signs. In the pre-processing stage, two segmentation processes are proposed to extract the region of interest (ROI) of hand gesture. The first skin-color segmentation is used to extract the hand image from the background. The second region of interest of the hand gesture is segmented by using the valley detection algorithm. The Discrete Wavelet Transform (DWT) and Discrete Cosine Transform (DCT) are applied for the purpose of extracting the features. Further, the Sobel Operator and Local Binary Pattern (LBP) are used for increasing the number of features. The mean and standard deviation of DWT, DCT and LBP are computed.

Gesture-controlled musical performance: from movement awareness to mastery

ABSTRACTTechnical advances in movement sensing and processing offer musicians greater scope to perform in ways that reflect their individual body signature. However, there is currently an asymmetry between innovations in gestural system design and the development of movement skills and awareness among performers. In particular, stagecraft for musicians to master and perform effectively with gesture-controlled instruments is underdeveloped. Few formal guidelines exist to equip musicians with the physical skills to calibrate whole body movements spatially in order to achieve precise and nuanced control of sound when using non-tactile controllers. This paper firstly examines the relevance of embodied interaction design approaches and movement awareness disciplines in addressing this challenge. An outline of the work Bodyscapes follows, composed for a novel gestural instrument, the Telechord, voice and drums. The piece explores the influence of metaphor-based mapping strategies on gesture-controlled performance experiences and skills. Analysis of autoethnographic insights that emerge from the work illuminates ways in which musicians inhabit their bodies to cultivate new forms of musicianship when performing with gestural instruments, drawing on techniques that include movement improvisation and kinaesthetic imagery.

Integrity and function of gestures in aphasia

ABSTRACTBackground: Gestures can provide an excellent natural alternative to verbal communication in people with aphasia (PWA). However, despite numerous studies focusing on gesture production in aphasia, it is still a matter of debate whether the gesture system remains intact after language impairment and how PWA use gestures to improve communication. A likely source for the contradicting results is that many studies were conducted on individual cases or in heterogeneous groups of individuals with additional cognitive deficits such as conceptual impairment and comorbid conditions such as limb apraxia.Aims: The goal of the current study was to evaluate the integrity and function of gestures in PWA in light of cognitive theories of language–gesture relationship. Since all such theories presuppose the integrity of the conceptual system, and the absence of comorbid conditions that selectively impair gesturing (i.e., limb apraxia), our sample was selected to fulfill these assumptions.Methods & Procedures: We examined gesture production in eight PWA with preserved auditory comprehension, no comorbidities, and various degrees of expressive deficit, as well as 11 age- and education-matched controls, while they described events in 20 normed video clips. Both speech and gesture data were coded for quantitative measures of informativeness, and gestures were grouped into several functional categories (matching, complementary, compensatory, social cueing, and facilitating lexical retrieval) based on correspondence to the accompanying speech. Using rigorous group analyses, individual-case analyses, and analyses of individual differences, we provide converging evidence for the integrity and type of function(s) served by gesturing in PWA.Outcomes & Results: Our results indicate that the gesture system can remain functional even when language production is severely impaired. Our PWA heavily relied on iconic gestures to compensate for their language impairment, and the degree of such compensation was correlated with the extent of language impairment. In addition, we found evidence that producing iconic gestures was related to higher success rates in resolving lexical retrieval difficulties.Conclusions: When comprehension and comorbidities are controlled for, impairment of language and gesture systems is dissociable. In PWA with good comprehension, gesturing can provide an excellent means to both compensate for the impaired language and act as a retrieval cue. Implications for cognitive theories of language–gesture relationship and therapy are discussed.

Linguistically deprived children: meta-analysis of published research underlines the importance of early syntactic language use for normal brain development

We analyzed all published reports of individuals not exposed to syntactic language until puberty: two feral children, who grew up without hearing any language, and eight deaf linguistic isolates, who grew up communicating to their families using homesign or kitchensign, a system of gestures which allows them to communicate simple commands but lacks much in the way of syntax. A common observation in these individuals is the lifelong difficulty understanding syntax and spatial prepositions, even after many years of rehabilitation. This debilitating condition stands in stark contrast to linguistic isolates’ performance on memory as well as semantic tests: they could easily remember hundreds of newly learned words and identify previously seen objects by name. The lack of syntactic language comprehension in linguistic isolates may stem from inability to understand words and/or grammar or inability to mentally synthesize known objects into novel configurations. We have previously shown that purposeful construction of novel mental images is the function of the lateral prefrontal cortex (LPFC) ability to dynamically control posterior cortex neurons (Vyshedskiy et al. 2017). Here we have ranked all tests performed on linguistic isolates by their reliance on the LPFC control of the posterior cortex: a) the amount of posterior cortex territory that needs to be recruited by the LPFC and b) the number of disparate objects that have to be combined together by the LPFC in order to answer the test question. According to our analysis, linguistic isolates performed well in all tests that did not involve the LPFC control of the posterior cortex, showed decreasing scores in tests that involved greater recruitment of the posterior cortex by the LPFC, and failed in tests that involved greatest recruitment of posterior cortex necessary for mental synthesis of multiple objects. This pattern is consistent with inadequate frontoposterior connections in linguistic isolates. We discuss implications of these findings for the importance of early syntactic language exposure in formation of frontoposterior connections.

Development of a gesture and voice controlled system for burn injury prevention in individuals with disabilities.

Burn injury is a major public health issue in developing countries, with most injuries being largely associated with the use of kitchen stoves. This study details the development of a cost-effective gesture and voice recognition controlled (GVC) system to be used by individuals with disabilities to reduce the likelihood of burn injury and improve their quality of life. The device replaces conventional dial controls with voice and hand gesture recognition sensors and software which are designed to be easily implemented into a household kitchen. Preliminary evaluation of the GVC system's performance in gesture and voice recognition, gas leak detection and ignition control tests were conducted using a Bunsen burner as a stove top surrogate. The voice and gesture recognition tests yielded sensitivities of 88% and 100%, respectively. These results suggest that the GVC system may be a promising solution for burn injury prevention pending further work to improve its reliability and robustness.

Factors affecting accuracy and intelligibility of transliterators who use cued speech

Some deaf individuals access spoken information via transliterators who use Cued Speech, a system of hand gestures that supplement information available through speechreading alone. In this presentation, the accuracy and intelligibility of 12 transliterators with varying degrees of experience are examined. Accuracy, or the percentage of cues correctly produced, was evaluated at three different speaking rates (slow, normal, and fast), and intelligibility, or the percentage of words correctly received, was measured by presenting the materials that the transliterators produced to nine expert receivers of Cued Speech. Results show that speaking rate had a large negative effect on accuracy, caused primarily by an increase in omitted cues, while increased experience level was generally associated with increased accuracy. Intelligibility was generally higher than accuracy, with accuracy accounting for roughly 25% of the variance in intelligibility scores. We conclude by discussing factors such as speechreadability that could explain additional portions of the variance.

Advanced information processing of MEMS motion sensors for gesture interaction

Abstract. Sensor-based gesture interaction technology has been widely adopted in consumer electronics. Nevertheless, bias, drift, and noise existing in sensor signals are difficult to eliminate, and accurate movement trajectory information is still needed to achieve flexible interaction application. This paper presents micro-electro-mechanical system (MEMS) motion sensor information processing algorithms designed on a gesture interaction system which integrates multiple low-cost MEMS motion sensors with ZigBee wireless technology to support embodied communication while acting together with machines. Sensor signal processing systems mainly solve noise removal, signal smoothing, gravity influence separation, coordinate system conversion, and position information retrieval. The attitude information which is an important movement parameter and required by position estimation is calculated with a quaternion-based extended Kalman filter (EKF). The effectiveness of the movement information retrieval of this gesture interface is verified by experiments and test analysis, both in static and moving cases. In the end, related applications of the described sensor information processing are discussed.

The effect of being human and the basis of grammatical word order: Insights from novel communication systems and young sign languages

This study identifies a central factor that gives rise to the different word orders found in the world’s languages. In the last decade, a new window on this long-standing question has been provided by data from young sign languages and invented gesture systems. Previous work has assumed that word order in both invented gesture systems and young sign languages is driven by the need to encode the semantic/syntactic roles of the verb’s arguments. Based on the responses of six groups of participants, three groups of hearing participants who invented a gestural system on the spot, and three groups of signers of relatively young sign languages, we identify a major factor in determining word order in the production of utterances in novel and young communication systems, not suggested by previous accounts, namely the salience of the arguments in terms of their human/animacy properties: human arguments are introduced before inanimate arguments (‘human first’). This conclusion is based on the difference in word order patterns found between responses to depicted simple events that vary as to whether both subject and object are human or whether the subject is human and the object inanimate. We argue that these differential patterns can be accounted for uniformly by the ‘human first’ principle. Our analysis accounts for the prevalence of SOV order in clauses with an inanimate object in all groups (replicating results of previous separate studies of deaf signers and hearing gesturers) and the prevalence of both SOV and OSV in clauses with a human object elicited from the three groups of participants who have the least interference from another linguistic system (nonliterate deaf signers who have had little or no exposure to another language). It also provides an explanation for the basic status of SOV order suggested by other studies, as well as the scarcity of the OSV order in languages of the world, despite its appearance in novel communication systems. The broadest implication of this study is that the basic cognitive distinction between humans and inanimate entities is a crucial factor in setting the wheels of word ordering in motion.

Successful communication does not drive language development: Evidence from adult homesign

Constructivist accounts of language acquisition maintain that the language learner aims to match a target provided by mature users. Communicative problem solving in the context of social interaction and matching a linguistic target or model are presented as primary mechanisms driving the language development process. However, research on the development of homesign gesture systems by deaf individuals who have no access to a linguistic model suggests that aspects of language can develop even when typical input is unavailable. In four studies, we examined the role of communication in the genesis of homesign systems by assessing how well homesigners’ family members comprehend homesign productions. In Study 1, homesigners’ mothers showed poorer comprehension of homesign descriptions produced by their now-adult deaf child than of spoken Spanish descriptions of the same events produced by one of their adult hearing children. Study 2 found that the younger a family member was when they first interacted with their deaf relative, the better they understood the homesigner. Despite this, no family member comprehended homesign productions at levels that would be expected if family members co-generated homesign systems with their deaf relative via communicative interactions. Study 3 found that mothers’ poor or incomplete comprehension of homesign was not a result of incomplete homesign descriptions. In Study 4 we demonstrated that Deaf native users of American Sign Language, who had no previous experience with the homesigners or their homesign systems, nevertheless comprehended homesign productions out of context better than the homesigners’ mothers. This suggests that homesign has comprehensible structure, to which mothers and other family members are not fully sensitive. Taken together, these studies show that communicative problem solving is not responsible for the development of structure in homesign systems. The role of this mechanism must therefore be re-evaluated in constructivist theories of language development.

Language from the Ground Up: A Study of Homesign Communication

Philosophers are often beholden to a picture of language as a largely static, well-defined structure which is handed over from generation to generation by an arduous process of learning: language, on this view, is something that we are given, and that we can make use of, but which we play no significant role in creating ourselves. This picture is often maintained in conjunction with the idea that several distinctively human cognitive capacities could only develop via the language acquisition process, as thus understood. This paper argues that the phenomenon of homesign, i.e., spontaneous gesture systems devised by deaf children for the purpose of communicating with their non-signing peers, can shed valuable empirical light on these convictions. Contrary to grounding assumptions of Wittgensteinian, Gricean, and Peircean approaches to language, homesign shows how core properties of language—including semantic properties—can be built from the ground up in idiosyncratic ways to serve the communicative needs of individuals.

Visible movements of the orofacial area

The age-old debate between the proponents of the gesture-first and speech-first positions has returned to occupy a central place in current language evolution theorizing. The gestural scenarios, suffering from the problem known as “modality transition” (why a gestural system would have changed into a predominantly spoken system), frequently appeal to the gestures of the orofacial area as a platform for this putative transition. Here, we review currently available evidence on the significance of the orofacial area in language evolution. While our review offers some support for orofacial movements as an evolutionary “bridge” between manual gesture and speech, we see the evidence as far more consistent with a multimodal approach. We also suggest that, more generally, the “gestural versus spoken” formulation is limiting and would be better expressed in terms of the relative input and interplay of the visual and vocal-auditory sensory modalities.

Reaching and pointing gestures calculated by a generic gesture system for social robots

Since the implementation of gestures for a certain robot generally involves the use of specific information about its morphology, these gestures are not easily transferable to other robots. To cope with this problem, we proposed a generic method to generate gestures, constructed independently of any configuration and therefore useable for different robots. In this paper, we discuss the novel end-effector mode of the method, which can be used to calculate gestures whereby the position of the end-effector is important, for example for reaching for or pointing towards an object. The interesting and innovative feature of our method is its high degree of flexibility in both the possible configurations wherefore the method can be used, as in the gestures to be calculated. The method was validated on several configurations, including those of the robots ASIMO, NAO and Justin. In this paper, the working principles of the end-effector mode are discussed and a number of results are presented.

Effects of auditory, haptic and visual feedback on performing gestures by gaze or by hand

ABSTRACTModern interaction techniques like non-intrusive gestures provide means for interacting with distant displays and smart objects without touching them. We were interested in the effects of feedback modality (auditory, haptic or visual) and its combined effect with input modality on user performance and experience in such interactions. Therefore, we conducted two exploratory experiments where numbers were entered, either by gaze or hand, using gestures composed of four stroke elements (up, down, left and right). In Experiment 1, a simple feedback was given on each stroke during the motor action of gesturing: an audible click, a haptic tap or a visual flash. In Experiment 2, a semantic feedback was given on the final gesture: the executed number was spoken, coded by haptic taps or shown as text. With simultaneous simple feedback in Experiment 1, performance with hand input was slower but more accurate than with gaze input. With semantic feedback in Experiment 2, however, hand input was only slower. Effects of feedback modality were of minor importance; nevertheless, semantic haptic feedback in Experiment 2 showed to be useless at least without extensive training. Error patterns differed between both input modes, but again not dependent on feedback modality. Taken together, the results show that in designing gestural systems, choosing a feedback modality can be given a low priority; it can be chosen according to the task, context and user preferences.

Basic Hand Gestures Classification Based on Surface Electromyography.

This paper presents an innovative classification system for hand gestures using 2-channel surface electromyography analysis. The system developed uses the Support Vector Machine classifier, for which the kernel function and parameter optimisation are conducted additionally by the Cuckoo Search swarm algorithm. The system developed is compared with standard Support Vector Machine classifiers with various kernel functions. The average classification rate of 98.12% has been achieved for the proposed method.

Gesture, sign, and language: The coming of age of sign language and gesture studies.

How does sign language compare with gesture, on the one hand, and spoken language on the other? Sign was once viewed as nothing more than a system of pictorial gestures without linguistic structure. More recently, researchers have argued that sign is no different from spoken language, with all of the same linguistic structures. The pendulum is currently swinging back toward the view that sign is gestural, or at least has gestural components. The goal of this review is to elucidate the relationships among sign language, gesture, and spoken language. We do so by taking a close look not only at how sign has been studied over the past 50 years, but also at how the spontaneous gestures that accompany speech have been studied. We conclude that signers gesture just as speakers do. Both produce imagistic gestures along with more categorical signs or words. Because at present it is difficult to tell where sign stops and gesture begins, we suggest that sign should not be compared with speech alone but should be compared with speech-plus-gesture. Although it might be easier (and, in some cases, preferable) to blur the distinction between sign and gesture, we argue that distinguishing between sign (or speech) and gesture is essential to predict certain types of learning and allows us to understand the conditions under which gesture takes on properties of sign, and speech takes on properties of gesture. We end by calling for new technology that may help us better calibrate the borders between sign and gesture.

Sistem Gesture Accelerometer dengan Metode Fast Dynamic Time Warping (FastDTW)

In the modern environment, the interaction between humans and computers require a more natural form of interaction. Therefore, it is important to be able to build a system that can meet these demands, such as by building a hand gesture recognition system or gesture to create a more natural form of interaction. This study aims to design a smartphone’s accelerometer gesture system as human computer interaction interfaces using FastDTW (Fast Dynamic Time Warping).The result of this study is form of gesture interaction which implemented in a system that can make the process of recognition of the human hand movements based on a smartphone accelerometer which generates a command to run the media player application functions as a case study. FastDTW as the development of Dynamic Time Warping method (DTW) is able to compute faster than DTW and have an accuracy approaching DTW. From the test results, FastDTW show a fairly high degree of accuracy reached 86% and showed a better computing speed compared to DTW

Phonological similarity affects production of gestures, even in the absence of overt speech.

Highlights Does phonological similarity affect gesture production in the absence of speech?Participants produced gestures from pictures with no words presented or spoken.Same pictures and gestures but different training labels were used.Phonologically similar labels led to more errors in subsequent gestures.Thus, phonological similarity affects gesture production in the absence of speech.Are manual gestures affected by inner speech? This study tested the hypothesis that phonological form influences gesture by investigating whether phonological similarity between words that describe motion gestures creates interference for production of those gestures in the absence of overt speech. Participants learned to respond to a picture of a bottle by gesturing to open the bottle's cap, and to a picture of long hair by gesturing to twirl the hair. In one condition, the gestures were introduced with phonologically-similar labels “twist” and “twirl” (similar condition), while in the other condition, they were introduced with phonologically-dissimilar labels “unscrew” and “twirl” (dissimilar condition). During the actual experiment, labels were not produced and participants only gestured by looking at pictures. In both conditions, participants also gestured to a control pair that was used as a baseline. Participants made significantly more errors on gestures in the similar than dissimilar condition after correction for baseline differences. This finding shows the influence of phonology on gesture production in the absence of overt speech and poses new constraints on the locus of the interaction between language and gesture systems.

Effect of armrest positioning and workspace constraints on muscle activity for touchless gestural controls

The recent development of affordable technologies now allow touchless gestural controls to be integrated in common computing environments, however, ergonomic issues (user fatigue, in particular) have not been solved. This experiment investigates shoulder muscle activity for different traditional supported or user-preferred touchless gesture interaction scenarios and quantifies the location that touchless gestural control users would like to position armrests in different gestural workspaces. A custom hand/finger-based gestural control system was developed using a motion tracking system. Users browed an online shopping website while a prototype office chair allowed them to positon armrests easily in a wide space in front of themselves. Results show that median muscle activity of the upper trapezius, medial deltoid, and anterior deltoid are significantly affected by armrest configuration. User-preferred armrest positions generally reduced muscle activity compared to traditional positions, but this reduction was greatest when the gesture space is constrained over the desktop. In those conditions, average user preferred positioning of the armrest was higher and closer to the sternum of the user.

Effect of control-display transfer function on pointing performance for a hand/finger based touchless gestural controls

Recent technologies now allow touchless gestural controls to be integrated as computing input devices. Unlike traditional input devices, the effect of design parameters (such as control-display transfer functions for physical to onscreen movements) for touchless controls is largely unknown. This paper presents an experiment where a hand- and finger-based touchless gestural control system replaced typical pointing and clicking functions of a mouse. The effect of different pointer-acceleration-based control-display transfer functions (dynamic CD gain with different scaling factors) on performance for a standard, 2-dimensional Fitts’ task was evaluated for 15 participants. Additionally, the influence of armrest support positioning was investigated. Results showed significantly increased error rates for the high level of control-display gain (SF=1000) in the gestural controls. Neither CD gain nor armrest position had significant effect on effective throughput or movement times, though lower CD gain showed the best performance and lowest error rates. Overall performance for the touchless gestural system was less than that of a traditional mouse, though comparable to previous gestural or remote pointing systems.