Head Gestures Research Articles

Adaptive Kalman Filter Applied to Vision Based Head Gesture Tracking for Playing Video Games

This paper proposes an adaptive Kalman filter (AKF) to improve the performance of a vision-based human machine interface (HMI) applied to a video game. The HMI identifies head gestures and decodes them into corresponding commands. Face detection and feature tracking algorithms are used to detect optical flow produced by head gestures. Such approaches often fail due to changes in head posture, occlusion and varying illumination. The adaptive Kalman filter is applied to estimate motion information and reduce the effect of missing frames in a real-time application. Failure in head gesture tracking eventually leads to malfunctioning game control, reducing the scores achieved, so the performance of the proposed vision-based HMI is examined using a game scoring mechanism. The experimental results show that the proposed interface has a good response time, and the adaptive Kalman filter improves the game scores by ten percent.

Communication for coordination: gesture kinematics and conventionality affect synchronization success in piano duos

Ensemble musicians often exchange visual cues in the form of body gestures (e.g., rhythmic head nods) to help coordinate piece entrances. These cues must communicate beats clearly, especially if the piece requires interperformer synchronization of the first chord. This study aimed to (1) replicate prior findings suggesting that points of peak acceleration in head gestures communicate beat position and (2) identify the kinematic features of head gestures that encourage successful synchronization. It was expected that increased precision of the alignment between leaders’ head gestures and first note onsets, increased gesture smoothness, magnitude, and prototypicality, and increased leader ensemble/conducting experience would improve gesture synchronizability. Audio/MIDI and motion capture recordings were made of piano duos performing short musical passages under assigned leader/follower conditions. The leader of each trial listened to a particular tempo over headphones, then cued their partner in at the given tempo, without speaking. A subset of motion capture recordings were then presented as point-light videos with corresponding audio to a sample of musicians who tapped in synchrony with the beat. Musicians were found to align their first taps with the period of deceleration following acceleration peaks in leaders’ head gestures, suggesting that acceleration patterns communicate beat position. Musicians’ synchronization with leaders’ first onsets improved as cueing gesture smoothness and magnitude increased and prototypicality decreased. Synchronization was also more successful with more experienced leaders’ gestures. These results might be applied to interactive systems using gesture recognition or reproduction for music-making tasks (e.g., intelligent accompaniment systems).

Multimodality and construction grammar

AbstractThe meaning-making process in face-to-face interaction relies on the integration of meaningful information being conveyed by speech as well as the tone of voice, facial expressions, hand and head gestures, body postures and movements (McNeill 1992; Kendon 2004). Hence, it is inherently multimodal. Usage-based linguistics attributes language use a fundamental role in linguistic theorizing by positing that the language system is grounded in and abstracted from (multimodal) language use. However, despite this inherent epistemological link, usage-based linguists have hitherto conceptualized language as a system of interconnectedverbal, i. e. monomodal units, leaving nonverbal usage aspects and the question of their potential entrenchment as part of language largely out of the picture.This is – at least at first sight – surprising because the usage-based model of Construction Grammar (C × G) seems particularly well-equipped to unite the natural interest of linguists in the units that define language systems with the multimodality of language use. Constructions are conceptualized as holistic “conventionalized clusters of features (syntactic, prosodic, pragmatic, semantic, textual, etc.) that recur as further indivisible associations between form and meaning” (Fried 2015: 974). Given its conceptual openess to all levels of usage features, several studies have recently argued for the need to open up the current focus of C × G towards kinesic recurrences (Günthner & Imo 2006; Deppermann 2011; Deppermann & Proske 2015; Andrén 2010; Schoonjans 2014; Schoonjans et al. 2015; Steen & Turner 2013; Zima 2014a; Zima 2014b, in press; Cienki 2012; Cienki 2015; Mittelberg 2014; Müller & Bressem 2014; Bergs 2015; Valenzuela 2015). Departing from the usage-based foundation of C × G which takes “grammar to be the cognitive organization of one’s experience with language” (Bybee 2006: 219), these studies suggest that the basic units of language, i. e. constructions, may be multimodal in nature.This paper presents some of the current issues for a Multimodal Construction Grammar. The aim is to frame the debate and to briefly summarize some of the discussion’s key issues. The individual papers in the special issue elaborate in more detail on particular points of discussion and/or present empirical case studies.

A Wheelchair Control System Using Human-Machine Interaction: Single-Modal and Multimodal Approaches

Abstract Recent research studies showed that brain-controlled systems/devices are breakthrough technology. Such devices can provide disabled people with the power to control the movement of the wheelchair using different signals (e.g. EEG signals, head movements, and facial expressions). With this technology, disabled people can remotely steer a wheelchair, a computer, or a tablet. This paper introduces a simple, low-cost human-machine interface system to help chaired people to control their wheelchair using several control sources. To achieve this paper’s aim, a laptop was installed on a wheelchair in front of the sitting person, and the 14-electrode Emotiv EPOC headset was used to collect the person’s head impressions from the skull surface. The superficially picked-up signals, containing the brain thoughts, head gestures, and facial emotions, were electrically encoded and then wirelessly sent to a personal computer to be interpreted and then translated into useful control instructions. Using these signals, two wheelchair control modes were proposed: automatic (using single-modal and multimodal approaches) and manual control. The automatic mode controller was accomplished using a software controller (Arduino), whereas a simple hardware controller was used for the manual mode. The proposed solution was designed using wheelchair, Emotiv EPOC EEG headset, Arduino microcontroller, and Processing language. It was then tested by totally chaired volunteers under different levels of trajectories. The results showed that the person’s thoughts can be used to seamlessly control his/her wheelchair and the proposed system can be configured to suit many levels and degrees of disability.

The timing of head movements: The role of prosodic heads and edges.

This study examines the influence of the position of prosodic heads (accented syllables) and prosodic edges (prosodic word and intonational phrase boundaries) on the timing of head movements. Gesture movements and prosodic events tend to be temporally aligned in the discourse, the most prominent part of gestures typically being aligned with prosodically prominent syllables in speech. However, little is known about the impact of the position of intonational phrase boundaries on gesture-speech alignment patterns. Twenty-four Catalan speakers produced spontaneous (experiment 1) and semi-spontaneous head gestures with a confirmatory function (experiment 2), along with phrase-final focused words in different prosodic conditions (stress-initial, stress-medial, and stress-final). Results showed (a) that the scope of head movements is the associated focused prosodic word, (b) that the left edge of the focused prosodic word determines where the interval of gesture prominence starts, and (c) that the speech-anchoring site for the gesture peak (or apex) depends both on the location of the accented syllable and the distance to the upcoming intonational phrase boundary. These results demonstrate that prosodic heads and edges have an impact on the timing of head movements, and therefore that prosodic structure plays a central role in the timing of co-speech gestures.

Eliminating drift of the head gesture reference to enhance Google Glass-based control of an NAO humanoid robot

This article presents a strategy for hand-free control of an NAO humanoid robot via head gesture detected by Google Glass-based multi-sensor fusion. First, we introduce a Google Glass-based robot system by integrating the Google Glass and the NAO humanoid robot, which is able to send robot commands through Wi-Fi communications between the Google Glass and the robot. Second, we detect the operator’s head gestures by processing data from multiple sensors including accelerometers, geomagnetic sensors and gyroscopes. Next, we use a complementary filter to eliminate drift of the head gesture reference, which greatly improves the control performance. This is accomplished by the high-pass filter component on the control signal. Finally, we conduct obstacle avoidance experiments while navigating the robot to validate the effectiveness and reliability of this system. The experimental results show that the robot is smoothly navigated from its initial position to its destination with obstacle avoidance via the Google Glass. This hands-free control system can benefit those with paralysed limbs.

Is nonverbal behavior in patients and interviewers relevant to the assessment of depression and its recovery? A study with Dutch and Brazilian patients

Nonverbal behaviors exhibited by patients with depression in their interactions with others may reflect social maladjustment and depression maintenance. Investigations of associations between unipolar depression and both patients’ and interviewers’ behaviors have been scarce and restricted to European samples. This study examined whether nonverbal behavior in patients and their interviewers is associated with depression severity and recovery. Cultural differences were explored. Seventy-eight depressed outpatients (28 Brazilians, 50 Dutch) were evaluated before and after 8-week pharmacological treatment. Patients were videotaped during the Hamilton Depression Scale interview before treatment, and the Brazilians were also videotaped after treatment. Nonverbal behaviors (patients’ speaking effort and interviewers’ encouragement) were analyzed using a two-factor ethogram. Results revealed that speaking effort was associated with encouragement and both are not influenced by baseline depression severity. However, from before to after treatment, whereas encouragement remained unchanged, speaking effort increased among unrecovered patients. Speaking effort was associated with patients’ culture: Brazilians exhibited higher speaking effort than Dutch. These findings highlight that whereas the supportive nonverbal behavior of the interviewer may be stable, the set of nonverbal behaviors composed by head movements, eye contact and gestures displayed by the patients during their speaking in clinical interviews reflects depression persistence after treatment.

Conversion of Tactile Sign Language into English for Deaf/Dumb Interaction

Natural language is the way of communication for normal human beings which includes spoken language, written and body gestures i.e. head gesture, hand gestures, facial expressions and lip motion etc. On the other hand, speech and hearing-impaired people uses sign language for communication which is not understandable for normal people thus they face problems of communication in society. In this problem interpreters are required but the human interpreters are costly and are not an efficient solution. Thus, there is a need of system which will translate the sign language into normal language which will be understandable by normal. The system proposed and explained in the paper is an efficient solution to this problem. In the system, sign recognition is done using CAMSHIFT and P2DHHM algorithm followed by Haar Cascade Classifier. After sign recognition, the language technology techniques of POS tagging and LALR parser are used to convert recognized sign words into English sentence. Till date no any system has worked on sentence framing. Results shows that this system produces 92% of accurate result which will bridge the gap between impaired and Normal people.

Research on Human-computer Interaction Technology of Wearable Devices Such as Augmented Reality Supporting Grid Work

This study is based on speech recognition and interaction, augmented reality, the movement of the head gesture interaction interactive technology, human-computer interaction system is designed to meet the power equipment installation, operation and maintenance operations, to enhance formation of reality for the show, with gestures and head movements as the main way of interaction, with voice commands and touch as a unified interactive model of secondary interaction the way.

A limited role of hand gestures and head nods in native English speakers’ production of Mandarin tones

Theories of embodied cognition have emphasized the importance of the whole body in human communication (Lakoff & Johnson, 1999), and numerous empirical studies have shown that bodily actions affect multiple aspects of speech production and perception. This study examined the roles that hand and head gestures play in production of Mandarin tones by native English speakers. Twenty-four native English speakers were asked to imitate the four Mandarin tones from a model across three conditions: (1) audio-only, (2) audio and hand gestures, and (3) audio and head nods. For (2) and (3), participants watched videos in which a native Mandarin speaker visually depicted the tones with her hand and head, respectively. The participants’ production was then presented to seven native Mandarin listeners in an identification test to determine percent correctly produced. A 3 (conditions) x 4 (tones) ANOVA revealed a significant main effect of tone (1 > 3, 4 > 2) but no main effect or interaction of condition, indicating that hand and head gestures did not help learners accurately produce Mandarin tones. The results corroborate earlier acoustic analysis of the production data, confirming a limited role of hand and head gestures in phonetic and phonological aspects of L2 learning.

A Smart Wheelchair System with Social Media update

Objective: To develop a smart wheelchair for blind and disabled persons. Methods and Analysis: The wheelchairs being used patients are not user friendly. In modern days, it is difficult for a person to attend a patient throughout the day. The proposed wheelchair can be self-manoeuvred easily by the patient using hand or head gestures. Apart from it, the wheelchair is aided by a voice guided indoor positioning system. Findings: We incorporated an IOT service device so that the status of the patient is regularly updated on a web server. This smart wheelchair is connected to the cell-phone of the guardian as well as the social media profile of the patient so that the patient can get help as soon as there's a mishap. Improvement: The wheelchair has ability to detect obstacles across the path of the patient and it can divert its path from the obstacle so as to avoid collision.

The ALICO corpus: analysing the active listener

The Active Listening Corpus (ALICO) is a multimodal data set of spontaneous dyadic conversations in German with diverse speech and gestural annotations of both dialogue partners. The annotations consist of short feedback expression transcriptions with corresponding communicative function interpretations as well as segmentations of interpausal units, words, rhythmic prominence intervals and vowel-to-vowel intervals. Additionally, ALICO contains head gesture annotations of both interlocutors. The corpus contributes to research on spontaneous human---human interaction, on functional relations between modalities, and timing variability in dialogue. It also provides data that differentiates between distracted and attentive listeners. We describe the main characteristics of the corpus and briefly present the most important results obtained from analyses in recent years.

Gaze perception and awareness in smart devices

Eye contact and gaze awareness play a significant role for conveying emotions and intentions during face-to-face conversation. Humans can perceive each other's gaze quite naturally and accurately. However, the gaze awareness/perception are ambiguous during video teleconferencing performed by computer-based devices (such as laptops, tablet, and smart-phones). The reasons for this ambiguity are the (i) camera position relative to the screen and (ii) 2D rendition of 3D human face i.e., the 2D screen is unable to deliver an accurate gaze during video teleconferencing. To solve this problem, researchers have proposed different hardware setups with complex software algorithms. The most recent solution for accurate gaze perception employs 3D interfaces, such as 3D screens and 3D face-masks. However, today commonly used video teleconferencing devices are smart devices with 2D screens. Therefore, there is a need to improve gaze awareness/perception in these smart devices. In this work, we have revisited the question; how to improve a remote user's gaze awareness among his/her collaborators. Our hypothesis is that 'an accurate gaze perception can be achieved bythe '3D embodiment' of a remote user's head gesture during video teleconferencing'. We have prototyped an embodied telepresence system (ETS) for the 3D embodiment of a remote user's head. Our ETS is based on a 3-DOF neck robot with a mounted smart device (tablet PC). The electromechanical platform in combination with a smart device is a novel setup that is used for studying gaze awareness/perception in 2D screen-based smart devices during video teleconferencing. Two important gaze-related issues are considered in this work; namely (i) 'Mona-Lisa Gaze Effect' - the gaze is always directed at the person independent of his position in the room, and (ii) 'Gaze Awareness/Faithfulness' - the ability to perceive an accurate spatial relationship between the observing person and the object by an actor. Our results confirm that the 3D embodiment of a remote user head not only mitigates the Mona Lisa gaze effect but also supports three levels of gaze faithfulness, hence, accurately projecting the human gaze in distant space. HighlightsHow to improve a gaze perception in 2D screens (tablet PC, iPad) based conferencing?How to curtail Mona-Lisa Gaze Effect in these 2D screen based devices?How to achieve gaze faithfulness (mutual, partial and full) in these devices?

Taxi Drivers’ Gestures in Jordan: A Sociolinguistic Approach

Taxi drivers in Jordan employ many strategies to communicate while driving using both verbal and nonverbal systems of communication. Nonverbal communication may involve the use of blinkers, headlights, eye gaze, head and hand gestures. The present study examined an aspect of taxi drivers’ interpersonal communicative behaviour while driving , namely, use of gestures providing a sociolinguistic analysis of these gestures and taking into consideration the sociolinguistic factors and dimensions that determine their choice. The study was conducted on a random sample consisting of 100 male taxi drivers in Amman, the capital city of Jordan. During the first phase, informants were asked to fill in a questionnaire on the gestures they use while driving or during their breaks. The second phase involved interviews with ten informants. Finally, three informants were videotaped during their breaks. The drivers’ responses were then classified into groups according to age, education and experience in taxi driving. Interviews were then transcribed and analyzed in light of the questions of the study and the gestures used by the videotaped drivers were analyzed. The study revealed that age and education of taxi drivers affected the type and frequency of gestures used. Young drivers tended to gesture more while old drivers considered gesturing as an immature linguistic behavior. Old drivers with university education gestured less while drivers with limited school education gestured more. Further research is recommended on identifying the role gender plays in the use of gestures as well as other verbal and nonverbal communicative features of this slice of society DOI: 10.5901/mjss.2016.v7n3s1p312

Evidence of heterospecific referential communication from domestic horses (Equus caballus) to humans.

Referential communication occurs when a sender elaborates its gestures to direct the attention of a recipient to its role in pursuit of the desired goal, e.g. by pointing or showing an object, thereby informing the recipient what it wants. If the gesture is successful, the sender and the recipient focus their attention simultaneously on a third entity, the target. Here we investigated the ability of domestic horses (Equus caballus) to communicate referentially with a human observer about the location of a desired target, a bucket of food out of reach. In order to test six operational criteria of referential communication, we manipulated the recipient's (experimenter) attentional state in four experimental conditions: frontally oriented, backward oriented, walking away from the arena and frontally oriented with other helpers present in the arena. The rate of gaze alternation was higher in the frontally oriented condition than in all the others. The horses appeared to use both indicative (pointing) and non-indicative (nods and shakes) head gestures in the relevant test conditions. Horses also elaborated their communication by switching from a visual to a tactile signal and demonstrated perseverance in their communication. The results of the tests revealed that horses used referential gestures to manipulate the attention of a human recipient so to obtain an unreachable resource. These are the first such findings in an ungulate species.

HeadBanger: controlling switchable software with head gesture

Purpose – The purpose of this paper is to present a novel non-contact method of using head movement to control software without the need for wearable devices. Design/methodology/approach – A webcam and software are used to track head position. When the head is moved through a virtual target, a keystroke is simulated. The system was assessed by participants with impaired mobility using Sensory Software’s Grid 2 software as a test platform. Findings – The target user group could effectively use this system to interact with switchable software. Practical implications – Physical head switches could be replaced with virtual devices, reducing fatigue and dissatisfaction. Originality/value – Using a webcam to control software using head gestures where the participant does not have to wear any specialised technology or a marker. This system is shown to be of benefit to motor impaired participants for operating switchable software.

Emotional head motion predicting from prosodic and linguistic features

Emotional head motion plays an important role in human-computer interaction (HCI), which is one of the important factors to improve users' experience in HCI. However, it is still not clear how head motions are influenced by speech features in different emotion states. In this study, we aim to construct a bimodal mapping model from speech to head motions, and try to discover what kinds of prosodic and linguistic features have the most significant influence on emotional head motions. A two-layer clustering schema is introduced to obtain reliable clusters from head motion parameters. With these clusters, an emotion related speech to head gesture mapping model is constructed by a Classification and Regression Tree (CART). Based on the statistic results of CART, a systematical statistic map of the relationship between speech features (including prosodic and linguistic features) and head gestures is presented. The map reveals the features which have the most significant influence on head motions in long or short utterances. We also make an analysis on how linguistic features contribute to different emotional expressions. The discussions in this work provide important references for realistic animation of speech driven talking-head or avatar.

Editorial: Understanding Social Signals: How Do We Recognize the Intentions of Others?

Editorial: Understanding Social Signals: How Do We Recognize the Intentions of Others?

Predicting Performance of Collaborative Storytelling Using Multimodal Analysis

This study focuses on modeling the storytelling performance of the participants in a group conversation. Storytelling performance is one of the fundamental communication techniques for providing information and entertainment effectively to a listener. We present a multimodal analysis of the storytelling performance in a group conversation, as evaluated by external observers. A new multimodal data corpus is collected through this group storytelling task, which includes the participants' performance scores. We extract multimodal (verbal and nonverbal) features regarding storytellers and listeners from a manual description of spoken dialog and from various nonverbal patterns, including each participant's speaking turn, utterance prosody, head gesture, hand gesture, and head direction. We also extract multimodal co-occurrence features, such as head gestures, and interaction features, such as storyteller utterance overlapped with listener's backchannel. In the experiment, we modeled the relationship between the performance indices and the multimodal features using machine-learning techniques. Experimental results show that the highest accuracy (R2) is 0.299 for the total storytelling performance (sum of indices scores) obtained with a combination of verbal and nonverbal features in a regression task.

深度センサを用いた座面昇降機能をもつ電動車椅子のハンドフリー操縦システム

Population of elderly people has been increasing not only in Japan but also in many countries. For the purpose of expanding the active field of them and persons with disabilities, and of improving their quality of life (QOL), the development of an intelligent wheelchair has been intensively studied. In this research, a new maneuver interface of the electric wheelchair instead of the joy-stick is considered. To develop such an interface, this paper deals with a simple non-contact detection method of horizontal head gesture motion using a depth sensor.