Gaze Gestures Research Articles

Guiding gaze gestures on smartwatches: Introducing fireworks

Smartwatches enable interaction anytime and anywhere, with both digital and augmented physical objects. However, situations with busy hands can prevent user inputs. To address this limitation, we propose Fireworks, an innovative hands-free alternative that empowers smartwatch users to trigger commands effortlessly through intuitive gaze gestures by providing post-activation guidance. Fireworks allows command activation by guiding users to follow targets moving from the screen center to the edge, mimicking real life fireworks. We present the experimental design and evaluation of two Fireworks instances. The first design employs temporal parallelization, displaying few dynamic targets during microinteractions (e.g., snoozing a notification while cooking). The second design sequentially displays targets to support more commands (e.g., 20 commands), ideal for various scenarios other than microinteractions (e.g., turn on lights in a smart home). Results show that Fireworks’ single straight gestures enable faster and more accurate command selection compared to state-of-the-art baselines, namely Orbits and Stroke. Additionally, participants expressed a clear preference for Fireworks’ original visual guidance.

Study on the brightness and graphical display object directions of the Single-Gaze-Gesture user interface

Single gaze gestures (SGGs), as the basic unit of gaze gestures, interact with graphical display objects (GDOs) on a graphical user interface (GUI) through user saccades to control the system. The brightness design of the interface and saccade direction can significantly affect interaction performance and user experience. Therefore, this study adopts the method of ergonomic experiments combined with subjective evaluation, proposing the design suggestions for the background brightness (BB), brightness contrast (BC), GDO (completion field) directions, and target brightness (TB) of the gaze gesture interface: (1) Background brightness: To ensure the efficiency and experience in actual use, the background of the interface should choose a brightness value slightly lower than 50%; (2) Brightness contrast: At 25% BB, it is advisable to choose a brightness value between 21.5% and 43.5%, and the best choice is 33.25%. Under the condition of BB above 50%, the brightness contrast value of 50% can meet the requirements of the operation task; (3) Target brightness: In the gaze gesture interface design, light color should be used to mark the target; (4) GDO Directions: The GDO should be placed on the upper side of the initiation field (IF), and it is best to be directly above the IF. The contribution of this research is the proposal of an interface design strategy that is beneficial for improving the performance and experience of gaze gesture interaction and can provide future directions and recommendations for interface design.

DynamicRead: Exploring Robust Gaze Interaction Methods for Reading on Handheld Mobile Devices under Dynamic Conditions

Enabling gaze interaction in real-time on handheld mobile devices has attracted significant attention in recent years. An increasing number of research projects have focused on sophisticated appearance-based deep learning models to enhance the precision of gaze estimation on smartphones. This inspires important research questions, including how the gaze can be used in a real-time application, and what type of gaze interaction methods are preferable under dynamic conditions in terms of both user acceptance and delivering reliable performance. To address these questions, we design four types of gaze scrolling techniques: three explicit technique based on Gaze Gesture, Dwell time, and Pursuit; and one implicit technique based on reading speed to support touch-free, page-scrolling on a reading application. We conduct a 20-participant user study under both sitting and walking settings and our results reveal that Gaze Gesture and Dwell time-based interfaces are more robust while walking and Gaze Gesture has achieved consistently good scores on usability while not causing high cognitive workload.

Boosted Gaze Gesture Recognition Using Underlying Head Orientation Sequence

People find it challenging to control smart systems with complex gaze gestures due to the vulnerability of eye saccades. Instead, the existing works achieved good recognition accuracy of simple gaze gestures because of sufficient eye gaze points but simple gaze gestures have limited applications compared to complex gaze gestures. Complex gaze gestures need a composition of multiple subunits of eye fixation to contain a sequence of gaze points that are clustered and rotated with an underlying complex head orientation relationship. This paper proposes a new set of eye gaze points and head orientation angles as new sequences to recognize complex gaze gestures. Eye gaze points and head orientation angles have a powerful influence on gaze gesture formation. The new sequence was obtained by aligning clustered gaze points and head orientation angles with a simple moving average (SMA) to denoise and interpolate the gap between successive eye fixations. The aligned new sequence of complex gaze gestures was utilized to train sequential machine learning (ML) algorithms. To evaluate the performance of the proposed method, we recruited and recorded the eye gaze and head orientation features of ten participants using an eye tracker. The results show that Boosted Hidden Markov Models (HMM) using Random Subspace methods achieved the best accuracies of 94.72% and 98.1% for complex, and simple gestures respectively, which outperformed the conventional methods.

Evaluating the Performance of Machine Learning Algorithms in Gaze Gesture Recognition Systems

Despite the utility of gaze gestures as an input method, there is a lack of guidelines available regarding how to design gaze gestures, what algorithms to use for gaze gesture recognition, and how these algorithms compare in terms of performance. To facilitate the development of applications that leverage gaze gestures, we have evaluated the performance of a combination of template-based and data-driven algorithms on two custom gesture sets that can map to user actions. Template-based algorithms had consistently high accuracies but the slowest runtimes, making them best for small gesture sets or accuracy-critical applications. Data-driven algorithms run much faster and scale better to larger gesture sets, but require more training data to achieve the accuracy of the template-based methods. The main takeaways for gesture set design are 1) gestures should have distinct forms even when performed imprecisely and 2) gestures should have clear key-points for the eyes to fixate onto.

Gaze Buttons Revisited.Looking for an Effective Way of "Clicking" without the Mouse

The fundamental problem for designing a gaze-based human-computer interaction is related to development of an effective method for activating graphical user interface elements by means of gaze only. Such a method should be easy for the user to apply, however at the same time, it requires eye movements that are clearly different from the natural behavior of the eye. We examined three methods of button activation by gaze, looking for the most effective way of gaze "clicking". These were: 1) the most standard method based on the use of dwell-time, 2) its modification based on detection of fixation located inside the buttons area and 3) and the most novel method based on gaze gestures consisting of movement into the button area and outward movement in the approximately opposite direction. We compared these gaze control methods under homogeneous conditions, which allows for a more reliable assessment of their relative usefulness. Two layouts of buttons were used: arranged on a grid, like on a telephone pad, and on a circle with an empty center. The experimental task was to enter a set of four-digit PINs using a set of gaze buttons corresponding to ten digits. A group of novices were instructed to use all the three methods and both button layouts (six experimental conditions). The activation methods were compared in terms of system usability, objectively measured by the PIN entry speed and the number of errors, as well as using a subjective SUS questionnaire. The system based on gaze gestures was worse in both measures; however, it had its followers. The method based on fixation detection instead of dwell-time did not significantly increase the entry speed due to the greater number of errors caused by non-intentional buttons activation. The circle layout turned out to be generally more convenient than the telephone pad layout.

Gaze Gesture Recognition by Graph Convolutional Networks.

Gaze gestures are extensively used in the interactions with agents/computers/robots. Either remote eye tracking devices or head-mounted devices (HMDs) have the advantage of hands-free during the interaction. Previous studies have demonstrated the success of applying machine learning techniques for gaze gesture recognition. More recently, graph neural networks (GNNs) have shown great potential applications in several research areas such as image classification, action recognition, and text classification. However, GNNs are less applied in eye tracking researches. In this work, we propose a graph convolutional network (GCN)–based model for gaze gesture recognition. We train and evaluate the GCN model on the HideMyGaze! dataset. The results show that the accuracy, precision, and recall of the GCN model are 97.62%, 97.18%, and 98.46%, respectively, which are higher than the other compared conventional machine learning algorithms, the artificial neural network (ANN) and the convolutional neural network (CNN).

Influence of Visitors on the Time Budget, Ranging and Strata Use of Lowe's Monkey (Cercopithecus lowei) at Boabeng-Fiema Monkey Sanctuary, Ghana.

Wildlife tourism could be a conservation tool; however, it may disrupt the natural behaviors of wild animals. We examined how wildlife tourism affects Lowe's monkeys (Cercopithecus lowei) at the Boabeng-Fiema Monkey Sanctuary, central Ghana. We examined and compared the time budget, aggression patterns, home range size and strata use of two C. lowei groups-one with a high level of provisioning by visitors/tour guides (HP group) versus one with a low level of provisioning by visitors/tour guides (LP group). We found evidence that the HP group fed less and rested more than the LP group. The HP group significantly increased the time spent feeding and decreased their time resting, but no significant differences were recorded for moving or socializing either in the presence of small or big groups of visitors (i.e., 1-10 visitors or > 10 visitors). In the presence of one to ten visitors, the HP group monkeys increased the use of ground by 22.10% and decreased the use of medium and high tree strata by 15.43% and 11.6%, respectively. Agonistic behaviors (i.e., threat, chase, and attack) were three times higher in the HP group (e.g., open-jawed gaze gesture or head-bobbing). In the presence of visitors, aggression in the HP group increased from 12.81% to 30.18%. The home range size of the C. lowei HP group was smaller (4.68 ha) compared to the LP group (14.25 ha) (i.e., 50.56% difference). The LP group spent significantly less time socializing and more time moving. They fed more on fruits and insects. On average, the daily travel of LP group was 0.58 km more than the HP group (i.e., 22.80% difference), and the former group also spent significantly more time in the mid strata of the canopy. Our results showed that continued provisioning of the monkeys with human foods is detrimental to their natural behavior of the monkeys and could have negative long-term effects on the conservation efforts for the species.

Design and Realization of Gaze Gesture Control System for Flight Simulation

In airplane cockpit, flight crew, display interface, control device and onboard control device constitute a complex human computer interaction system. As a natural interaction way, eye movement can simplify the interaction process of pilots, avoiding mistakes caused by fatigue or distraction. What’s more, since this interactive modality only requires eye movement as input, it can be adapted to some special situations and free the user’s hands. In this paper, we design a gaze gesture interaction system for flight simulation and compare it with the existing prominent interaction methods.

Gestatten: Estimation of User's Attention in Mobile MOOCs From Eye Gaze and Gaze Gesture Tracking

The rapid proliferation of Massive Open Online Courses (MOOC) has resulted in many-fold increase in sharing the global classrooms through customized online platforms, where a student can participate in the classes through her personal devices, such as personal computers, smartphones, tablets, etc. However, in the absence of direct interactions with the students during the delivery of the lectures, it becomes difficult to judge their involvements in the classroom. In academics, the degree of student's attention can indicate whether a course is efficacious in terms of clarity and information. An automated feedback can hence be generated to enhance the utility of the course. The precision of discernment in the context of human attention is a subject of surveillance. However, visual patterns indicating the magnitude of concentration can be deciphered by analyzing the visual emphasis and the way an individual visually gesticulates, while contemplating the object of interest. In this paper, we develop a methodology called Gestsatten which captures the learner's attentiveness from his visual gesture patterns. In this approach, the learner's visual gestures are tracked along with the region of focus. We consider two aspects in this approach -- first, we do not transfer learner's video outside her device, so we apply in-device computing to protect her privacy; second, considering the fact that a majority of the learners use handheld devices like smartphones to observe the MOOC videos, we develop a lightweight approach for in-device computation. A three level estimation of learner's attention is performed based on these information. We have implemented and tested Gestatten over 48 participants from different age groups, and we observe that the proposed technique can capture the attention level of a learner with high accuracy (average absolute error rate is 8.68%), which meets her ability to learn a topic as measured through a set of cognitive tests.

Gaze Touch Cross PIN: Secure Multimodal Authentication using Gaze and Touch PIN

The basic goal of information security is, to protect the privacy, reliability, and availability of information on devices that manipulate and store the information. To protect this information, the fundamental step is user authentication. The most common method for authentication on devices is the personal identification number (PIN) method, which is vulnerable to shoulder surfing attack. Shoulder surfing attack used by attacker especially in the crowded public places. For shoulder surfing attack prevention several methods had been proposed. This paper proposed a GazeTouchCrossPIN authentication method that overcome the limitations found in the earlier work. we propose a multimodal authentication system that combines between the gaze gesture and touch PIN authentication systems. The results illustrate that the proposed GazeTouchCrossPIN method is more secure hence it decreases the shoulder surfing rate in both side attacks and iterative attacks.

Joint visual attention and locomotor experience: A longitudinal study of infants with treated idiopathic clubfoot

AbstractThe aim of this study was to examine the link between joint attention skills and locomotor experiences in infants with locomotor delay due to congenital idiopathic clubfoot compared to typically developing infants. Using a following of point and gaze gesture task, we repeatedly tested joint attention skills of 11 infants with congenital idiopathic clubfoot and 14 typically developing infants at the ages of 6, 9, and 12 months. Additionally, we tested all typically developing infants and eight infants treated for idiopathic clubfoot at onset of crawling on hand and knees and after they had been crawling for about 5 to 6 weeks. Our results confirmed a delayed onset of locomotion in infants treated for idiopathic clubfoot; however, our results did not indicate lower joint attention performances in those infants in general. Moreover, we found a positive effect of crawling on the development of joint attention skills in typically developing infants; however, not in infants treated for idiopathic clubfoot. Our results complement previous work by indicating that locomotor experience, especially crawling is an important, however not necessary agent for developing joint attention skills.Highlights We repeatedly tested the link between crawling experience and joint attention skills in typically developing and infants with locomotor delay due to congenital idiopathic clubfoot. We did not find poorer joint attention performances in clubfoot infants, however crawling facilitated the development of joint attention skills in typically developing infants. Results indicated that locomotor experience, especially crawling is an important, however not necessary agent for developing joint attention skills.

Design and evaluation of an eye tracking support system for the scrub nurse.

The availability of an increasing number of medical devices in the digital operating room has led to increased interaction demands of the surgical staff. To counteract the risk of bacterial contamination induced by device interactions, touchless interaction techniques are required. Support systems based on eye tracking enable interaction while maintaining sterility and freeing the hands to manipulate surgical instruments. A system using eye tracking glasses was developed. In an evaluation, participants completed tasks using gaze gestures. Three use cases were evaluated in an intraoperative setup. System performance, user acceptance, and workload were measured. The system was evaluated in a laboratory environment with 26 participants. The precision of the gaze gesture recognition is 97.9%, and the true positive rate is 98.5%. The participants rated the system useful and were satisfied. Touchless interaction ensures sterility, although the increasing availability of medical devices in the operating room.

Network Anomaly Analysis using the Microsoft HoloLens

We investigate the feasibility of using Microsoft HoloLens, a mixed reality device, to visually analyze network capture data and locate anomalies. We developed MINER, a prototype application to visualize details from network packet captures as 3D stereogram charts. MINER employs a novel approach to time-series visualization that extends the time dimension across two axes, thereby taking advantage of the immersive 3D space available via the HoloLens. Users navigate the application through eye gaze and hand gestures to view summary and detailed bar graphs. Callouts display additional detail based on the user’s immediate gaze. In a user study, volunteers used MINER to locate network attacks in a dataset from the 2013 VAST Challenge. We compared the time and effort with a similar test using traditional tools on a desktop computer. Our findings suggest that network anomaly analysis with the HoloLens achieved comparable effectiveness, efficiency and satisfaction. We describe user metrics and feedback collected from these experiments; lessons learned and suggested future work.

Gaze-based interaction: A 30 year retrospective

Gaze-based interaction is reviewed, categorized within a taxonomy that splits interaction into four forms, namely diagnostic (off-line measurement), active (selection, look to shoot), passive (foveated rendering, a.k.a. gaze-contingent displays), and expressive (gaze synthesis). Diagnostic interaction is the mainstay of eye-tracked applications, including training or assessment of expertise, and is possibly the longest standing use of gaze due to its mainly offline requirements. Diagnostic analysis of gaze is still very much in demand, especially in training situations such as flight or surgery training. Active interaction is rooted in the desire to use the eyes to point and click, with gaze gestures growing in popularity. Passive interaction is the manipulation of scene elements in response to gaze direction, e.g., to improve frame rate. Expressive eye movement is drawn from its synthesis, which can make use of a procedural (stochastic) model of eye motion driven by goal-oriented tasks such as reading. In discussing each form of interaction, seminal results and recent advancements are reviewed, highlighting outstanding research problems. The survey paper extends an invited proceedings contribution to VS-Games 2017.

Using embodied pedagogical agents in collaborative learning by knowledge integration: An experimental investigation on instruction and gaze gestures using eye movement analysis

Using embodied pedagogical agents in collaborative learning by knowledge integration: An experimental investigation on instruction and gaze gestures using eye movement analysis

Gaze gesture based human robot interaction for laparoscopic surgery.

While minimally invasive surgery offers great benefits in terms of reduced patient trauma, bleeding, as well as faster recovery time, it still presents surgeons with major ergonomic challenges. Laparoscopic surgery requires the surgeon to bimanually control surgical instruments during the operation. A dedicated assistant is thus required to manoeuvre the camera, which is often difficult to synchronise with the surgeon's movements. This article introduces a robotic system in which a rigid endoscope held by a robotic arm is controlled via the surgeon's eye movement, thus forgoing the need for a camera assistant. Gaze gestures detected via a series of eye movements are used to convey the surgeon's intention to initiate gaze contingent camera control. Hidden Markov Models (HMMs) are used for real-time gaze gesture recognition, allowing the robotic camera to pan, tilt, and zoom, whilst immune to aberrant or unintentional eye movements. A novel online calibration method for the gaze tracker is proposed, which overcomes calibration drift and simplifies its clinical application. This robotic system has been validated by comprehensive user trials and a detailed analysis performed on usability metrics to assess the performance of the system. The results demonstrate that the surgeons can perform their tasks quicker and more efficiently when compared to the use of a camera assistant or foot switches.

Investigation of continual operation procedures via a user centered gaze control by means of flexible gaze gestures

This work presents the pursued research results of a study about the continual control of virtual control elements by gaze, which was released in the proceedings of the spring congress 2017 arranged by the Society of Ergonomics and Work Science (German: GfA). The motivation to provide a more intuitive and robust gaze interaction method leads to the usage of specific gaze gestures. These gestures are based on unique gaze patterns which are distinct from typical intended as well as unintended viewing. For enabling extensive interactions through those gestures a gesture set was developed and tested in a first attempt in terms of its self-descriptiveness and learnability. The second objective of this study arises from the necessity to provide the user with vital information contents about his operation possibilities during the interaction by gaze gestures. By means of the design parameterization two different interfaces were tested concerning its design parameter shape. The objective was to come to first conclusions about the guiding method of representing specific visual interface elements to establish a satisfying gaze control. The results show that the applied gesture set can be learned quite easily and helps the user to connect the symbolic of the gesture with the corresponding execution. Furthermore, the experimental results give evidence of the preference of reference points (grid points) as the guiding elements rather than a polyline (square) shape in the gaze-gesture-controlled interface Practical Relevance: The research of gaze control is on the cutting edge. The control by gestures, for example through touch operation, has asserted themselves in various fields of application. These methods were employed to promote trends like autonomous driving, service design and individualization. Powerful and at the same time cost efficient eye tracking systems favor the extensive application of this technology beyond the recent research fields of medical diagnostic and individual interaction solutions for handicapped people. Furthermore, gaze control has great potential to help the user in fields like teleoperation, complex interfaces and cockpits as well as in the public service sector. Consequently it constitutes an innovative, intuitive and inspiring way of human system interaction which could also be useful combined with other interaction modalities. Besides all these arising opportunities in the mentioned fields, there is a big necessity to conscientiously concern the advantages and disadvantages of this technology. According to the research which has been accomplished in other, more established fields of interaction a well-defined design guideline, both for the gaze-controlled interface and the applied control characteristic is inevitable. The presented work depicts another step towards this approach.

Vibrotactile stimulation of the head enables faster gaze gestures

Gaze gestures are a promising input technology for wearable devices especially in the smart glasses form factor because gaze gesturing is unobtrusive and leaves the hands free for other tasks. We were interested in how gaze gestures can be enhanced with vibrotactile feedback. We studied the effects of haptic feedback on the head and haptic prompting on the speed of completing gaze gestures. The vibrotactile stimulation was given to the skin of the head through actuators in a sun glass frame. The haptic feedback enabled about 10% faster gaze gestures with more consistent completion times. Longer duration of haptic prompts tended to result in longer duration of gestures. However, the magnitude of the increase was marginal. Our results can inform the design of efficient gaze gesture user interfaces and recognition algorithms. HighlightsThe gaze gestures are completed faster when theźhaptic feedback is given.Haptic feedback reduces the variation in gaze gesture completion times.The duration of the haptic prompt affects the duration of the following gaze gesture.

Taxi Drivers’ Gestures in Jordan: A Sociolinguistic Approach

Taxi drivers in Jordan employ many strategies to communicate while driving using both verbal and nonverbal systems of communication. Nonverbal communication may involve the use of blinkers, headlights, eye gaze, head and hand gestures. The present study examined an aspect of taxi drivers’ interpersonal communicative behaviour while driving , namely, use of gestures providing a sociolinguistic analysis of these gestures and taking into consideration the sociolinguistic factors and dimensions that determine their choice. The study was conducted on a random sample consisting of 100 male taxi drivers in Amman, the capital city of Jordan. During the first phase, informants were asked to fill in a questionnaire on the gestures they use while driving or during their breaks. The second phase involved interviews with ten informants. Finally, three informants were videotaped during their breaks. The drivers’ responses were then classified into groups according to age, education and experience in taxi driving. Interviews were then transcribed and analyzed in light of the questions of the study and the gestures used by the videotaped drivers were analyzed. The study revealed that age and education of taxi drivers affected the type and frequency of gestures used. Young drivers tended to gesture more while old drivers considered gesturing as an immature linguistic behavior. Old drivers with university education gestured less while drivers with limited school education gestured more. Further research is recommended on identifying the role gender plays in the use of gestures as well as other verbal and nonverbal communicative features of this slice of society DOI: 10.5901/mjss.2016.v7n3s1p312