Head-mounted Eye Tracker Research Articles

Action prediction during real-time parent-infant interactions.

Social interactions provide a crucial context for early learning and cognitive development during infancy. Action prediction-the ability to anticipate an observed action-facilitates successful, coordinated interaction and is an important social-cognitive skill in early development. However, current knowledge about infant action prediction comes largely from screen-based laboratory tasks. We know little about what infants' action prediction skills look like during real-time, free-flowing interactions with a social partner. In the current study, we used head-mounted eyetracking to quantify 9-month-old infants' visual anticipations of their parents' actions during free-flowing parent-child play. Our findings reveal that infants do anticipate their parents' actions during dynamic interactions at rates significantly higher than would be expected by chance. In addition, the frequency with which they do so is associated with child-led joint attention and hand-eye coordination. These findings are the first to reveal infants' action prediction behaviors in a more naturalistic context than prior screen-based studies, and they support the idea that action prediction is inherently linked to motor development and plays an important role in infants' social-cognitive development. A video abstract of this article can be viewed at https://www.youtube.com/watch?v=9HrmcicfiqE.

What are the building blocks of parent-infant coordinated attention in free-flowing interaction?

The present article investigated the composition of different joint gaze components used to operationalize various types of coordinated attention between parents and infants and which types of coordinated attention were associated with future vocabulary size. Twenty-five 9-month-old infants and their parents wore head-mounted eye trackers as they played with objects together. With high-density gaze data, a variety of coordinated attention bout types were quantitatively measured by combining different gaze components, such as mutual gaze, joint object looks, face looks, and triadic gaze patterns. The key components of coordinated attention that were associated with vocabulary size at 12 and 15months included the simultaneous combination of parent triadic gaze and infant object looking. The results from this article are discussed in terms of the importance of parent attentional monitoring and infant sustained attention for language development.

Social origins of self-regulated attention during infancy and their disruption in autism spectrum disorder: Implications for early intervention.

To understand the complex relationships between autism spectrum disorder (ASD) and other frequently comorbid conditions, a growing number of studies have investigated the emergence of ASD during infancy. This research has suggested that symptoms of ASD and highly related comorbid conditions emerge from complex interactions between neurodevelopmental vulnerabilities and early environments, indicating that developing treatments to prevent ASD is highly challenging. However, it also suggests that attenuating the negative effects of ASD on future development once identified is possible. The present paper builds on this by conceptualizing developmental delays in nonsocial skills as the potential product of altered caregiver-infant interactions following the emergence of ASD during infancy. And, following emerging findings from caregiver-infant dyadic head-mounted eye-tracking (D-ET) research, it also suggests that a multiple pathway model of joint attention can provide mechanistic insights into how ASD alters the ability of caregiver and infant to create a context for infant learning. The potential for this view to inform early intervention is further discussed and illustrated through D-ET data collected prior to and following a brief, parent-mediated intervention for infant ASD. While promising, further research informing how a multiple pathway model of joint attention can inform ASD early intervention is needed.

The stability of the MEMS 2D mirror's operating point in terms of eye tracking systems

This paper describes research on the stability of the MEMS mirror for use in eye tracking systems. MEMS mirrors are the main element in scanning methods (which is one of the methods of eye tracking). Due to changes in the mirror pitch, the system can scan the area of the eye with a laser and collect the signal reflected. However, this method works on the assumption that the inclinations are constant in each period. The instability of this causes errors. The aim of this work is to examine the error level caused by pitch instability at different points of work. Full Text: PDF ReferencesW. Fuhl, M. Tonsen, A. Bulling, and E. Kasneci, "Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art," Mach. Vis. Appl., vol. 27, no. 8, pp. 1275-1288, 2016, CrossRef X. Wang, S. Koch, K. Holmqvist, and M. Alexa, "Tracking the gaze on objects in 3D," ACM Trans. Graph., vol. 37, no. 6, pp. 1-18, Dec. 2018 CrossRef X. Xiong and H. Xie, "MEMS dual-mode electrostatically actuated micromirror," Proc. 2014 Zo. 1 Conf. Am. Soc. Eng. Educ. - "Engineering Educ. Ind. Involv. Interdiscip. Trends", ASEE Zo. 1 2014, no. Dmd, 2014 CrossRef E. Pengwang, K. Rabenorosoa, M. Rakotondrabe, and N. Andreff, "Scanning micromirror platform based on MEMS technology for medical application," Micromachines, vol. 7, no. 2, 2016 CrossRef J. P. Giannini, A. G. York, and H. Shroff, "Anticipating, measuring, and minimizing MEMS mirror scan error to improve laser scanning microscopy's speed and accuracy," PLoS One, vol. 12, no. 10, pp. 1-14, 2017 CrossRef C. Hennessey, B. Noureddin, and P. Lawrence, "A single camera eye-gaze tracking system with free head motion," Eye Track. Res. Appl. Symp., vol. 2005, no. March, pp. 87-94, 2005 CrossRef C. H. Morimoto and M. R. M. Mimica, "Eye gaze tracking techniques for interactive applications," Comput. Vis. Image Underst., vol. 98, no. 1, pp. 4-24, Apr. 2005 CrossRef S. T. S. Holmström, U. Baran, and H. Urey, "MEMS laser scanners: A review," J. Microelectromechanical Syst., vol. 23, no. 2, pp. 259-275, 2014 CrossRef C. W. Cho, "Gaze Detection by Wearable Eye-Tracking and NIR LED-Based Head-Tracking Device Based on SVR," ETRI J., vol. 34, no. 4, pp. 542-552, Aug. 2012 CrossRef T. Santini, W. Fuhl, and E. Kasneci, "PuRe: Robust pupil detection for real-time pervasive eye tracking," Comput. Vis. Image Underst., vol. 170, pp. 40-50, May 2018 CrossRef O. Solgaard, A. A. Godil, R. T. Howe, L. P. Lee, Y. A. Peter, and H. Zappe, "Optical MEMS: From micromirrors to complex systems," J. Microelectromechanical Syst., vol. 23, no. 3, pp. 517-538, 2014 CrossRef J. Wang, G. Zhang, and Z. You, "UKF-based MEMS micromirror angle estimation for LiDAR," J. Micromechanics Microengineering, vol. 29, no. 3, 201 CrossRef

The importance of monitoring vergence eye movements for solutions using virtual technologies

Eyesight monitoring systems are of particular interest to virtual technology research teams and concern mainly three key aspects: physiological, technical and functional. The first one is to solve the so-called conflict of vergence and accommodation. In contrast to typical gaming applications, e.g. vision therapy requires the ability to smoothly transfer attention between the points of near and far. The technical aspect is primarily the search for new measurement methods, which is particularly important in the case of mobile goggles with limited computing capabilities of the processor. The last aspect concerns the possibility of creating effective HMI (Human Machine Interface) solutions. Full Text: PDF ReferencesJ. D. Bayliss et. al., "Lazy Eye Shooter: a Novel Game Therapy for Visual Recovery in Adult Amblyopia", 2012 IEEE International Games Innovation Conference CrossRef Y. L. Chen., L. Shi, J. W. L. Lewis, M. Wang, "Infrared Retinoscopy", Photonics 2014, 1, 303-322. CrossRef W. Fuhl, M. Tonsen, A. Bulling, and E. Kasneci, "Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art," Mach. Vis. Appl., vol. 27, no. 8, pp. 1275-1288, 2016 CrossRef N. Sarkar, D. Strathearn, G. Lee, M. Olfat, A. Rohani and R. R. Mansour, "A large angle, low voltage, small footprint micromirror for eye tracking and near-eye display applications", 2015 Transducers - 2015 18th International Conference on Solid-State Sensors, Actuators and Microsystems (TRANSDUCERS), Anchorage, AK, 2015, pp. 855-858, CrossRef

Using Human Gaze to Improve Robustness Against Irrelevant Objects in Robot Manipulation Tasks

Deep imitation learning enables the learning of complex visuomotor skills from raw pixel inputs. However, this approach suffers from the problem of overfitting to the training images. The neural network can easily be distracted by task-irrelevant objects. In this letter, we use the human gaze measured by a head-mounted eye tracking device to discard task-irrelevant visual distractions. We propose a mixture density network-based behavior cloning method that learns to imitate the human gaze. The model predicts gaze positions from raw pixel images and crops images around the predicted gazes. Only these cropped images are used to compute the output action. This cropping procedure can remove visual distractions because the gaze is rarely fixated on task-irrelevant objects. This robustness against irrelevant objects can improve the manipulation performance of robots in scenarios where task-irrelevant objects are present. We evaluated our model on four manipulation tasks designed to test the robustness of the model to irrelevant objects. The results indicate that the proposed model can predict the locations of task-relevant objects from gaze positions, is robust to task-irrelevant objects, and exhibits impressive manipulation performance especially in multi-object handling.

Eye-tracking reveals agency in assisted autistic communication

About one-third of autistic people have limited ability to use speech. Some have learned to communicate by pointing to letters of the alphabet. But this method is controversial because it requires the assistance of another person—someone who holds a letterboard in front of users and so could theoretically cue them to point to particular letters. Indeed, some scientists have dismissed the possibility that any nonspeaking autistic person who communicates with assistance could be conveying their own thoughts. In the study reported here, we used head-mounted eye-tracking to investigate communicative agency in a sample of nine nonspeaking autistic letterboard users. We measured the speed and accuracy with which they looked at and pointed to letters as they responded to novel questions. Participants pointed to about one letter per second, rarely made spelling errors, and visually fixated most letters about half a second before pointing to them. Additionally, their response times reflected planning and production processes characteristic of fluent spelling in non-autistic typists. These findings render a cueing account of participants’ performance unlikely: The speed, accuracy, timing, and visual fixation patterns suggest that participants pointed to letters they selected themselves, not letters they were directed to by the assistant. The blanket dismissal of assisted autistic communication is therefore unwarranted.

Attentional Flexibility Predicts A-Not-B Task Performance in 14-Month-Old-Infants: A Head-Mounted Eye Tracking Study.

Early individual differences in executive functions (EFs) are predictive of a range of developmental outcomes. However, despite the importance of EFs, little is known about the processes underlying these early individual differences. Therefore, we investigated the association between 14-month-old infants’ attention on a reaching version of the A-not-B task and task success. We hypothesized that both strategic focused attention (measured as percentage looking time towards the correct location during delay) and attentional flexibility (measured as number of looks per second to available stimuli during delay) would relate positively to task performance. Infants performed the A-not-B task wearing a head-mounted eye tracker (N = 24). Results were trial-dependent and partially supported the hypotheses: (1) infants who were better able to flexibly shift attention between available stimuli on the first pre-switch trial showed better task performance overall; and (2) strategic focused attention to the hiding location during the first switch trial was positively related to performance on that particular trial only (trend-level effect). Thus, the study shows preliminary evidence that particularly attentional flexibility is a key factor underlying EF performance in young children. Advantages and challenges of working with head-mounted eye tracking in infants are discussed.

Using Head-Mounted Eye-Tracking to Study Handwriting Development

Even with the increasing use of technology in the classroom, handwriting remains a developmental foundation of education. When children fail to learn to write efficiently, they encounter cascading difficulties in using writing to communicate and learn content. Traditionally, the development of handwriting has been studied via neuropsychological testing or the moment-to-moment kinematics of pen movements. By measuring children’s handwriting with neuropsychological testing, investigators have determined that children’s visual-motor integration abilities predict children’s handwriting. Further, by measuring children’s pen movements while writing, investigators have determined that children’s handwriting becomes more fluent as they become skilled writers. Both of these literatures have advanced our understanding of handwriting substantially, but fall short of providing a full account of handwriting development. Here, we offer a perception–action account of handwriting development by describing how eye and hand movements become integrated during early writing. We describe how head-mounted eye-tracking technology can be used to measure children’s eye movements as they write in real-time. We illustrate this approach with findings from research on letter, form, and word copying in school-entry age children. We conclude by discussing how a perception–action approach can be extended for use with atypical populations.

Wearing a head-mounted eye tracker may reduce body sway

This study investigated the effects of wearing a head-mounted eye tracker on upright balance during different visual tasks. Twenty five young adults stood upright on a force plate while performing the visual tasks of fixation, horizontal saccades, and eyes closed, during eighteen trials wearing or not a head-mounted eye tracker. While wearing the eye tracker, participants showed a reduction in mean sway amplitude and velocity of the CoP in the AP and ML directions and more regular CoP fluctuations, in the ML axis in all conditions. Higher mean sway amplitude and velocity of CoP were observed during eyes closed than fixation and saccades. Moreover, horizontal saccades reduced mean sway velocity of CoP compared to fixation. Therefore, wearing the eye tracker minimized the body sway of young adults; however, visual task-related effects on postural stability remained unchanged.

Culture modulates face scanning during dyadic social interactions

Recent studies have revealed significant cultural modulations on face scanning strategies, thereby challenging the notion of universality in face perception. Current findings are based on screen-based paradigms, which offer high degrees of experimental control, but lack critical characteristics common to social interactions (e.g., social presence, dynamic visual saliency), and complementary approaches are required. The current study used head-mounted eye tracking techniques to investigate the visual strategies for face scanning in British/Irish (in the UK) and Japanese adults (in Japan) who were engaged in dyadic social interactions with a local research assistant. We developed novel computational data pre-processing tools and data-driven analysis techniques based on Monte Carlo permutation testing. The results revealed significant cultural differences in face scanning during social interactions for the first time, with British/Irish participants showing increased mouth scanning and the Japanese group engaging in greater eye and central face looking. Both cultural groups further showed more face orienting during periods of listening relative to speaking, and during the introduction task compared to a storytelling game, thereby replicating previous studies testing Western populations. Altogether, these findings point to the significant role of postnatal social experience in specialised face perception and highlight the adaptive nature of the face processing system.

Transition From Crawling to Walking Changes Gaze Communication Space in Everyday Infant-Parent Interaction.

Acquisition of walking changes not only infants' locomotion itself but also infants' exploratory behavior and social interaction, such as gaze communication. To understand the ecological context in which gaze communication occurs and how it changes with walking development from the point of view of the spatial arrangement of infants, parents, and objects, we analyzed longitudinal data of daily eye contact scenes recorded from head-mounted eye trackers worn by parents as infants grew from 10 to 15.5 months, focusing on infant-parent distance and the number of objects between the dyad. A Bayesian state-space model revealed that the interpersonal distance at which infants initiated eye contact with their parents increased with the time ratio of walking to crawling. This result could not be explained by the developmental change in the amount of time that the infants were far from the parents, which is not limited to the gaze communication context. Moreover, the interpersonal distance at which the parents initiated eye contact with the infants did not increase with the time ratio of walking to crawling. The number of objects on the floor between infants and parents at the time of eye contact increased with interpersonal distance. Taken together, these results indicate that the transition from crawling to walking changes the ecological context in which infants initiate gaze communication to a visual environment characterized by a larger interpersonal distance and, therefore, more objects cluttered between the dyad. The present study has wider implications for the developmental change of shared attention in conjunction with walking development.

GlassesViewer: Open-source software for viewing and analyzing data from the Tobii Pro Glasses 2 eye tracker

We present GlassesViewer, open-source software for viewing and analyzing eye-tracking data of the Tobii Pro Glasses 2 head-mounted eye tracker as well as the scene and eye videos and other data streams (pupil size, gyroscope, accelerometer, and TTL input) that this headset can record. The software provides the following functionality written in MATLAB: (1) a graphical interface for navigating the study- and recording structure produced by the Tobii Glasses 2; (2) functionality to unpack, parse, and synchronize the various data and video streams comprising a Glasses 2 recording; and (3) a graphical interface for viewing the Glasses 2’s gaze direction, pupil size, gyroscope and accelerometer time-series data, along with the recorded scene and eye camera videos. In this latter interface, segments of data can furthermore be labeled through user-provided event classification algorithms or by means of manual annotation. Lastly, the toolbox provides integration with the GazeCode tool by Benjamins et al. (2018), enabling a completely open-source workflow for analyzing Tobii Pro Glasses 2 recordings.

Using head-mounted eye-trackers to study sensory-motor dynamics of coordinated attention.

In this chapter, we introduce recent research using head-mounted eye-trackers to record sensory-motor behaviors at a high resolution and examine parent-child interactions at a micro-level. We focus on one important research topic in early social and cognitive development: how young children and their parents coordinate their visual attention in social interactions. We start by introducing head-mounted eye-tracking and recent studies conducted using this method. We then present two sets of novel analysis techniques that examine how manual actions of parents and children with and without hearing loss contribute to their attention coordination. In the first set of analyses, we investigated different pathways parents and children used to coordinate their visual attention in toy play. After that, we used Sankey diagrams to represent the temporal dynamics of parents' and children's manual actions prior to and during coordinated attention. These two sets of analyses allowed us to explore how participants' sensory-motor behaviors contribute to the establishment and maintenance of coordinated attention. More generally, head-mounted eye-tracking allows us to ask new questions and conduct new analyses that were not previously possible. With this new sensing technology, the results here highlight the importance of understanding early social interaction from a multimodal, embodied view.

Road Safety Review update by using innovative technologies to investigate driver behaviour

Urban arterial roads provide high-speed passage to facilitate traffic in urban areas. However, unlike freeways, they consist of ramps, roundabouts and unique characteristics due to the limited space in the urban (or semi-urban) environment. The existing studies use the Road Safety Review (RSR) to evaluate geometry, identifying high accident concentration sections and to classify the network based on expert judgment. Therefore the classical methodology does not consider the interaction between driver and infrastructure. The present study aims to investigate the road safety of an urban arterial motorway, integrating traditional checklist with innovative solutions applied in an experimental site test with participants. The driver visual behaviour has been recorded by head-mounted eye tracker that is used to find the gaze behaviour. The vehicle used for the test was equipped with the satellite positioning system (GPS), inertial measurement unit (IMU), vehicle CAN data reader (OBD2) and video recorder to monitor the driver behaviour and vehicle trajectory during the track. The results showed that the use of innovative techniques could improve the RSR, by identifying new hazardous points based on driver behavior.

ISeeColor: Method for Advanced Visual Analytics of Eye Tracking Data

Recent advances in head-mounted eye-tracking technology have allowed researchers to monitor eye movements during locomotion in real-world environments, increasing the ecological validity of research on human gaze behavior. While collecting eye-tracking data is becoming more accessible, visual analytics of eye-tracking data remains difficult and time-consuming. As such, there is a significant need for developing efficient visualization and analysis tools for large-scale eye-tracking data. This work develops a first-of-its-kind eye-tracking data visualization and analysis system that allows for automatic recognition of independent objects within field-of-vision, using deep-learning-based semantic segmentation. This system recolors the fixated objects-of-interest by integrating gaze fixation information with semantic maps. The system effectively allows researchers to automatically infer what objects users view and for how long in dynamic contexts. The contributions are 1) a data visualization and analysis system that uses deep-learning technology along with eye-tracking data to automatically recognize objects-of-interest from head-mounted eye-tracking video recordings, and 2) a graphical user interface that presents objects-of-interest annotation along with eye-tracking data information. The architecture is tested with an outdoor case study of users walking around the Tufts University campus as part of a navigation study, which was administered by a team of research psychologists.

Eye tracking and virtual reality in the rehabilitation of mobility of hemianopia patients: a user experience study

Purpose: To test the usability of eye tracking and virtual reality during vision rehabilitation training of hemianopia patients. Methods: Individuals with hemianopia (n = 13) and normal-sighted controls (n = 4) performed various exercises that are commonly used in vision rehabilitation for mobility, while wearing a head-mounted eye tracker or a head-mounted virtual reality (VR) display. Occupational therapists (n = 4) guided them through the exercises. All participants (including therapists) filled out a questionnaire, assessing their experience with the used device. Individuals with hemianopia were split into three groups according to their stage in vision rehabilitation therapy and performed 1 (beginner), 2 (intermediate) or 3 (advanced) different exercises. Results: Individuals with hemianopia rated the mobile eye tracker with a score of 3.97 ± 0.5 points (beginner), 3.8 ± 0.5 points (intermediate) and 4 ± 0 points (advanced) the corresponding occupational therapists with a score of 3.6 ± 0.6, 3.4 ± 0.9 and 3.87 ± 0.6 points (out of a maximum of 4 points). The VR headset was rated with 3.9 ± 0.5 points by individuals with hemianopia, 3.8 ± 0.5 points by normal-sighted controls and 2.5 ± 1.4 points by the occupational therapist in a virtual hallway scenario. In a street-crossing scenario, it was rated with 3.7 ± 0.5 points by individuals with hemianopia, 3.7 ± 0.8 points by controls and 2.8 ± 1.2 by occupational therapists. In a walking along a pavement scenario the individual with hemianopia gave 4 ± 0 points and the controls 3.8 ± 0.4 points on average. Conclusions: Both devices were seen as useful additions to vision rehabilitation therapy, as they enable better feedback to patients and the opportunity to do different exercises at different levels of difficulty.

3D Gaze Estimation for Head-Mounted Eye Tracking System With Auto-Calibration Method

The general challenges of 3D gaze estimation for head-mounted eye tracking systems are inflexible marker-based calibration procedure and significant errors of depth estimation. In this paper, we propose a 3D gaze estimation with an auto-calibration method. To acquire the accurate 3D structure of the environment, an RGBD camera is applied as the scene camera of our system. By adopting the saliency detection method, saliency maps can be acquired through scene images, and 3D salient pixels in the scene are considered potential 3D calibration targets. The 3D eye model is built on the basis of eye images to determine gaze vectors. By combining 3D salient pixels and gaze vectors, the auto-calibration can be achieved with our calibration method. Finally, the 3D gaze point is obtained through the calibrated gaze vectors, and the point cloud is generated from the RGBD camera. The experimental result shows that the proposed system can achieve an average accuracy of 3.7° in the range of 1 m to 4 m indoors and 4.0° outdoors. The proposed system also presents a great improvement in depth measurement, which is sufficient for tracking users' visual attention in real scenes.

Accurate Regression-Based 3D Gaze Estimation Using Multiple Mapping Surfaces

Accurate 3D gaze estimation using a simple setup remains a challenging issue for head-mounted eye tracking. Current regression-based gaze direction estimation methods implicitly assume that all gaze directions intersect at one point called the eyeball pseudo-center. The effect of this implicit assumption on gaze estimation is unknown. In this paper, we find that this assumption is approximate based on a simulation of all intersections of gaze directions, and it is conditional based on a sensitivity analysis of the assumption in gaze estimation. Hence, we propose a gaze direction estimation method with one mapping surface that satisfies conditions of the assumption by configuring one mapping surface and achieving a high-quality calibration of the eyeball pseudo-center. This method only adds two additional calibration points outside the mapping surface. Furthermore, replacing the eyeball pseudo-center with an additional calibrated surface, we propose a gaze direction estimation method with two mapping surfaces that further improves the accuracy of gaze estimation. This method improves accuracy on the state-of-the-art method by 20 percent (from a mean error of 1.84 degrees to 1.48 degrees) on a public dataset with a usage range of 1 meter and by 17 percent (from a mean error of 2.22 degrees to 1.85 degrees) on a public dataset with a usage range of 2 meters.

What leads to coordinated attention in parent-toddler interactions? Children's hearing status matters.

Coordinated attention between children and their parents plays an important role in their social, language, and cognitive development. The current study used head-mounted eye-trackers to investigate the effects of children's prelingual hearing loss on how they achieve coordinated attention with their hearing parents during free-flowing object play. We found that toddlers with hearing loss (age: 24-37months) had similar overall gaze patterns (e.g., gaze length and proportion of face looking) as their normal-hearing peers. In addition, children's hearing status did not affect how likely parents and children attended to the same object at the same time during play. However, when following parents' attention, children with hearing loss used both parents' gaze directions and hand actions as cues, whereas children with normal hearing mainly relied on parents' hand actions. The diversity of pathways leading to coordinated attention suggests the flexibility and robustness of developing systems in using multiple pathways to achieve the same functional end.