Human Gaze Research Articles

Deep Active Learning with Contaminated Tags for Image Aesthetics Assessment.

Image aesthetic quality assessment has becoming an indispensable technique that facilitates a variety of image applications, e.g., photo retargeting and non-realistic rendering. Conventional approaches suffer from the following limitations: 1) the inefficiency of semantically describing images due to the inherent tag noise and incompletion, 2) the difficulty of accurately reflecting how humans actively perceive various regions inside each image, and 3) the challenge of incorporating the aesthetic experiences of multiple users. To solve these problems, we propose a novel semi-supervised deep active learning (SDAL) algorithm, which discovers how humans perceive semantically important regions from a large quantity of images partially assigned with contaminated tags. More specifically, as humans usually attend to the foreground objects before understanding them, we extract a succinct set of BING (binarized normed gradients) [60]-based object patches from each image. To simulate human visual perception, we propose SDAL which hierarchically learns human gaze shifting path (GSP) by sequentially linking semantically important object patches from each scenery. Noticeably, SDLA unifies the semantically important regions discovery and deep GSP feature learning into a principled framework, wherein only a small proportion of tagged images are adopted. Moreover, based on the sparsity penalty, SDLA can optimally abandon the noisy or redundant low-level image features. Finally, by leveraging the deeply-learned GSP features, a probabilistic model is developed for image aesthetics assessment, where the experience of multiple professional photographers can be encoded. Besides, auxiliary quality-related features can be conveniently integrated into our probabilistic model. Comprehensive experiments on a series of benchmark image sets have demonstrated the superiority of our method. As a byproduct, eye tracking experiments have shown that GSPs generated by our SDAL are about 93% consistent with real human gaze shifting paths.

Sex, skull length, breed, and age predict how dogs look at faces of humans and conspecifics.

The gaze of other dogs and humans is informative for dogs, but it has not been explored which factors predict face-directed attention. We used image presentations of unfamiliar human and dog heads, facing the observer (portrait) or facing away (profile), and measured looking time responses. We expected dog portraits to be aversive, human portraits to attract interest, and tested dogs of different sex, skull length and breed function, which in previous work had predicted human-directed attention. Dog portraits attracted longer looking times than human profiles. Mesocephalic dogs looked at portraits longer than at profiles, independent of the species in the image.Overall, brachycephalic dogs and dogs of unspecified breed function (such as mixed breeds) displayed the longest looking times. Among the latter, females observed the images for longer than males, which is in line with human findings on sex differences in processing faces. In a subsequent experiment, we tested whether dog portraits functioned as threatening stimuli. We hypothesized that dogs will avoid food rewards or approach them more slowly in the presence of a dog portrait, but found no effect of image type. In general, older dogs took longer to approach food placed in front of the images and mesocephalic dogs were faster than dogs of other skull length types. The results suggest that short-headed dogs are more attentive to faces, while sex and breed function predict looking times through complex interactions.

Calibration free, user-independent gaze estimation with tensor analysis

Human gaze directly signals visual attention, therefore, estimation of gaze has been an important research topic in fields such as human attention modeling and human-computer interaction. Accurate gaze estimation requires user, system and even session dependent parameters, which can be obtained by calibration process. However, this process has to be repeated whenever the parameter changes (head movement, camera movement, monitor movement). This study aims to eliminate the calibration process of gaze estimation by building a user-independent, appearance-based gaze estimation model. The system is ideal for multimodal interfaces, where the gaze is tracked without the cooperation from the users. The main goal is to capture the essential representation of the gaze appearance of the target user. We investigate the tensor analysis framework that decomposes the high dimension gaze data into different factors including individual differences, gaze differences, user-screen distances and session differences. The axis that is representative for a particular subject is automatically chosen in the tensor analysis framework using LASSO regression. The proposed approaches show promising results on capturing the test subject gaze changes. To address the estimation shift caused by the variations in individual heights, or relative position to the monitor, we apply domain adaptation to adjust the gaze estimation, observing further improvements. These promising results suggest that the proposed gaze estimation approach is a feasible and flexible scheme to facilitate gaze-based multimodal interfaces.

What Is a Ruin? The Western Definition

What Is a Ruin? The Western Definition

A deep-learning based feature hybrid framework for spatiotemporal saliency detection inside videos

Abstract Although research on detection of saliency and visual attention has been active over recent years, most of the existing work focuses on still image rather than video based saliency. In this paper, a deep learning based hybrid spatiotemporal saliency feature extraction framework is proposed for saliency detection from video footages. The deep learning model is used for the extraction of high-level features from raw video data, and they are then integrated with other high-level features. The deep learning network has been found extremely effective for extracting hidden features than that of conventional handcrafted methodology. The effectiveness for using hybrid high-level features for saliency detection in video is demonstrated in this work. Rather than using only one static image, the proposed deep learning model take several consecutive frames as input and both the spatial and temporal characteristics are considered when computing saliency maps. The efficacy of the proposed hybrid feature framework is evaluated by five databases with human gaze complex scenes. Experimental results show that the proposed model outperforms five other state-of-the-art video saliency detection approaches. In addition, the proposed framework is found useful for other video content based applications such as video highlights. As a result, a large movie clip dataset together with labeled video highlights is generated.

Predicting human gaze with multi-level information

Eye fixation models, which try to quantitatively predict human eye attended areas in visual fields, have received increasing interest in recent years. In this paper, a novel framework is proposed for the detection of eye fixations. First, a multi-channel detection module, which extracts information of color contrast, salient object proposals and center bias from input image, is conducted to introduce various useful information into the subsequent fixations detection. In salient object detection channel, we employ the multi-instance learning (MIL) algorithm to determine which object proposal can attract attention, which avoids the fuzzyness of positive sample selection. Second, an adaptive weighted fusion method achieved by deep learning framework is proposed to fuse the multi-level information (i.e., contrast, objective, center bias) together for the detection of fixations, so that the integration of information between each level becomes more scientific. Finally, the detection result is optimized by embedding semantic information. Experimental results show that the algorithm has achieved competitive results in MIT1003, MIT300 and Toronto120 dataset.

Using context from inside‐out vision for improved activity recognition

The authors propose a method to improve activity recognition by including the contextual information from first person vision (FPV). Adding the context, i.e. objects seen while performing an activity, increases the activity recognition precision. This is because, in goal‐oriented tasks, human gaze precedes the action and tends to focus on relevant objects. They extract object information from FPV images and combine it with the activity information from external or FPV videos to train an Artificial Neural Network (ANN). They used four configurations as combination of gaze/eye‐tracker, head‐mounted and externally mounted cameras using three standard cooking datasets from Georgia Tech Egocentric Activities Gaze, Technische Universität München kitchen and CMU multi‐modal activity database. Adding object information when training the ANN increased the precision and accuracy of activity recognition from average 58.02% (and 89.78%) to 74.03% (and 93.42%). Experiments also showed that when objects are not considered, having an external camera is necessary. However, when objects are considered, the combination of internal and external cameras is optimal because of their complementary advantages in observing hand and objects. Adding object information also decreases ANN training cycles from 513.25 to 139, which shows that it provides critical information that speeds up training.

Perceived robot personality affects social attention in real-time human-robot interaction

Perceived robot personality affects social attention in real-time human-robot interaction

What can aftereffects reveal about the functional architecture of human gaze perception?

What can aftereffects reveal about the functional architecture of human gaze perception?

Camera-Assisted Video Saliency Prediction and Its Applications.

Video saliency prediction is an indispensable yet challenging technique which can facilitate various applications, such as video surveillance, autonomous driving, and realistic rendering. Based on the popularity of embedded cameras, we in this paper predict region-level saliency from videos by leveraging human gaze locations recorded using a camera, (e.g., those equipped on an iMAC and laptop PC). Our proposed camera-assisted mechanism improves saliency prediction by discovering human attended regions inside a video clip. It is orthogonal to the current saliency models, i.e., any existing video/image saliency model can be boosted by our mechanism. First of all, the spatial-and temporal-level visual features are exploited collaboratively for calculating an initial saliency map. We notice that the current saliency models are not sufficiently adaptable to the variations in lighting, different view angles, and complicated backgrounds. Therefore, assisted by a camera tracking human gaze movements, a non-negative matrix factorization algorithm is designed to accurately localize the semantically/visually salient video regions perceived by humans. Finally, the learned human gaze locations as well as the initial saliency map are integrated to optimize video saliency calculation. Empirical results thoroughly demonstrated that: 1) our approach achieves the state-of-the-art video saliency prediction accuracy by outperforming 11 mainstream algorithms considerably and 2) our method can conveniently and successfully enhance video retargeting, quality estimation, and summarization.

Gaze-Following and Reaction to an Aversive Social Interaction Have Corresponding Associations with Variation in the OXTR Gene in Dogs but Not in Human Infants.

It has been suggested that dogs' remarkable capacity to use human communicative signals lies in their comparable social cognitive skills; however, this view has been questioned recently. The present study investigated associations between oxytocin receptor gene (OXTR) polymorphisms and social behavior in human infants and dogs with the aim to unravel potentially differential mechanisms behind their responsiveness to human gaze. Sixteen-month-old human infants (N = 99) and adult Border Collie dogs (N = 71) participated in two tasks designed to test (1) their use of gaze-direction as a cue to locate a hidden object, and (2) their reactions to an aversive social interaction (using the still face task for children and a threatening approach task for dogs). Moreover, we obtained DNA samples to analyze associations between single nucleotide polymorphisms (SNP) in the OXTR (dogs: −213AG, −94TC, −74CG, rs8679682, children: rs53576, rs1042778, rs2254298) and behavior. We found that OXTR genotype was significantly associated with reactions to an aversive social interaction both in dogs and children, confirming the anxiolytic effect of oxytocin in both species. In dogs, the genotypes linked to less fearful behavior were associated also with a higher willingness to follow gaze whereas in children, OXTR gene polymorphisms did not affect gaze following success. This pattern of gene-behavior associations suggests that for dogs the two situations are more alike (potentially fear-inducing or competitive) than for human children. This raises the possibility that, in contrast to former studies proposing human-like cooperativeness in dogs, dogs may perceive human gaze in an object-choice task in a more antagonistic manner than children.

Efficient eye typing with 9-direction gaze estimation

Vision based text entry systems aim to help disabled people achieve text communication using eye movement. Most previous methods have employed an existing eye tracker to predict gaze direction and design an input method based upon that. However, these methods can result in eye tracking quality becoming easily affected by various factors and lengthy amounts of time for calibration. Our paper presents a novel efficient gaze based text input method, which has the advantage of low cost and robustness. Users can type in words by looking at an on-screen keyboard and blinking. Rather than estimate gaze angles directly to track eyes, we introduce a method that divides the human gaze into nine directions. This method can effectively improve the accuracy of making a selection by gaze and blinks. We build a Convolutional Neural Network (CNN) model for 9-direction gaze estimation. On the basis of the 9-direction gaze, we use a nine-key T9 input method which is widely used in candy bar phones. Bar phones were very popular in the world decades ago and have cultivated strong user habits and language models. To train a robust gaze estimator, we created a large-scale dataset with images of eyes sourced from 25 people. According to the results from our experiments, our CNN model is able to accurately estimate different people's gaze under various lighting conditions by different devices. In considering disable people's needs, we removed the complex calibration process. The input methods can run in screen mode and portable off-screen mode. Moreover, The datasets used in our experiments are made available to the community to allow further experimentation.

Perceptual multi-channel visual feature fusion for scene categorization

Effectively recognizing sceneries from a variety of categories is an indispensable but challenging technique in computer vision and intelligent systems. In this work, we propose a novel image kernel based on human gaze shifting, aiming at discovering the mechanism of humans perceiving visually/semantically salient regions within a scenery. More specifically, we first design a weakly supervised embedding algorithm which projects the local image features (i.e., graphlets in this work) onto the pre-defined semantic space. Thereby, we describe each graphlet by multiple visual features at both low-level and high-level. It is generally acknowledged that humans attend to only a few regions within a scenery. Thus we formulate a sparsity-constrained graphlet ranking algorithm which incorporates visual clues at both the low-level and the high-level. According to human visual perception, these top-ranked graphlets are either visually or semantically salient. We sequentially connect them into a path which mimics human gaze shifting. Lastly, a so-called gaze shifting kernel (GSK) is calculated based on the learned paths from a collection of scene images. And a kernel SVM is employed for calculating the scene categories. Comprehensive experiments on a series of well-known scene image sets shown the competitiveness and robustness of our GSK. We also demonstrated the high consistency of the predicted path with real human gaze shifting path.

Seen from the Viewpoint of a Fish: Posthumanist Observation in Lucien Castaing-Taylor and Véréna Paravel’s ‘Leviathan’

This article conceptualises Lucien Castaing-Taylor and Véréna Paravel’s experimental documentary Leviathan (2012) using Niklas Luhmann’s observation theory and Cary Wolfe’s writing on posthumanism which, significantly influenced by Luhmann’s attack on anthropocentrism in social theory, questions the importance of human agency for social and psychic systems. Leviathan, I argue, engenders perspectives drawn from visual culture that enable a rethinking of hierarchical humanist ethics based on species membership, contributing to posthumanist critical discourse.Leviathan offers a radically non-anthropocentric take on the topic of industrial fishing, presented via innovative use of camera placements and cinematic points of view. Unpredictable camera movements involving contingent framing and angles generate an open-ended work not tied to the human gaze. Shooting from a caught marine animal’s point of view forces the viewer to assume an unexpected perspective. To analyse this particular perspective, I turn to Luhmann’s theories about an observation not tied to human subjectivity, where the subject of observation is simultaneously an object, and where the external world is equally inaccessible to humans and nonhumans alike. In this schema, present in any observation is a constitutive blindness that can only be seen by another observer, but it is this very blindness which makes the observation possible.

Learning a gaze estimator with neighbor selection from large-scale synthetic eye images

Appearance-based gaze estimation works well in inferring human gaze under real-world condition. But one of the significant limitations in appearance-based methods is the need for huge amounts of training data. Eye image synthesis addresses this problem by generating huge amounts of synthetic eye images with computer graphics. To fully use the large-scale synthetic eye images, a simple-but-effective appearance-based gaze estimation framework with neighbor selection is proposed in this paper. The proposed framework hierarchically fuses multiple k-NN queries (in head pose, pupil center and eye appearance spaces) to choose closest samples with more relevant features. Considering the structure characters of the closet samples, neighbor regression methods then can be applied to predict the gaze directions. Experimental results demonstrate that the representative neighbor regression methods under the proposed framework achieve better performance for within-subject and cross-subject gaze estimation.

InvisibleEye

Analysis of everyday human gaze behaviour has significant potential for ubiquitous computing, as evidenced by a large body of work in gaze-based human-computer interaction, attentive user interfaces, and eye-based user modelling. However, current mobile eye trackers are still obtrusive, which not only makes them uncomfortable to wear and socially unacceptable in daily life, but also prevents them from being widely adopted in the social and behavioural sciences. To address these challenges we present InvisibleEye, a novel approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames. To compensate for the cameras’ low image resolution of only a few pixels, our approach uses multiple cameras to capture different views of the eye, as well as learning-based gaze estimation to directly regress from eye images to gaze directions. We prototypically implement our system and characterise its performance on three large-scale, increasingly realistic, and thus challenging datasets: 1) eye images synthesised using a recent computer graphics eye region model, 2) real eye images recorded of 17 participants under controlled lighting, and 3) eye images recorded of four participants over the course of four recording sessions in a mobile setting. We show that InvisibleEye achieves a top person-specific gaze estimation accuracy of 1.79° using four cameras with a resolution of only 5 × 5 pixels. Our evaluations not only demonstrate the feasibility of this novel approach but, more importantly, underline its significant potential for finally realising the vision of invisible mobile eye tracking and pervasive attentive user interfaces.

Towards pervasive eye tracking

Abstract The human gaze provides paramount cues for communication and interaction. Following this insight, gaze-based interfaces have been proposed for human-computer interaction (HCI) since the early 90s, with some believing that such interfaces will revolutionize the way we interact with our devices. Since then gaze-based HCI in stationary scenarios (e. g., desktop computing) has been rapidly maturing, and the production costs of mainstream eye trackers have been steadily decreasing. In consequence, a variety of new applications with the ambitious goal to apply eye tracking to dynamic, real-world HCI tasks and scenarios have emerged. This article gives an overview of the research conducted by the Perception Engineering Group at the University of Tübingen.

Multisensory integration in orienting behavior: Pupil size, microsaccades, and saccades

Signals from different sensory modalities are integrated in the brain to optimize behavior. Although multisensory integration has been demonstrated in saccadic eye movements, its influence on other orienting responses, including pupil size and microsaccades, is still poorly understood. We examined human gaze orienting responses following presentation of visual, auditory, or combined audiovisual stimuli. Transient pupil dilation and microsaccade inhibition were evoked shortly after the appearance of a salient stimulus. Audiovisual stimuli evoked larger pupil dilation, greater microsaccade inhibition, and faster saccade reaction times compared to unimodal conditions. Trials with faster saccadic reaction times were accompanied with greater pupil dilation responses. Similar modulation of pre-stimulus pupil-size-change rate was observed between stimulus-evoked saccadic and pupillary responses. Thus, multisensory integration impacts multiple components of orienting, with coordination between saccade and pupil responses, implicating the superior colliculus in coordinating these responses because of its central role in both orienting behavior and multisensory integration.

Media Quality Assessment by Perceptual Gaze-Shift Patterns Discovery

Quality assessment is an indispensable technique in a large body of media applications, i.e., photo retargeting, scenery rendering, and video summarization. In this paper, a fully automatic framework is proposed to mimic how humans subjectively perceive media quality. The key is a locality-preserved sparse encoding algorithm that accurately discovers human gaze shifting paths from each image or video clip. In particular, we first extract local image descriptors from each image/video, and subsequently project them into the so-called perceptual space. Then, a nonnegative matrix factorization (NMF) algorithm is proposed that represents each graphlet by a linear and sparse combination of the basis ones. Since each graphlet is visually/semantically similar to its neighbors, a locality-preserved constraint is encoded into the NMF algorithm. Mathematically, the saliency of each graphlet is quantified by the norm of its sparse codes. Afterward, we sequentially link them into a path to simulate human gaze allocation. Finally, a probabilistic quality model is learned based on such paths extracted from a collection of photos/videos, which are marked as high quality ones via multiple Flickr users. Comprehensive experiments have demonstrated that: 1) our quality model outperforms many of its competitors significantly, and 2) the learned paths are on average 89.5% consistent with real human gaze shifting paths.

Do pet dogs (Canis familiaris) follow ostensive and non-ostensive human gaze to distant space and to objects?

Dogs are renowned for being skilful at using human-given communicative cues such as pointing. Results are contradictory, however, when it comes to dogs' following human gaze, probably due to methodological discrepancies. Here we investigated whether dogs follow human gaze to one of two food locations better than into distant space even after comparable pre-training. In Experiments 1 and 2, the gazing direction of dogs was recorded in a gaze-following into distant space and in an object-choice task where no choice was allowed, in order to allow a direct comparison between tasks, varying the ostensive nature of the gazes. We found that dogs only followed repeated ostensive human gaze into distant space, whereas they followed all gaze cues in the object-choice task. Dogs followed human gaze better in the object-choice task than when there was no obvious target to look at. In Experiment 3, dogs were tested in another object-choice task and were allowed to approach a container. Ostensive cues facilitated the dogs’ following gaze with gaze as well as their choices: we found that dogs in the ostensive group chose the indicated container at chance level, whereas they avoided this container in the non-ostensive group. We propose that dogs may perceive the object-choice task as a competition over food and may interpret non-ostensive gaze as an intentional cue that indicates the experimenter's interest in the food location she has looked at. Whether ostensive cues simply mitigate the competitive perception of this situation or they alter how dogs interpret communicative gaze needs further investigation. Our findings also show that following gaze with one's gaze and actually choosing one of the two containers in an object-choice task need to be considered as different variables. The present study clarifies a number of questions related to gaze-following in dogs and adds to a growing body of evidence showing that human ostensive cues can strongly modify dog behaviour.