Human Attention Model Research Articles

Towards Modeling Human Attention from Eye Movements for Neural Source Code Summarization

Neural source code summarization is the task of generating natural language descriptions of source code behavior using neural networks. A fundamental component of most neural models is an attention mechanism. The attention mechanism learns to connect features in source code to specific words to use when generating natural language descriptions. Humans also pay attention to some features in code more than others. This human attention reflects experience and high-level cognition well beyond the capability of any current neural model. In this paper, we use data from published eye-tracking experiments to create a model of this human attention. The model predicts which words in source code are the most important for code summarization. Next, we augment a baseline neural code summarization approach using our model of human attention. We observe an improvement in prediction performance of the augmented approach in line with other bio-inspired neural models.

Where to look at the movies: Analyzing visual attention to understand movie editing.

In the process of making a movie, directors constantly care about where the spectator will look on the screen. Shot composition, framing, camera movements, or editing are tools commonly used to direct attention. In order to provide a quantitative analysis of the relationship between those tools and gaze patterns, we propose a new eye-tracking database, containing gaze-pattern information on movie sequences, as well as editing annotations, and we show how state-of-the-art computational saliency techniques behave on this dataset. In this work, we expose strong links between movie editing and spectators gaze distributions, and open several leads on how the knowledge of editing information could improve human visual attention modeling for cinematic content. The dataset generated and analyzed for this study is available at https://github.com/abruckert/eye_tracking_filmmaking.

Effective Video Summarization Approach Based on Visual Attention

Video summarization is applied to reduce redundancy and develop a concise representation of key frames in the video, more recently, video summaries have been used through visual attention modeling. In these schemes, the frames that stand out visually are extracted as key frames based on human attention modeling theories. The schemes for modeling visual attention have proven to be effective for video summaries. Nevertheless, the high cost of computing in such techniques restricts their usability in everyday situations. In this context, we propose a method based on KFE (key frame extraction) technique, which is recommended based on an efficient and accurate visual attention model. The calculation effort is minimized by utilizing dynamic visual highlighting based on the temporal gradient instead of the traditional optical flow techniques. In addition, an efficient technique using a discrete cosine transformation is utilized for the static visual salience. The dynamic and static visual attention metrics are merged by means of a non-linear weighted fusion technique. Results of the system are compared with some existing state-of-the-art techniques for the betterment of accuracy. The experimental results of our proposed model indicate the efficiency and high standard in terms of the key frames extraction as output.

Evaluation of the Relationships between Saliency Maps and Keypoints

The saliency map is proposed by Itti et al., to represent the conspicuity or saliency in the visual field and to guide the selection of attended locations based on the spatial distribution of saliency, which works as the trigger of bottom-up attention. If a certain location in the visual field is sufficiently different from its surrounding, we naturally pay attention to the characteristic of visual scene. In the research of computer vision, image feature extraction methods such as Scale-Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF), Binary Robust Invariant Scalable Keypoint (BRISK) etc., have been proposed to extract keypoints robust to size change or rotation of target objects. These feature extraction methods are inevitable techniques for image mosaicking and Visual SLAM (Simultaneous Localization and Mapping), on the other hand, have big influence to photographing condition change of luminance, defocusing and so on. However, the relation between human attention model, Saliency map, and feature extraction methods in computer vision is not well discussed. In this paper, we propose a new saliency map and discuss the stability of keypoints extraction and their locations using BRISK by comparing other saliency maps.

How is Gaze Influenced by Image Transformations? Dataset and Model.

Data size is the bottleneck for developing deep saliency models, because collecting eye-movement data is very time-consuming and expensive. Most of current studies on human attention and saliency modeling have used high-quality stereotype stimuli. In real world, however, captured images undergo various types of transformations. Can we use these transformations to augment existing saliency datasets? Here, we first create a novel saliency dataset including fixations of 10 observers over 1900 images degraded by 19 types of transformations. Second, by analyzing eye movements, we find that observers look at different locations over transformed versus original images. Third, we utilize the new data over transformed images, called data augmentation transformation (DAT), to train deep saliency models. We find that label-preserving DATs with negligible impact on human gaze boost saliency prediction, whereas some other DATs that severely impact human gaze degrade the performance. These label-preserving valid augmentation transformations provide a solution to enlarge existing saliency datasets. Finally, we introduce a novel saliency model based on generative adversarial networks (dubbed GazeGAN). A modified U-Net is utilized as the generator of the GazeGAN, which combines classic "skip connection" with a novel "center-surround connection" (CSC) module. Our proposed CSC module mitigates trivial artifacts while emphasizing semantic salient regions, and increases model nonlinearity, thus demonstrating better robustness against transformations. Extensive experiments and comparisons indicate that GazeGAN achieves state-of-the-art performance over multiple datasets. We also provide a comprehensive comparison of 22 saliency models on various transformed scenes, which contributes a new robustness benchmark to saliency community. Our code and dataset are available at.

Formal modeling and analysis of safety-critical human multitasking

When a person is concurrently interacting with different systems, the amount of cognitive resources required (cognitive load) could be too high and might prevent some tasks from being completed. When such human multitasking involves safety-critical tasks, such as in an airplane, a spacecraft, or a car, failure to devote sufficient attention to the different tasks could have serious consequences. For example, using a GPS with high cognitive load while driving might take the attention away for too long from the safety-critical task of driving the car. To study this problem, we define an executable formal model of human attention and multitasking in Real-Time Maude. It includes a description of the human working memory and the cognitive processes involved in the interaction with a device. Our framework enables us to analyze human multitasking through simulation, reachability analysis, and LTL and timed CTL model checking, and we show how a number of prototypical multitasking problems can be analyzed in Real-Time Maude. We illustrate our modeling and analysis framework by studying: (i) the interaction with a GPS navigation system while driving, (ii) some typical scenarios involving human errors in air traffic control (ATC), and (iii) a medical operator setting multiple infusion pumps simultaneously. We apply model checking to show that in some cases the cognitive load of the navigation system could cause the driver to keep the focus away from driving for too long, and that working memory overload and distraction may cause an air traffic controller or a medical operator to make critical mistakes.

A Survey of Attention Management Systems in Ubiquitous Computing Environments

Today's information and communication devices provide always-on connectivity, instant access to an endless repository of information, and represent the most direct point of contact to almost any person in the world. Despite these advantages, devices such as smartphones or personal computers lead to the phenomenon of attention fragmentation, continuously interrupting individuals' activities and tasks with notifications. Attention management systems aim to provide active support in such scenarios, managing interruptions, for example, by postponing notifications to opportune moments for information delivery. In this article, we review attention management system research with a particular focus on ubiquitous computing environments. We first examine cognitive theories of attention and extract guidelines for practical attention management systems. Mathematical models of human attention are at the core of these systems, and in this article, we review sensing and machine learning techniques that make such models possible. We then discuss design challenges towards the implementation of such systems, and finally, we investigate future directions in this area, paving the way for new approaches and systems supporting users in their attention management.

Calibration free, user-independent gaze estimation with tensor analysis

Human gaze directly signals visual attention, therefore, estimation of gaze has been an important research topic in fields such as human attention modeling and human-computer interaction. Accurate gaze estimation requires user, system and even session dependent parameters, which can be obtained by calibration process. However, this process has to be repeated whenever the parameter changes (head movement, camera movement, monitor movement). This study aims to eliminate the calibration process of gaze estimation by building a user-independent, appearance-based gaze estimation model. The system is ideal for multimodal interfaces, where the gaze is tracked without the cooperation from the users. The main goal is to capture the essential representation of the gaze appearance of the target user. We investigate the tensor analysis framework that decomposes the high dimension gaze data into different factors including individual differences, gaze differences, user-screen distances and session differences. The axis that is representative for a particular subject is automatically chosen in the tensor analysis framework using LASSO regression. The proposed approaches show promising results on capturing the test subject gaze changes. To address the estimation shift caused by the variations in individual heights, or relative position to the monitor, we apply domain adaptation to adjust the gaze estimation, observing further improvements. These promising results suggest that the proposed gaze estimation approach is a feasible and flexible scheme to facilitate gaze-based multimodal interfaces.

Automated detection of foveal center in SD-OCT images using the saliency of retinal thickness maps.

To develop an automated method based on saliency map of the retinal thickness map to determine foveal center in spectral-domain optical coherence tomography (SD-OCT) images. This paper proposes an automatic method for the detection of the foveal center in SD-OCT images. Initially, a retinal thickness map is generated by considering the axial distance between the internal limiting membrane (ILM) and the Bruch's membrane (BM). Both the ILM and BM boundaries are automatically segmented by a known retinal segmentation technique. The macular foveal region is identified as a salient feature in the retinal thickness map, and segmented by the saliency detection method based on a human vision attention model. Finally, the foveal center is identified by searching for the lowest point from the determined macular fovea region. Experimental results in 39 scans from 35 healthy eyes and 58 scans from 29 eyes diagnosed with several stages of age-related macular degeneration (AMD), from mild or intermediate stages to severe dry or wet stages, demonstrated that the proposed method achieves good performance. The mean radial distance error of the automatically detected foveal center locations when compared to consensus manual determination established by repeated sessions from two expert readers was 52 ± 56 μm for the normal eyes and 73 ± 63 μm for AMD eyes. The proposed algorithm was more effective for detecting the foveal center automatically in SD-OCT images than the state-of-art methods.

Weather Radar Spatiotemporal Saliency: A First Look at an Information Theory–Based Human Attention Model Adapted to Reflectivity Images

AbstractForecasters often monitor and analyze large amounts of data, especially during severe weather events, which can be overwhelming. Thus, it is important to effectively allocate their finite perceptual and cognitive resources for the most relevant information. This paper introduces a novel analysis tool that quantifies the amount of spatial and temporal information in time series of constant-elevation weather radar reflectivity images. The proposed Weather Radar Spatiotemporal Saliency (WR–STS) is based on the mathematical model of the human attention system (referred to as saliency) adapted to radar reflectivity images and makes use of information theory concepts. It is shown that WR-STS highlights spatially and temporally salient (attention attracting) regions in weather radar reflectivity images, which can be associated with meteorologically important regions. Its skill in highlighting current regions of interest is assessed by analyzing the WR-STS values within regions in which severe weather is likely to strike in the near future as defined by National Weather Service forecasters. The performance of WR-STS is demonstrated for a severe weather case and analyzed for a set of 10 diverse cases. Results support the hypothesis that WR-STS can identify regions with meteorologically important echoes and could assist in discerning fast-changing, highly structured weather echoes during complex severe weather scenarios, ultimately allowing forecasters to focus their attention and spend more time analyzing those regions.

Stimulus-Driven Reorienting Impairs Executive Control of Attention: Evidence for a Common Bottleneck in Anterior Insula.

A classical model of human attention holds that independent neural networks realize stimulus-driven reorienting and executive control of attention. Questioning full independence, the two functions do, however, engage overlapping networks with activations in cingulo-opercular regions such as anterior insula (AI) and a reverse pattern of activation (stimulus-driven reorienting), and deactivation (executive control) in temporoparietal junction (TPJ). To test for independent versus shared neural mechanisms underlying stimulus-driven and executive control of attention, we used fMRI and a task that isolates individual from concurrent demands in both functions. Results revealed super-additive increases of left AI activity and behavioral response costs under concurrent demands, suggesting a common bottleneck for stimulus-driven reorienting and executive control of attention. These increases were mirrored by non-additive decreases of activity in the default mode network (DMN), including posterior TPJ, regions where activity increased with off-task processes. The deactivations in posterior TPJ were spatially separated from stimulus-driven reorienting related activation in anterior TPJ, a differentiation that replicated in task-free resting state. Furthermore, functional connectivity indicated inhibitory coupling between posterior TPJ and AI during concurrent attention demands. These results demonstrate a role of AI in stimulus-driven and executive control of attention that involves down-regulation of internally directed processes in DMN.

Human Visual Attention Model Based on Analysis of Magic for Smooth Human–Robot Interaction

In order to smoothly interact with humans, it is desirable that a robot can guide human attention and behaviors. In this study, we developed a model of human visual attention for guiding human attention based on an analysis of a magic trick performance. We measured human gaze points of people watching a video of a magic trick performance and compared them with the area where the magician intended to draw a spectator’s attention. The analysis showed that the relationship between the magician’s face, hands, and gaze plays an important role in guiding the spectator’s attention. On the basis of the preliminary user studies on watching the magic video, we integrated a saliency map and a manipulation map that describes the relationship between gaze and hands to develop a novel human attention model. The evaluation using the observed gaze points demonstrated that the proposed model can better explain human visual attention than the saliency map while people are watching a video of a magic trick performance.

Perceptual synoptic view of pixel, object and semantic based attributes of video

For a scene, what are the object and semantic based attributes, other than the pixel based attributes, and how do they affect our attentional selection are some of the questions we need to address. We studied the effects of various attributes on our attentional perspective. We described a new saliency prediction model that accounts for different pixel-level attributes as color, contrast and intensity; object level attributes such as size, shape of objects and semantic level attributes as motion and speed of objects. We quantified these attributes based on motion contrast, motion energy and motion chromism. With this in view, we examined the problem of information prioritizing and filtering with emphasis on directing this exercise using object and semantic based attributes of the human attention model. We have evaluated proposed approach on different types of videos for their quantitative and qualitative comparison. The promising results create a gateway for synopsis view.

Divide-and-conquer based summarization framework for extracting affective video content

Recent advances in multimedia technology have led to tremendous increases in the available volume of video data, thereby creating a major requirement for efficient systems to manage such huge data volumes. Video summarization is one of the key techniques for accessing and managing large video libraries. Video summarization can be used to extract the affective contents of a video sequence to generate a concise representation of its content. Human attention models are an efficient means of affective content extraction. Existing visual attention driven summarization frameworks have high computational cost and memory requirements, as well as a lack of efficiency in accurately perceiving human attention. To cope with these issues, we propose a divide-and-conquer based framework for an efficient summarization of big video data. We divide the original video data into shots, where an attention model is computed from each shot in parallel. Viewer's attention is based on multiple sensory perceptions, i.e., aural and visual, as well as the viewer's neuronal signals. The aural attention model is based on the Teager energy, instant amplitude, and instant frequency, whereas the visual attention model employs multi-scale contrast and motion intensity. Moreover, the neuronal attention is computed using the beta-band frequencies of neuronal signals. Next, an aggregated attention curve is generated using an intra- and inter-modality fusion mechanism. Finally, the affective content in each video shot is extracted. The fusion of multimedia and neuronal signals provides a bridge that links the digital representation of multimedia with the viewer’s perceptions. Our experimental results indicate that the proposed shot-detection based divide-and-conquer strategy mitigates the time and computational complexity. Moreover, the proposed attention model provides an accurate reflection of the user preferences and facilitates the extraction of highly affective and personalized summaries.

Development of attention networks and their interactions in childhood.

In the present study, we investigated developmental trajectories of alerting, orienting, and executive attention networks and their interactions over childhood. Two cross-sectional experiments were conducted with different samples of 6- to 12-year-old children using modified versions of the attention network task (ANT). In Experiment 1 (N = 106), alerting and orienting cues were independently manipulated, thus allowing examination of interactions between these 2 networks, as well as between them and the executive attention network. In Experiment 2 (N = 159), additional changes were made to the task in order to foster exogenous orienting cues. Results from both studies consistently revealed separate developmental trajectories for each attention network. Children younger than 7 years exhibited stronger benefits from having an alerting auditory signal prior to the target presentation. Developmental changes in orienting were mostly observed on response accuracy between middle and late childhood, whereas executive attention showed increases in efficiency between 7 years and older ages, and further improvements in late childhood. Of importance, across both experiments, significant interactions between alerting and orienting, as well as between each of these and the executive attention network, were observed. Alerting cues led to speeding shifts of attention and enhancing orienting processes. Also, both alerting and orienting cues modulated the magnitude of the flanker interference effect. These findings inform current theoretical models of human attention and its development, characterizing for the first time, the age-related course of attention networks interactions that, present in adults, stem from further refinements over childhood.

Efficient visual attention based framework for extracting key frames from videos

The huge amount of video data on the internet requires efficient video browsing and retrieval strategies. One of the viable solutions is to provide summaries of the videos in the form of key frames. The video summarization using visual attention modeling has been used of late. In such schemes, the visually salient frames are extracted as key frames on the basis of theories of human attention modeling. The visual attention modeling schemes have proved to be effective in video summarization. However, the high computational costs incurred by these techniques limit their applicability in practical scenarios. In this context, this paper proposes an efficient visual attention model based key frame extraction method. The computational cost is reduced by using the temporal gradient based dynamic visual saliency detection instead of the traditional optical flow methods. Moreover, for static visual saliency, an effective method employing discrete cosine transform has been used. The static and dynamic visual attention measures are fused by using a non-linear weighted fusion method. The experimental results indicate that the proposed method is not only efficient, but also yields high quality video summaries.

Towards the automatic selection of optimal seam line locations when merging optical remote-sensing images

Image mosaicking is an important task in remote sensing due to the need for imagery with a greater spatial extent than provided by individual scenes. Merging of images requires the selection of a seam line within their area of overlap along which the scenes are merged. The seam line has a better chance of being invisible if it lies in regions where the images to be merged are very similar. The automatic detection of an optimal seam line is not a trivial task as it is difficult to find laterally continuous regions with high image similarity, and to identify image similarities when there are variations in the images, for example due to different illuminations or viewing directions, or shadow differences of tall structures. This article presents an automatic seam line location technique for remote-sensing images and achieves the following three objectives: to trace along the locations with minimal image difference so that the merged data set appears seamless; to avoid creating discontinuities within salient features within the images; and to ensure that the more accurate radiometric values that are associated with the least distance from the nadir point are better preserved in the mosaic image. Therefore, our method uses pixel-based image similarity measurement to choose the locations with high colour, edge and texture similarity; a region-based saliency map that is based on a human attention model to identify and avoid the areas with visibly dominant foreground objects; and location preference to encourage the seam line to lie as close as possible to an equal distance from the two nadir points of the images being merged. These measures are used as input to a cost function and the estimated costs are used to guide the tracing of the seam line in a dynamic programming algorithm. Our experiments demonstrate that the consideration of a combination of factors produces superior results to using just one or two of the variables as controls when merging high-resolution images containing complex structures.

A SOFTWARE ENVIRONMENT FOR AN ADAPTIVE HUMAN-AWARE SOFTWARE AGENT SUPPORTING ATTENTION-DEMANDING TASKS

This paper presents a software environment providing human-aware ambient support for a human performing a task that demands substantial amounts of attention. The agent obtains human attention-awareness in an adaptive manner by use of a dynamical model of human attention, gaze sensoring by an eye-tracker, and information about features of the objects in the environment which is parameterised for characteristics of the human specified above. The agent uses a built-in adaptation model to adapt on the fly, the values of these parameters to the personal characteristics of the human. The software agent has been implemented in a component-based manner within the Adobe®Flex®environment, thereby also integrating the Tobii®eye-tracker. It has been applied in a setup for a task where the human has to identify enemies and allies, and eliminate the enemies.

A software environment for a human-aware ambient agent supporting attention-demanding tasks

Abstract This paper presents a software environment for a human-aware ambient agent providing support for a human performing an attention-demanding task. The agent obtains human attention-awareness by use of a dynamical model of human attention, gaze sensoring by an eye-tracker, and information about features of the objects in the environment. It has been implemented in a component-based, event-driven manner within the Adobe ® Flex ® environment, thereby also integrating the Tobii ® eye-tracker. It has been applied in a setup for a task where the human has to identify enemies and allies, and eliminate the enemies.

Predicting Pilot Performance in Off-Nominal Conditions: A Meta-Analysis and Model Validation

Pilot response to off-nominal (very rare) events represents a critical component to understanding the safety of next generation airspace technology and procedures. We describe a meta-analysis designed to integrate the existing data regarding pilot accuracy of detecting rare, unexpected events such as runway incursions in realistic flight simulations. Thirty-five studies were identified and pilot responses were categorized by expectancy, event location, and whether the pilot was flying with a highway-in-the-sky display. All three dichotomies produced large, significant effects on event miss rate. A model of human attention and noticing, N-SEEV, was then used to predict event noticing performance as a function of event salience and expectancy, and retinal eccentricity. Eccentricity is predicted from steady state scanning by the SEEV model of attention allocation. The model was used to predict miss rates for the expectancy, location and highway-in-the-sky (HITS) effects identified in the meta-analysis. The correlation between model-predicted results and data from the meta-analysis was 0.72.