Human Visual Attention Research Articles

A human visual attention analysis model for remote interaction interface of unmanned agricultural vehicles

When unmanned agricultural vehicles (UAVs) encounter unexpected failures or incidents during autonomous operations, a highly efficient remote human–machine interaction (HMI) interface is needed to assist in the manual joint control process. The highly complex cognitive mechanisms and visual characteristics of humans make it difficult to design efficient human–machine interfaces. Therefore, a human visual attention analysis model was proposed based on fully convolutional networks for the remote interaction interface of unmanned agricultural vehicles. First, visual attention experimental software combined with gaze-contingent real-time simulation methods was developed to build the dataset. Subsequently, the human visual attention analysis model (HVAAM) is constructed, and the corresponding training strategy is proposed, which is trained on the ILSVRC2012 standard and self-constructed datasets. The accuracy of the HVAAM analysis on the test dataset was validated. Ultimately, a human–machine interaction test of random failure warning for remote monitoring interface of unmanned plowing operation is performed with a self-developed unmanned electric tractor as the carrier. In the test dataset, the average analysis accuracy of HVAAM was 79.2 %, and the proposed training strategy led to a 20.7 % improvement in the model accuracy. In the unmanned electric tractor human–machine interaction test, HVAAM achieved 77.9 % accuracy in the visual attention analysis of the monitoring interface. This study can effectively facilitate the analysis, design, and optimization of remote monitoring interfaces for unmanned agricultural vehicles.

Learning From Human Attention for Attribute-Assisted Visual Recognition.

With prior knowledge of seen objects, humans have a remarkable ability to recognize novel objects using shared and distinct local attributes. This is significant for the challenging tasks of zero-shot learning (ZSL) and fine-grained visual classification (FGVC), where the discriminative attributes of objects have played an important role. Inspired by human visual attention, neural networks have widely exploited the attention mechanism to learn the locally discriminative attributes for challenging tasks. Though greatly promoted the development of these fields, existing works mainly focus on learning the region embeddings of different attribute features and neglect the importance of discriminative attribute localization. It is also unclear whether the learned attention truly matches the real human attention. To tackle this problem, this paper proposes to employ real human gaze data for visual recognition networks to learn from human attention. Specifically, we design a unified Attribute Attention Network (A 2 Net) that learns from human attention for both ZSL and FGVC tasks. The overall model consists of an attribute attention branch and a baseline classification network. On top of the image feature maps provided by the baseline classification network, the attribute attention branch employs attribute prototypes to produce attribute attention maps and attribute features. The attribute attention maps are converted to gaze-like attentions to be aligned with real human gaze attention. To guarantee the effectiveness of attribute feature learning, we further align the extracted attribute features with attribute-defined class embeddings. To facilitate learning from human gaze attention for the visual recognition problems, we design a bird classification game to collect real human gaze data using the CUB dataset via an eye-tracker device. Experiments on ZSL and FGVC tasks without/with real human gaze data validate the benefits and accuracy of our proposed model. This work supports the promising benefits of collecting human gaze datasets and automatic gaze estimation algorithms learning from human attention for high-level computer vision tasks.

Video Salient Object Detection Via Multi-level Spatiotemporal Bidirectional Network Using Multi-scale Transfer Learning

Video saliency prediction aims to resemble human visual attention by identifying the most relevant and significant elements in a video frame or sequence. This task becomes notably intricate in scenarios characterized by dynamic elements such as rapid motion, occlusions, blur, background variations, and nonrigid deformations. Therefore, the inherent complexity of human visual attention behavior during dynamic scenes necessitates the assessment of both temporal and spatial data. Existing video saliency frameworks often falter under such conditions, and relying solely on image saliency models neglects crucial temporal information in videos. This study presents a new Video Salient Object Detection via Multi-level Spatiotemporal Bidirectional Network using Multi-scale Transfer Learning (MSB-Net) to address the problem of identifying significant objects in videos. The proposed MSB-Net achieves notable results for a given sequence of frames by employing multi-scale transfer learning with an encoder and decoder approach to acquire knowledge and saliency map attributes spatially and temporally. The proposed MSB-Net model has bidirectional LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network) components. The VGG16 (Video Geometry Group) and VGG19 architectures extract multi-scale features from the input video frames. Evaluation of diverse datasets, namely DAVIS-T, SegTrack-V2, ViSal, VOS-T, and DAVSOD-T, demonstrates the model's effectiveness, outperforming other competitive models based on parameters such as MAE, F-measure, and S-measure.

Introducing Mplots: scaling time series recurrence plots to massive datasets

Time series similarity matrices (informally, recurrence plots or dot-plots), are useful tools for time series data mining. They can be used to guide data exploration, and various useful features can be derived from them and then fed into downstream analytics. However, time series similarity matrices suffer from very poor scalability, taxing both time and memory requirements. In this work, we introduce novel ideas that allow us to scale the largest time series similarity matrices that can be examined by several orders of magnitude. The first idea is a novel algorithm to compute the matrices in a way that removes dependency on the subsequence length. This algorithm is so fast that it allows us to now address datasets where the memory limitations begin to dominate. Our second novel contribution is a multiscale algorithm that computes an approximation of the matrix appropriate for the limitations of the user’s memory/screen-resolution, then performs a local, just-in-time recomputation of any region that the user wishes to zoom-in on. Given that this largely removes time and space barriers, human visual attention then becomes the bottleneck. We further introduce algorithms that search massive matrices with quadrillions of cells and then prioritize regions for later examination by either humans or algorithms. We will demonstrate the utility of our ideas for data exploration, segmentation, and classification in domains as diverse as astronomy, bioinformatics, entomology, and wildlife monitoring.

Urban Perception Assessment from Street View Images Based on a Multifeature Integration Encompassing Human Visual Attention

Urban perception is a multifaceted process, spanning from vision to cognitive interpretation. Earlier studies on urban perception assessment have primarily concentrated on the analysis of graphic features of street view images such as color or semantic elements, neglecting the vital human sensory fixation features in cognitive processes. This study leveraged eye-tracking technology to accumulate human visual attention data during emotion scoring of street view images. Additionally, a deep learning network model was proposed to forecast human visual attention areas by assimilating environmental features such as image color, depth, and semantic elements, and subsequently amalgamating the aforementioned environmental features and visual attention to establish a perception prediction model, capable of predicting urban perception on a larger scale. Finally, the reliability and superiority of the urban perception prediction method were confirmed through ablation experiments and bias analysis. On the one hand, the method enhances the prediction accuracy of urban perception; on the other, it investigates the comprehensive influence of multiple features on human subjective emotions from the external environment perspective and individual perspective. This approach transcends the former limited reliance on single-image features, providing a novel and scientific approach for assessing urban perception.

Video saliency prediction for First-Person View UAV videos: Dataset and benchmark

Visual saliency prediction plays a crucial role in Unmanned Aerial Vehicle (UAV) video analysis tasks. In this paper, an eye-tracking dataset of the immersive viewing of videos captured from a First-Person View (FPV) of UAVs is developed, which consists of 200 video clips captured by DJI FPV drones, with a resolution of 4K QHD. The videos cover six different genres and fourteen unique scenes. To study human visual attention in watching FPV videos, fixation points are recorded using an eye tracker integrated into a VR headset. Based on the dataset, a simple yet effective FPV UAV video Saliency prediction model (FUAVSal) is proposed as a baseline, considering spatial–temporal feature, camera motion information and FPV prior. To establish benchmarks for saliency prediction in immersive FPV UAV video viewing, sixteen computational models are evaluated on this dataset. Detailed quantitative and qualitative comparisons are provided. The developed dataset and benchmarks aim to facilitate research on visual saliency prediction for First-Person View UAV videos.

Human gaze prediction for 3D light field display based on multi-attention fusion network

Existing methods for simulating human visual attention primarily focus on 2D displays and limited research has been conducted on predicting visual attention in three-dimensional (3D) light field content. 3D light field displays provide a heightened sense of stereoscopic realism to viewers. To ensure that the content of the 3D light field display appears more consistent with human visual characteristics, we proposed a novel method for predicting human eye fixation in 3D light field display images. Firstly, we collected real eye movement data and utilized it to create an eye movement dataset based on 3D light field display images. This solves the problem of missing datasets in the field of human gaze based on three-dimensional light field images. Then, we proposed a convolutional neural network model with multiple inputs and outputs, integrating attention modules. This model was trained and used to predict eye fixation within the constructed eye movement dataset. A correlation exists between predicted human gaze of multiple distinct views of same light field image. Finally, we predicted the human gaze area of light field multi-view images based on our model. Experimental results demonstrate that our model accurately predicts human gaze regions across different views of a 3D light field image. The human gaze predicted by the model on each view is basically consistent and relatively accurate. By leveraging proposed method, we can effectively anticipate where viewers will focus their attention on the 3D light field display, which is beneficial for targeted improvement of 3D light field display content.

Comparison of clinical geneticist and computer visual attention in assessing genetic conditions.

Artificial intelligence (AI) for facial diagnostics is increasingly used in the genetics clinic to evaluate patients with potential genetic conditions. Current approaches focus on one type of AI called Deep Learning (DL). While DL- based facial diagnostic platforms have a high accuracy rate for many conditions, less is understood about how this technology assesses and classifies (categorizes) images, and how this compares to humans. To compare human and computer attention, we performed eye-tracking analyses of geneticist clinicians (n = 22) and non-clinicians (n = 22) who viewed images of people with 10 different genetic conditions, as well as images of unaffected individuals. We calculated the Intersection-over-Union (IoU) and Kullback-Leibler divergence (KL) to compare the visual attentions of the two participant groups, and then the clinician group against the saliency maps of our deep learning classifier. We found that human visual attention differs greatly from DL model's saliency results. Averaging over all the test images, IoU and KL metric for the successful (accurate) clinician visual attentions versus the saliency maps were 0.15 and 11.15, respectively. Individuals also tend to have a specific pattern of image inspection, and clinicians demonstrate different visual attention patterns than non-clinicians (IoU and KL of clinicians versus non-clinicians were 0.47 and 2.73, respectively). This study shows that humans (at different levels of expertise) and a computer vision model examine images differently. Understanding these differences can improve the design and use of AI tools, and lead to more meaningful interactions between clinicians and AI technologies.

Eye-Tracking Analysis of Proposed Signage Design to Prevent Accidents Caused by the Abrupt Appearance of Dividers on Indian Roads

The road transport system is expanding considerably in developing countries. Villages are connecting to major cities for business, education, health, and many other reasons because of road development and smooth transportation. There has been a rise in the number of road accidents observed, caused by abruptly appearing dividers on roads and a lack of required signage systems. This paper discusses scenarios of accidents due to such abruptly appearing dividers and offers a strategy to design appropriate signage to avoid road accidents in the future. It has been observed that permanent or movable arbitrary fixtures, such as a barricade or a small partition block wall, are installed to separate lanes, in addition to white-colored stripes that are typically employed for lane separation on roads. These fixtures, although they are intended as lane-dividing solutions on roads, cause serious, and at times, fatal accidents, due their sudden, uninitiated appearance on the road. To address this problem, alternative signage designs were designed and tested on Indian roads, based on human cognitive responses and visual attention analysis on signage using an eye-tracking method. In addition, the semantic quality and legibility of alternate signage designs were evaluated using a questionnaire to judge their overall efficacy. Hence, the best signage design solution is proposed for implementation near or before occurrences of road dividers to avoid accidents.

Research on image classification leveraging deep convolutional neural networks and visual cognition

The field of image classification has experienced remarkable improvements with the advent of deep learning techniques, especially Deep Convolutional Neural Networks. The present study provides an extensive exploration of the junction where image classification based on Deep Convolutional Neural Networks meets human visual cognition. Utilizing the inherent ability of these networks to automatically learn hierarchical features from raw pixel data, this research examines their potential in classifying images from diverse complex datasets, emphasizing predominantly on the extensively utilized ImageNet dataset. The initial aspect of this study involves training and evaluating models based on Deep Convolutional Neural Networks on the ImageNet dataset, which comprises millions of labeled images spanning across thousands of categories. Well-established network architectures such as AlexNet, VGGNet, GoogLeNet, and ResNet are employed, and their performance in the challenging task of image classification is assessed. Rigorous experiments highlight the strengths and weaknesses of each model while emphasizing the prospects of transfer learning and fine-tuning. Following this, the interpretability of Deep Convolutional Neural Networks is explored by using visualization techniques to comprehend the learned feature representations. By visualizing activation maps and class-specific saliency maps, valuable insights are gained into the regions of interest that guide the decision-making of these models. Moreover, the correlation between the features extracted by these models and human visual attention mechanisms is examined to shed light on the focus of attention of the models. The study also addresses the difficulties that adversarial attacks, data bias, and generalization capabilities present to Deep Convolutional Neural Networks. Strategies to enhance the robustness and adaptability of the models across various domains are examined, linking these observations to human cognitive behavior.

BI-AVAN: A Brain-Inspired Adversarial Visual Attention Network for Characterizing Human Visual Attention from Neural Activity

BI-AVAN: A Brain-Inspired Adversarial Visual Attention Network for Characterizing Human Visual Attention from Neural Activity

I like the way you move: how animate motion affects visual attention in early human infancy.

The ability to detect animates (as compared with inanimates) rapidly is advantageous for human survival. Due to its relevance, not only the adult human brain has evolved specific neural mechanisms to discriminate animates, but it has been proposed that selection finely tuned the human visual attention system to prioritize visual cues that signal the presence of living things. Among them, animate motion-i.e., the motion of animate entities -, is one of the most powerful cues that triggers humans' attention. From a developmental point of view, whether such specialization is inborn or acquired through experience is a fascinating research topic. This mini-review aims to summarize and discuss recent behavioral and electrophysiological research that suggests that animate motion has an attentional advantage in the first year of life starting from birth. Specifically, the rationale underlying this paper concerns how attention deployment is affected by animate motion conveyed both by the movement of a single dot and, also, when the single dot is embedded in a complex array, named biological motion. Overall, it will highlight the importance of both inborn predispositions to pay attention preferentially to animate motion, mainly supported by subcortical structures, and the exposure to certain experiences, shortly after birth, to drive the cortical attentional visual system to become the way it is in adults.

Assessing target optical camouflage effects using brain functional networks: A feasibility study

Brain functional networks model the brain's ability to exchange information across different regions, aiding in the understanding of the cognitive process of human visual attention during target searching, thereby contributing to the advancement of camouflage evaluation. In this study, images with various camouflage effects were presented to observers to generate electroencephalography (EEG) signals, which were then used to construct a brain functional network. The topological parameters of the network were subsequently extracted and input into a machine learning model for training. The results indicate that most of the classifiers achieved accuracy rates exceeding 70%. Specifically, the Logistic algorithm achieved an accuracy of 81.67%. Therefore, it is possible to predict target camouflage effectiveness with high accuracy without the need to calculate discovery probability. The proposed method fully considers the aspects of human visual and cognitive processes, overcomes the subjectivity of human interpretation, and achieves stable and reliable accuracy.

Crop disease recognition based on the YOLOv5 and transfer learning

Crop disease detection is an important factor in agricultural production. Traditional object detection methods can't effectively screen key features, resulting in weak crop disease control in many countries. In recent years, several convolutional neural networks for object detection have been proposed, which makes it possible to apply computer vision to crop disease identification through deep learning. YOLOv5 is an advanced object detection network, which can extract key features and use human visual attention mechanism as a reference. This paper would evaluate the performance of four pre-trained models of YOLOv5 in object detection of crop diseases. And transfer learning was used to train the corresponding dataset. The experiment results have showed that the F1 values of the four models all reached above 0.93, and the Yolov5x got the best result, which achieved 0.963. Furthermore, the detection accuracy of the four models has reached more than 98%. This shows that the YOLOv5 series network models have great application prospects in the identification of crop diseases. In the near future, the object detection model can be applied to various mobile devices, even unmanned aerial vehicles, which would play a significant role in crop disease prevention.

An Infrared Small Target Detection Method Based on Attention Mechanism.

The human visual attention system plays an important role in infrared target recognition because it can quickly and accurately recognize infrared small targets and has good scene adaptability. This paper proposes an infrared small target detection method based on an attention mechanism, which consists of three modules: a bottom-up passive attention module, a top-down active attention module, and decision feedback equalization. In the top-down active attention module, given the Gaussian characteristics of infrared small targets, the idea of combining knowledge-experience Gaussian shape features is applied to implement feature extraction, and quaternion cosine transform is performed to achieve multi-dimensional fusion of Gaussian shape features, thereby achieving complementary fusion of multi-dimensional feature information. In the bottom-up passive attention module, considering that the difference in contrast and motion between the target and the background can attract attention easily, an optimal fast local contrast algorithm and improved circular pipeline filtering are adopted to find candidate target regions. Meanwhile, the multi-scale Laplacian of the Gaussian filter is adopted to estimate the optimal size of the infrared small target. The fast local contrast algorithm based on box filter acceleration and structure optimization is employed to extract local contrast features, and candidate target regions can be obtained by using an adaptive threshold. Besides, the mean gray, target size, Gaussian consistency, and circular region constraint are used in pipeline filtering to extract motion regions, and the false-alarm rate is reduced effectively. Finally, decision feedback equalization is adopted to obtain real targets. Experiments are conducted on some real infrared images involving complex backgrounds with sea, sky, and ground clutters, and the experimental results indicate that the proposed method can achieve better detection performance than conventional baseline methods, such as RLCM, ILCM, PQFT, MPCM, and ADMD. Also, mathematical proofs are provided to validate the proposed method.

Spatio-Temporal Feature Pyramid Interactive Attention Network for Egocentric Gaze Prediction

Over the past few years, gaze prediction technology has rapidly developed in human visual attention mechanism research. However, there is still a large gap between the prediction models and the human visual system (HVS). This work makes four contributions to video egocentric gaze prediction research. First, we introduce a new benchmark named ECVG (egocentric video gaze), which consists of 3K high-quality, elaborately selected video sequences spanning a large range of scenes. Existing eye-tracking datasets limit human activities. In contrast, ECVG achieves the collection of eye movement data under real human activity. Second, we establish a visual attention-selection and gaze-tracking model, which summarizes the human visual attention process. Third, we propose a novel feature pyramid interactive attention 3D network (FPIANet) for egocentric gaze prediction, which directly captures the long-term relationship between spatiotemporal features at different time steps and achieves feature nonlocal interactions across temporal, spatial, and scales. In addition, we propose a multiscale interactive attention (MSIA) module, which first explicitly integrates the human top-down and bottom-up visual attention mechanisms to transform any spatiotemporal feature into another feature of the same size but with richer contexts. Furthermore, we propose a new gaze transfer path consistency evaluation and thoroughly examine the performance of our model on two large datasets (ECVG and LEDOV). The experimental results show that our model outperforms the existing state-of-the-art methods.

Saliency prediction in 360° architectural scenes: Performance and impact of daylight variations

Saliency models are image-based prediction models that estimate human visual attention. Such models, when applied to architectural spaces, could pave the way for design decisions where visual attention is taken into account. In this study, we tested the performance of eleven commonly used saliency models that combine traditional and deep learning methods on 126 rendered interior scenes with associated head tracking data. The data was extracted from three experiments conducted in virtual reality between 2016 and 2018. Two of these datasets pertain to the perceptual effects of daylight and include variations of daylighting conditions for a limited set of interior spaces, thereby allowing to test the influence of light conditions on human head movement. Ground truth maps were extracted from the collected head tracking logs, and the prediction accuracy of the models was tested via the correlation coefficient between ground truth and prediction maps. To address the possible inflation of results due to the equator bias, we conducted complementary analyses by restricting the area of investigation to the equatorial image regions. Although limited to immersive virtual environments, the promising performance of some traditional models such as GBVS360eq and BMS360eq for colored and textured architectural rendered spaces offers us the prospect of their possible integration into design tools. We also observed a strong correlation in head movements for the same space lit by different types of sky, a finding whose generalization requires further investigations based on datasets more specifically developed to address this question.

Testing a relational account of search templates in visual foraging

Search templates guide human visual attention toward relevant targets. Templates are often seen as encoding exact target features, but recent studies suggest that templates rather contain “relational properties” (e.g., they facilitate “redder” stimuli instead of specific hues of red). Such relational guidance seems helpful in naturalistic searches where illumination or perspective renders exact feature values unreliable. So far relational guidance has only been demonstrated in rather artificial single-target search tasks with briefly flashed displays. Here, we investigate whether relational guidance also occurs when humans interact with the search environment for longer durations to collect multiple target elements. In a visual foraging task, participants searched for and collected multiple targets among distractors of different relationships to the target colour. Distractors whose colour differed from the environment in the same direction as the targets reduced foraging efficiency to the same amount as distractors whose colour matched the target colour. Distractors that differed by the same colour distance but in the opposite direction of the target colour did not reduce efficiency. These findings provide evidence that search templates encode relational target features in naturalistic search tasks and suggest that attention guidance based on relational features is a common mode in dynamic, real-world search environments.

Temporal information oriented motion accumulation and selection network for RGB-based action recognition

Numerous studies have highlighted the crucial role of motion information in accurate action recognition in videos. However, current methods heavily rely on temporal differences of features extracted by convolutional neural networks (CNNs) to represent motion, which may have two potential limitations: (1) incomplete representation of the moving target contour due to the difference operation, and (2) equal treatment of all extracted motion features, regardless of their relevance to the classification task, which may negatively impact performance. To address these limitations, we propose a novel approach called the Motion Accumulation and Selection Network (MAS-Net). Although our new approach also considers spatial attributes, it draws inspiration from the cumulative and selective nature of human visual attention, with a primary focus on capturing the temporal attributes of actions for recognition. Further, an motion selection module is exploited to prioritize relevant temporal features while filtering out irrelevant ones. Currently, there is a growing demand for action recognition with strong temporal information, as opposed to conventional scene-related datasets such as UCF-101 and HMDB-51. Therefore, we evaluated MAS-Net on benchmark video datasets that primarily emphasize temporal information, including Something-Something V1&V2, Diving48, and Kinetics-400. Our experimental results demonstrate that MAS-Net achieves state-of-the-art performance on Something-Something V1&V2 and Diving48 datasets. Furthermore, when compared to other 2D CNN-based models, MAS-Net exhibits competitive results on the Kinetics-400 dataset while maintaining computational efficiency. These findings highlight the effectiveness and efficiency of MAS-Net for temporal modeling in video analysis tasks.

Self-attention in vision transformers performs perceptual grouping, not attention

Recently, a considerable number of studies in computer vision involve deep neural architectures called vision transformers. Visual processing in these models incorporates computational models that are claimed to implement attention mechanisms. Despite an increasing body of work that attempts to understand the role of attention mechanisms in vision transformers, their effect is largely unknown. Here, we asked if the attention mechanisms in vision transformers exhibit similar effects as those known in human visual attention. To answer this question, we revisited the attention formulation in these models and found that despite the name, computationally, these models perform a special class of relaxation labeling with similarity grouping effects. Additionally, whereas modern experimental findings reveal that human visual attention involves both feed-forward and feedback mechanisms, the purely feed-forward architecture of vision transformers suggests that attention in these models cannot have the same effects as those known in humans. To quantify these observations, we evaluated grouping performance in a family of vision transformers. Our results suggest that self-attention modules group figures in the stimuli based on similarity of visual features such as color. Also, in a singleton detection experiment as an instance of salient object detection, we studied if these models exhibit similar effects as those of feed-forward visual salience mechanisms thought to be utilized in human visual attention. We found that generally, the transformer-based attention modules assign more salience either to distractors or the ground, the opposite of both human and computational salience. Together, our study suggests that the mechanisms in vision transformers perform perceptual organization based on feature similarity and not attention.