Saliency Model Research Articles

Reactive Gaze during Locomotion in Natural Environments

AbstractAnimating gaze behavior is crucial for creating believable virtual characters, providing insights into their perception and interaction with the environment. In this paper, we present an efficient yet natural‐looking gaze animation model applicable to real‐time walking characters exploring natural environments. We address the challenge of dynamic gaze adaptation by combining findings from neuroscience with a data‐driven saliency model. Specifically, our model determines gaze focus by considering the character's locomotion, environment stimuli, and terrain conditions. Our model is compatible with both automatic navigation through pre‐defined character trajectories and user‐guided interactive locomotion, and can be configured according to the desired degree of visual exploration of the environment. Our perceptual evaluation shows that our solution significantly improves the state‐of‐the‐art saliency‐based gaze animation with respect to the character's apparent awareness of the environment, the naturalness of the motion, and the elements to which it pays attention.

Comparative analysis of saliency map algorithms in capturing visual priorities for building inspections

This study investigates the efficacy of saliency mapping algorithms in capturing the visual priorities of building inspectors for structural damage assessment. Our work established a ground truth dataset by implementing eye-tracking technology to capture the gaze patterns of building inspectors. Further, it enables a detailed evaluation of the saliency models’ ability to reflect experts' visual attention during inspection tasks. Our comparative analysis assesses the performance of three saliency models— EnDec, DeepGaze, and SALICON— against this ground truth data, using conventional saliency metrics such as Area under the Curve, Similarity, Normalized Scanpath Saliency, Correlation Coefficient, and Kullback-Leibler Divergence. Our findings reveal that while the SALICON model demonstrates a marginally better performance and highlights areas where these models fall short, particularly in accurately reflecting the critical visual properties of inspectors, this insight is crucial for advancing the field. By highlighting these limitations, we have drawn attention to the need for developing more specialized saliency models tailored to the unique demands of building inspection tasks. Thus, the study not only fulfills its objectives of comparative analysis but also contributes to the broader discourse on improving automated structural inspection systems. This study highlights the need to develop specialized computer vision models to address specific building inspection challenges. By identifying strengths and improvement areas, this research contributes valuable insights and highlights the potential and current limitations of applying computer vision techniques to real-world building inspection tasks.

Focus-SLAM: A visual Monocular SLAM

Abstract This research presents a novel simultaneous localization and mapping algorithm, called Focus-SLAM, which simulates human navigation strategies by synthesizing a optical saliency model, SalNavNet, in a traditional single-shot SLAM paradigm. SalNavNet introduces an innovative design that incorporates a correlation module and an adaptive Exponential Moving Average (EMA) component, effectively addressing the prevalent center bias issue found in contemporary saliency models. As a result, the system enhances target fixation by accentuating salient features more effectively. Extensive experimental evaluations, conducted across a spectrum of indoor environments exhibiting diverse luminosity conditions. These findings attest to the efficacy of Focus-SLAM in enhancing both the accuracy and efficiency of SLAM operations under challenging real-world conditions.

An egocentric video and eye-tracking dataset for visual search in convenience stores

We introduce an egocentric video and eye-tracking dataset, comprised of 108 first-person videos of 36 shoppers searching for three different products (orange juice, KitKat chocolate bars, and canned tuna) in a convenience store, along with the frame-centered eye fixation locations for each video frame. The dataset also includes demographic information about each participant in the form of an 11-question survey. The paper describes two applications using the dataset — an analysis of eye fixations during search in the store, and a training of a clustered saliency model for predicting saliency of viewers engaged in product search in the store. The fixation analysis shows that fixation duration statistics are very similar to those found in image and video viewing, suggesting that similar visual processing is employed during search in 3D environments and during viewing of imagery on computer screens. A clustering technique was applied to the questionnaire data, which resulted in two clusters being detected. Based on these clusters, personalized saliency prediction models were trained on the store fixation data, which provided improved performance in prediction saliency on the store video data compared to state-of-the art universal saliency prediction methods.

GINet:Graph interactive network with semantic-guided spatial refinement for salient object detection in optical remote sensing images

There are many challenging scenarios in the task of salient object detection in optical remote sensing images (RSIs), such as various scales and irregular shapes of salient objects, cluttered backgrounds, etc. Therefore, it is difficult to directly apply saliency models targeting natural scene images to optical RSIs. Besides, existing models often do not give sufficient exploration for the potential relationship of different salient objects or different parts of the salient object. In this paper, we propose a graph interaction network (i.e. GINet) with semantic-guided spatial refinement to conduct salient object detection in optical RSIs. The key advantages of GINet lie in two points. Firstly, the graph interactive reasoning (GIR) module conducts information exchange of different-level features via the graph interaction operation, and enhances features along spatial and channel dimensions via the graph reasoning operation. Secondly, we designed the global content-aware refinement (GCR) module, which incorporates the foreground and background feature-based local information and the semantic feature-based global information simultaneously. Experiments results on two public optical RSIs datasets clearly show the effectiveness and superiority of the proposed GINet when compared with the state-of-the-art models.

Infrared low-altitude and slow-speed small target detection via fusion of target sparsity and motion saliency

Infrared (IR) small target detection exerts a significant role in IR early warning and UAV surveillance. However, in the low-altitude slow-speed small (LSS) target detection scene, the existing algorithms cannot effectively suppress high-contrast corners and sparse edges in the low-altitude background, resulting in many false alarms. To solve this problem, we propose an IR LSS target detection method based on fusion of target sparsity and motion saliency (TSMS). In the low-rank sparse model, we introduce a robust dual-window gradient operator to construct a fine local prior, which avoids the influence of highlighted edges and corners; The Geman norm is used to approximate the background rank to accurately estimate the background and effectively extract sparse targets. Then, a motion saliency model based on inter-frame local matching is constructed to accurately extract the inter-frame features of small target. Finally, the real LSS target is obtained by fusing target sparsity and motion saliency. Experiments indicate that, compared with existing advanced methods, the proposed method has stronger robustness and can effectively detect LSS targets under complex low-altitude background.

Employing a Computational Model to Reveal Stimulus-Driven Influences in Interfaces

It is challenging to design complex interfaces that support efficient visual searches. It requires understanding how numerous sources of information contribute to attentional selection, along with an understanding of the tools and tasks used to study visual search. This work is intended to examine visual salience, a pre-attentive process often overlooked in traditional design approaches. Although the importance of visual salience in task-driven searches has been debated, recent evidence suggests this stimulus-driven property contributes to the guidance of attention. Computational modeling has proven effective in predicting the influence of salience (e.g., N-SEEV and Attentional Priority models). With advances in this area of research, continuing examination of the applications and limitations of computational salience models is warranted. Therefore, the present work also examines the predictive power of computational models in displays like web pages and mobile interfaces. Emphasis is placed on the role salience should play in optimizing interface search efficiency.

Video Salient Object Detection Via Multi-level Spatiotemporal Bidirectional Network Using Multi-scale Transfer Learning

Video saliency prediction aims to resemble human visual attention by identifying the most relevant and significant elements in a video frame or sequence. This task becomes notably intricate in scenarios characterized by dynamic elements such as rapid motion, occlusions, blur, background variations, and nonrigid deformations. Therefore, the inherent complexity of human visual attention behavior during dynamic scenes necessitates the assessment of both temporal and spatial data. Existing video saliency frameworks often falter under such conditions, and relying solely on image saliency models neglects crucial temporal information in videos. This study presents a new Video Salient Object Detection via Multi-level Spatiotemporal Bidirectional Network using Multi-scale Transfer Learning (MSB-Net) to address the problem of identifying significant objects in videos. The proposed MSB-Net achieves notable results for a given sequence of frames by employing multi-scale transfer learning with an encoder and decoder approach to acquire knowledge and saliency map attributes spatially and temporally. The proposed MSB-Net model has bidirectional LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network) components. The VGG16 (Video Geometry Group) and VGG19 architectures extract multi-scale features from the input video frames. Evaluation of diverse datasets, namely DAVIS-T, SegTrack-V2, ViSal, VOS-T, and DAVSOD-T, demonstrates the model's effectiveness, outperforming other competitive models based on parameters such as MAE, F-measure, and S-measure.

Learning Complementary Spatial-Temporal Transformer for Video Salient Object Detection.

Besides combining appearance and motion information, another crucial factor for video salient object detection (VSOD) is to mine spatial-temporal (ST) knowledge, including complementary long-short temporal cues and global-local spatial context from neighboring frames. However, the existing methods only explored part of them and ignored their complementarity. In this article, we propose a novel complementary ST transformer (CoSTFormer) for VSOD, which has a short-global branch and a long-local branch to aggregate complementary ST contexts. The former integrates the global context from the neighboring two frames using dense pairwise attention, while the latter is designed to fuse long-term temporal information from more consecutive frames with local attention windows. In this way, we decompose the ST context into a short-global part and a long-local part and leverage the powerful transformer to model the context relationship and learn their complementarity. To solve the contradiction between local window attention and object motion, we propose a novel flow-guided window attention (FGWA) mechanism to align the attention windows with object and camera movements. Furthermore, we deploy CoSTFormer on fused appearance and motion features, thus enabling the effective combination of all three VSOD factors. Besides, we present a pseudo video generation method to synthesize sufficient video clips from static images for training ST saliency models. Extensive experiments have verified the effectiveness of our method and illustrated that we achieve new state-of-the-art results on several benchmark datasets.

How will Generative AI impact communication?

This paper examines the impact of Generative AI (GAI) on communication through the lens of salience and signalling models. It explores how GAI affects both senders’ ability to create salient messages and receivers’ costs of absorbing them. The analysis reveals that while GAI can increase communication by reducing costs, it may also disrupt traditional signalling mechanisms. In a salience model, GAI generally improves outcomes but can potentially reduce receiver welfare. In a pure signalling model, GAI may hinder effective communication by making it harder to distinguish high-quality messages. This suggests that GAI’s introduction necessitates new instruments and mechanisms to facilitate effective communication and quality assessment in this evolving landscape.

A hybrid spatiotemporal distribution forecast methodology for IES vulnerabilities under uncertain and imprecise space-air-ground monitoring data scenarios

The weak spots in an integrated energy system that may jeopardize the overall reliability call for timely and efficient Inspection and Maintenance (I&M). One core step is the reasonable allocation and deployment of limited I&M personnel or apparatus to the regions or periods with higher event risks, which requires a pinpoint spatiotemporal distribution forecast of future vulnerabilities. This paper presents a hybrid forecast methodology, the Saliency-Rough Fuzzy Utility Pattern recognition ensemble, in light of space-air-ground multi-source-heterogeneous input data. A parallel learning architecture is established and identifies the critical components with higher yields to enhance efficiency. Accordingly, more reasonable quantitative and qualitative evaluations can be carried out concurrently. Potential imprecise and uncertain data scenes are handled in quantitative assessments, both the failure hazard path sets and survival function likelihood boxes are incorporated in the designed relative path-Fussell Vesely Saliency (rp-FVS) model; and in qualitative analyses, the underlying perilous components can be distinguished via a combination of the variable precision-rough model. The rp-FVS-based fuzzy inference logic configures all membership functions identically according to components’ impacts. These two parts are integrated into the rough-fuzzy Utility Measure to discover concealed component-vulnerability interconnection patterns. Finally, an empirical case study is conducted for validation.

TranSalNet+: Distortion-aware saliency prediction

Predicting the saliency of images affected by distortion is a challenging but emerging research problem. Given a distorted image, we wish to accurately predict saliency as perceived by humans. A recent distortion-aware saliency benchmark – the CUDAS database – reveals the inadequacy of existing saliency models in handling distorted images. In this paper, we devise a deep learning Distortion-Aware Saliency Module (DASM) that enables capturing saliency features related to image distortions, and integrates this module into a saliency prediction architecture. To achieve the high expressive capability of DASM using supervised learning, we create a dedicated dataset that draws upon a large-scale saliency dataset and machine-generated image quality assessments. Experimental results demonstrate the superior performance of the proposed model in predicting the saliency of distorted images.

Scanpath Prediction on Information Visualisations.

We propose Unified Model of Saliency and Scanpaths (UMSS)- a model that learns to predict multi-duration saliency and scanpaths (i.e. sequences of eye fixations) on information visualisations. Although scanpaths provide rich information about the importance of different visualisation elements during the visual exploration process, prior work has been limited to predicting aggregated attention statistics, such as visual saliency. We present in-depth analyses of gaze behaviour for different information visualisation elements (e.g. Title, Label, Data) on the popular MASSVIS dataset. We show that while, overall, gaze patterns are surprisingly consistent across visualisations and viewers, there are also structural differences in gaze dynamics for different elements. Informed by our analyses, UMSS first predicts multi-duration element-level saliency maps, then probabilistically samples scanpaths from them. Extensive experiments on MASSVIS show that our method consistently outperforms state-of-the-art methods with respect to several, widely used scanpath and saliency evaluation metrics. Our method achieves a relative improvement in sequence score of 11.5% for scanpath prediction, and a relative improvement in Pearson correlation coefficient of up to 23.6% for saliency prediction. These results are auspicious and point towards richer user models and simulations of visual attention on visualisations without the need for any eye tracking equipment.

Shifting Focus with HCEye: Exploring the Dynamics of Visual Highlighting and Cognitive Load on User Attention and Saliency Prediction

Visual highlighting can guide user attention in complex interfaces. However, its effectiveness under limited attentional capacities is underexplored. This paper examines the joint impact of visual highlighting (permanent and dynamic) and dual-task-induced cognitive load on gaze behaviour. Our analysis, using eye-movement data from 27 participants viewing 150 unique webpages reveals that while participants' ability to attend to UI elements decreases with increasing cognitive load, dynamic adaptations (i.e., highlighting) remain attention-grabbing. The presence of these factors significantly alters what people attend to and thus what is salient. Accordingly, we show that state-of-the-art saliency models increase their performance when accounting for different cognitive loads. Our empirical insights, along with our openly available dataset, enhance our understanding of attentional processes in UIs under varying cognitive (and perceptual) loads and open the door for new models that can predict user attention while multitasking.

High-Speed Spatial–Temporal Saliency Model: A Novel Detection Method for Infrared Small Moving Targets Based on a Vectorized Guided Filter

Infrared (IR) imaging-based detection systems are of vital significance in the domains of early warning and security, necessitating a high level of precision and efficiency in infrared small moving target detection. IR targets often appear dim and small relative to the background and are easily buried by noise and difficult to detect. A novel high-speed spatial–temporal saliency model (HS-STSM) based on a guided filter (GF) is proposed, which innovatively introduces GF into IR target detection to extract the local anisotropy saliency in the spatial domain, and substantially suppresses the background region as well as the bright clutter false alarms present in the background. Moreover, the proposed model extracts the motion saliency of the target in the temporal domain through vectorization of IR image sequences. Additionally, the proposed model significantly improves the detection efficiency through a vectorized filtering process and effectively suppresses edge components in the background by integrating a prior weight. Experiments conducted on five real infrared image sequences demonstrate the superior performance of the model compared to existing algorithms in terms of the detection rate, noise suppression, real-time processing, and robustness to the background.

Saliency Models Reveal Reduced Top-Down Attention in Attention-Deficit/Hyperactivity Disorder: A Naturalistic Eye-Tracking Study

Saliency Models Reveal Reduced Top-Down Attention in Attention-Deficit/Hyperactivity Disorder: A Naturalistic Eye-Tracking Study

Breast Tumour Segmentation Using Advanced UNet with Saliency, Channel, and Spatial Attention Models

Cancers are getting pretty common these days and in that the second most common cancer in the world after lung cancer is breast cancer. The primary screening techniques for early diagnosis of cancerous nodules in women breast are Ultrasound, Mammography, and MRI analysis. However, accurately identifying and outlining the boundaries of tumor regions remains challenging due to their unpredictable shapes, random variations, and blurry outlines. Currently, the CAD based image analysis systems have gained huge attention in biomedical image processing systems along with Machine Learning (ML) and Deep Learning (DL) methods. The conventional machine learning methods suffer from accuracy related issues therefore deep learning-based schemes have been adopted. Specifically, UNet based approach is widely employed for biomedical image segmentation. This research focuses on development of a UNet based deep learning architecture for breast cancer image analysis and segmentation. The proposed model improves the segmentation accuracy by adding saliency model, channel and spatial attention model to consider the low- and high-level features. These modules help to achieve the clear boundary during segmentation. The performance of proposed model is evaluated by using publicly available mammogram dataset known as DDSM dataset.

Enhancing big data analytics deployment: uncovering stakeholder dynamics and balancing salience in project roles

Deployment constitutes a pivotal aspect of data science projects, such as big data analytics (BDA). A comprehensive definition of successful deployment necessitates the integration of perspectives from both the project stakeholders and the end-users. However, adequate consideration of project stakeholders remains notably absent within the broader view of project deployment. This paper investigates the role of stakeholders in the deployment of BDA projects by applying an ethnographic research design throughout a 12-month period within a large multinational organization. The study employs critical systems heuristics concepts to identify stakeholder roles, which are subsequently classified and analyzed according to the salience model. The empirical findings point towards the missing link between the technical and the business aspects of a BDA project. The organizational function and product management, (capable of comprehending both the technical and business dimensions) must undertake a highly salient stakeholder role to effectively guide the project toward the successful deployment. Additionally, this role will be able to identify the exact beneficiaries, thus enabling them to increase their salience and their interests to resonate across the spectrum of project stakeholders. This study advances the knowledge and understanding of BDA deployment through the lens of a stakeholder perspective and systems thinking. It uncovers the necessary resources by mapping the social roles of a project and assessing their salience. Balancing role-based salience contributes to successful BDA project deployment.

Auxiliary Tasks Enhanced Dual-Affinity Learning for Weakly Supervised Semantic Segmentation.

Most existing weakly supervised semantic segmentation (WSSS) methods rely on class activation mapping (CAM) to extract coarse class-specific localization maps using image-level labels. Prior works have commonly used an off-line heuristic thresholding process that combines the CAM maps with off-the-shelf saliency maps produced by a general pretrained saliency model to produce more accurate pseudo-segmentation labels. We propose AuxSegNet + , a weakly supervised auxiliary learning framework to explore the rich information from these saliency maps and the significant intertask correlation between saliency detection and semantic segmentation. In the proposed AuxSegNet + , saliency detection and multilabel image classification are used as auxiliary tasks to improve the primary task of semantic segmentation with only image-level ground-truth labels. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps. In particular, we propose a cross-task dual-affinity learning module to learn both pairwise and unary affinities, which are used to enhance the task-specific features and predictions by aggregating both query-dependent and query-independent global context for both saliency detection and semantic segmentation. The learned cross-task pairwise affinity can also be used to refine and propagate CAM maps to provide better pseudo labels for both tasks. Iterative improvement of segmentation performance is enabled by cross-task affinity learning and pseudo-label updating. Extensive experiments demonstrate the effectiveness of the proposed approach with new state-of-the-art WSSS results on the challenging PASCAL VOC and MS COCO benchmarks.

Feature Saliencies in Asymmetric Hidden Markov Models.

Many real-life problems are stated as nonlabeled high-dimensional data. Current strategies to select features are mainly focused on labeled data, which reduces the options to select relevant features for unsupervised problems, such as clustering. Recently, feature saliency models have been introduced and developed as clustering models to select and detect relevant variables/features as the model is learned. Usually, these models assume that all variables are independent, which narrows their applicability. This article introduces asymmetric hidden Markov models with feature saliencies, i.e., models capable of simultaneously determining during their learning phase relevant variables/features and probabilistic relationships between variables. The proposed models are compared with other state-of-the-art approaches using synthetic data and real data related to grammatical face videos and wear in ball bearings. We show that the proposed models have better or equal fitness than other state-of-the-art models and provide further data insights.