Saliency Model Research Articles

Vision-Based On-Road Nighttime Vehicle Detection and Tracking Using Improved HOG Features.

The lack of discernible vehicle contour features in low-light conditions poses a formidable challenge for nighttime vehicle detection under hardware cost constraints. Addressing this issue, an enhanced histogram of oriented gradients (HOGs) approach is introduced to extract relevant vehicle features. Initially, vehicle lights are extracted using a combination of background illumination removal and a saliency model. Subsequently, these lights are integrated with a template-based approach to delineate regions containing potential vehicles. In the next step, the fusion of superpixel and HOG (S-HOG) features within these regions is performed, and the support vector machine (SVM) is employed for classification. A non-maximum suppression (NMS) method is applied to eliminate overlapping areas, incorporating the fusion of vertical histograms of symmetrical features of oriented gradients (V-HOGs). Finally, the Kalman filter is utilized for tracking candidate vehicles over time. Experimental results demonstrate a significant improvement in the accuracy of vehicle recognition in nighttime scenarios with the proposed method.

Distracted by Previous Experience: Integrating Selection History, Current Task Demands and Saliency in an Algorithmic Model

AbstractAttention can be biased by previous learning and experience. We present an algorithmic-level model of this selection history bias in visual attention that predicts quantitatively how stimulus-driven processes, goal-driven control and selection history compete to control attention. In the model, the output of saliency maps as stimulus-driven guidance interacts with a history map that encodes learning effects and a goal-driven task control to prioritize visual features. The model works on coded features rather than image pixels which is common in many traditional saliency models. We test the model on a reaction time (RT) data from a psychophysical experiment. The model accurately predicts parameters of reaction time distributions from an integrated priority map that is comprised of an optimal, weighted combination of separate maps. Analysis of the weights confirms selection history effects on attention guidance. The model is able to capture individual differences between participants’ RTs and response probabilities per group. Moreover, we demonstrate that a model with a reduced set of maps performs worse, indicating that integrating history, saliency and task information are required for a quantitative description of human attention. Besides, we show that adding intertrial effect to the model (as another lingering bias) improves the model’s predictive performance.

Comparison of clinical geneticist and computer visual attention in assessing genetic conditions.

Artificial intelligence (AI) for facial diagnostics is increasingly used in the genetics clinic to evaluate patients with potential genetic conditions. Current approaches focus on one type of AI called Deep Learning (DL). While DL- based facial diagnostic platforms have a high accuracy rate for many conditions, less is understood about how this technology assesses and classifies (categorizes) images, and how this compares to humans. To compare human and computer attention, we performed eye-tracking analyses of geneticist clinicians (n = 22) and non-clinicians (n = 22) who viewed images of people with 10 different genetic conditions, as well as images of unaffected individuals. We calculated the Intersection-over-Union (IoU) and Kullback-Leibler divergence (KL) to compare the visual attentions of the two participant groups, and then the clinician group against the saliency maps of our deep learning classifier. We found that human visual attention differs greatly from DL model's saliency results. Averaging over all the test images, IoU and KL metric for the successful (accurate) clinician visual attentions versus the saliency maps were 0.15 and 11.15, respectively. Individuals also tend to have a specific pattern of image inspection, and clinicians demonstrate different visual attention patterns than non-clinicians (IoU and KL of clinicians versus non-clinicians were 0.47 and 2.73, respectively). This study shows that humans (at different levels of expertise) and a computer vision model examine images differently. Understanding these differences can improve the design and use of AI tools, and lead to more meaningful interactions between clinicians and AI technologies.

Stakeholders and issue management utilization in mining industry: Case study warehouse PT Freeport Indonesia

This research proposed the improvement of warehouse operations by employing a comprehensive analysis of stakeholders and implementing an issue management framework. The three-dimensional model of stakeholder identification and salience is applied to evaluate the attributes of power, legitimacy, and urgency among key stakeholders: customer, warehouse crew, and transportation crew. Utilizing tools like Empathy Map and Stakeholder Maps, the study gains insight into stakeholder perspectives and relationships within the material management environment. The analysis explains the crucial role of the warehouse crew in connecting and managing relationships. The Issue Management Framework, supported with Empathy Maps, is used to identify and address potential issues, while Stakeholder Maps aid in prioritizing solutions. The research concludes with a strategic plan and providing a holistic approach to potential challenges. The findings emphasize proactive measures to prevent stockouts and lateness of material services, proposing a direct material delivery system and regular training programs for customers to enhance understanding and reduce errors in material orders.

The Gaze of Schizophrenia Patients Captured by Bottom-up Saliency

Schizophrenia (SCHZ) notably impacts various human perceptual modalities, including vision. Prior research has identified marked abnormalities in perceptual organization in SCHZ, predominantly attributed to deficits in bottom-up processing. Our study introduces a novel paradigm to differentiate the roles of top-down and bottom-up processes in visual perception in SCHZ. We analysed eye-tracking fixation ground truth maps from 28 SCHZ patients and 25 healthy controls (HC), comparing these with two mathematical models of visual saliency: one bottom-up, based on the physical attributes of images, and the other top-down, incorporating machine learning. While the bottom-up (GBVS) model revealed no significant overall differences between groups (beta = 0.01, p = 0.281, with a marginal increase in SCHZ patients), it did show enhanced performance by SCHZ patients with highly salient images. Conversely, the top-down (EML-Net) model indicated no general group difference (beta = −0.03, p = 0.206, lower in SCHZ patients) but highlighted significantly reduced performance in SCHZ patients for images depicting social interactions (beta = −0.06, p < 0.001). Over time, the disparity between the groups diminished for both models. The previously reported bottom-up bias in SCHZ patients was apparent only during the initial stages of visual exploration and corresponded with progressively shorter fixation durations in this group. Our research proposes an innovative approach to understanding early visual information processing in SCHZ patients, shedding light on the interplay between bottom-up perception and top-down cognition.

SalDA: DeepConvNet Greets Attention for Visual Saliency Prediction

Predicting salient regions in images requires the capture of contextual information in the scene. Conventional saliency models typically use the encoder-decoder architecture and multi-scale feature fusion for modeling contextual features, which, however, possess huge computational cost and model parameters. In this paper, we address the saliency prediction task by capturing long-range dependencies based on the self-attention mechanism. Self-attention has been widely used in image recognition or other classification tasks, but is still rarely being considered in regression based saliency prediction task. Inspired by non-local block, we propose a new SALiency prediction network in which DeepConvNet is integrated with the Attention mechanism, namely SalDA. Considering each feature map may capture different salient regions, our spatial attention module firstly adaptively aggregates the feature at each position by a weighted sum of the features at all positions within each independent channel. Meanwhile, in order to capture interdependence between channels, we also introduce a channel attention module to integrate different features among different channels. We combine these two attention modules into a multi-attention module to further improve the saliency map prediction for the network. We show the effectiveness of SalDA on the largest saliency prediction dataset SALICON. Compared to other state-of-the-art methods in this area, we can yield comparable saliency prediction performance, but with substantially less model parameters and shorter inference time.

Patterns of saliency and semantic features distinguish gaze of expert and novice viewers of surveillance footage

When viewing the actions of others, we not only see patterns of body movements, but we also "see" the intentions and social relations of people. Experienced forensic examiners – Closed Circuit Television (CCTV) operators – have been shown to convey superior performance in identifying and predicting hostile intentions from surveillance footage than novices. However, it remains largely unknown what visual content CCTV operators actively attend to, and whether CCTV operators develop different strategies for active information seeking from what novices do. Here, we conducted computational analysis for the gaze-centered stimuli captured by experienced CCTV operators and novices' eye movements when viewing the same surveillance footage. Low-level image features were extracted by a visual saliency model, whereas object-level semantic features were extracted by a deep convolutional neural network (DCNN), AlexNet, from gaze-centered regions. We found that the looking behavior of CCTV operators differs from novices by actively attending to visual contents with different patterns of saliency and semantic features. Expertise in selectively utilizing informative features at different levels of visual hierarchy may play an important role in facilitating the efficient detection of social relationships between agents and the prediction of harmful intentions.

Assessment of Socio-Economic Characteristics and Drivers of Deforestation of Mandara Hills in Madagali Local Government Area of Adamawa State

There is sufficient proof that the entire world is fronting an ecological disaster because of huge deforestation. This study assessed the drivers of deforestation and the socio-economic characteristics of the respondents. Information was sourced by administering questionnaires to 200 respondents drawn from the support zone dwellers of Mandara hills in Madagali Local Government Area of Adamawa State, and alongside interviews scheduled using participatory rural appraisal technique. Data collected were analyzed using descriptive statistics and smith’s saliency model. The later was used for ranking of benefits derived from the hills and the conservation measures proffered by the respondents. The following age classes 26-30 years with 30 respondents, 31-35 years (35), 36-40 years (32) and 41-45 years (30) had the majority of the respondents. On gender, the majority (84.50%) of the respondents were males. Percentage of respondents for civil servants (21.50%) and farming (27.50%). The mainstream of the respondents had formal education represented by primary (19.5%), secondary (24.50%) and tertiary (36.0%). Most of the respondents had income that ranged from N100,000.00 - N2,000,000.00. The major benefits derived from the hills included; human protection during wars and insurgency (32.50%) and farming (30.50%). Drivers of deforestation based on; bad, worse and worst effect rankings cumulatively showed agricultural expansion topping the list, followed by indiscriminate destruction of wildlife habitat. Conservation measures suggested in order of frequency of respondents included; reduced human settlement around the hills (34.50%) and implementation of natural forest regeneration and restoration (31.50%).

Salience effect and yield curve

This paper studies the influence of salience on nominal and real yield curves by introducing the salience effect in the Piazzesi and Schneider model (hereafter PS). We construct the salience values based on the expected consumption growth using U.S. data. We find that salience values are negatively correlated with the expected consumption growth rates. Based on U.S. data from 1960q1 to 2020q4, we find that the salience model can generate upward nominal and real yield curves within reasonable risk aversion coefficients (less than 10), as well as well-fitted average yields with actual data. The salience model compensates for the PS or recursive preference model’s inability to generate an upward nominal or real yield curve within reasonable risk aversion. Furthermore, we provide empirical support for model implications.

Predicting Radiologists' Gaze With Computational Saliency Models in Mammogram Reading

Previous studies have shown that there is a strong correlation between radiologists' diagnoses and their gaze when reading medical images. The extent to which gaze is attracted by content in a visual scene can be characterised as visual saliency. There is a potential for the use of visual saliency in computer-aided diagnosis in radiology. However, little is known about what methods are effective for diagnostic images, and how these methods could be adapted to address specific applications in diagnostic imaging. In this study, we investigate 20 state-of-the-art saliency models including 10 traditional models and 10 deep learning-based models in predicting radiologists' visual attention while reading 196 mammograms. We found that deep learning-based models represent the most effective type of methods for predicting radiologists' gaze in mammogram reading; and that the performance of these saliency models can be significantly improved by transfer learning. In particular, an enhanced model can be achieved by pre-training the model on a large-scale natural image saliency dataset and then fine-tuning it on the target medical image dataset. In addition, based on a systematic selection of backbone networks and network architectures, we proposed a parallel multi-stream encoded model which outperforms the state-of-the-art approaches for predicting saliency of mammograms.

How is Visual Attention Influenced by Text Guidance? Database and Model.

The analysis and prediction of visual attention have long been crucial tasks in the fields of computer vision and image processing. In practical applications, images are generally accompanied by various text descriptions, however, few studies have explored the influence of text descriptions on visual attention, let alone developed visual saliency prediction models considering text guidance. In this paper, we conduct a comprehensive study on text-guided image saliency (TIS) from both subjective and objective perspectives. Specifically, we construct a TIS database named SJTU-TIS, which includes 1200 text-image pairs and the corresponding collected eye-tracking data. Based on the established SJTU-TIS database, we analyze the influence of various text descriptions on visual attention. Then, to facilitate the development of saliency prediction models considering text influence, we construct a benchmark for the established SJTU-TIS database using state-of-the-art saliency models. Finally, considering the effect of text descriptions on visual attention, while most existing saliency models ignore this impact, we further propose a text-guided saliency (TGSal) prediction model, which extracts and integrates both image features and text features to predict the image saliency under various text-description conditions. Our proposed model significantly outperforms the state-of-the-art saliency models on both the SJTU-TIS database and the pure image saliency databases in terms of various evaluation metrics. The SJTU-TIS database and the code of the proposed TGSal model will be released at: https://github.com/IntMeGroup/TGSal.

Fabric defect detection via saliency model based on adjacent context coordination and transformer

Fabric defect detection is a pivotal step in quality control in the textile manufacturing industry. Due to the diversity and complexity of defects, manual visual inspection and traditional fabric defect detection methods suffer from low efficiency and accuracy. To address the issues, a saliency model capable of mining local and global information from CNN and vision Transformer is proposed for fabric defect detection in this paper, named ACCTNet. Specifically, to enhance the feature interaction of different scales, an adjacent context coordination module composed of one local branch and two adjacent branches is proposed. Meanwhile, a contrast-aggregation module is proposed to highlight the defects from low contrast background using pooling and subtraction operations. In addition, vision Transformer is adopted to capture global contextual information with long-range dependencies, which can guide local information to further refines the defect detection results. Experimental results demonstrate that the proposed method can accurately inspect the defects from plain and patterned fabric surfaces, achieving Em values of 78.49% and 97.19% respectively, which significantly surpasses the existing state-of-the-art fabric defect detection methods.

Improved lightweight saliency model based on neural network for non-cooperative spacecraft detection

Improved lightweight saliency model based on neural network for non-cooperative spacecraft detection

Rectify ViT Shortcut Learning by Visual Saliency.

Shortcut learning in deep learning models occurs when unintended features are prioritized, resulting in degenerated feature representations and reduced generalizability and interpretability. However, shortcut learning in the widely used vision transformer (ViT) framework is largely unknown. Meanwhile, introducing domain-specific knowledge is a major approach to rectifying the shortcuts that are predominated by background-related factors. For example, eye-gaze data from radiologists are effective human visual prior knowledge that has the great potential to guide the deep learning models to focus on meaningful foreground regions. However, obtaining eye-gaze data can still sometimes be time-consuming, labor-intensive, and even impractical. In this work, we propose a novel and effective saliency-guided ViT (SGT) model to rectify shortcut learning in ViT with the absence of eye-gaze data. Specifically, a computational visual saliency model (either pretrained or fine-tuned) is adopted to predict saliency maps for input image samples. Then, the saliency maps are used to filter the most informative image patches. Considering that this filter operation may lead to global information loss, we further introduce a residual connection that calculates the self-attention across all the image patches. The experiment results on natural and medical image datasets show that our SGT framework can effectively learn and leverage human prior knowledge without eye-gaze data and achieves much better performance than baselines. Meanwhile, it successfully rectifies the harmful shortcut learning and significantly improves the interpretability of the ViT model, demonstrating the promise of transferring human prior knowledge derived visual saliency in rectifying shortcut learning.

Spatial Structure Constraints for Weakly Supervised Semantic Segmentation.

The image-level label has prevailed in weakly supervised semantic segmentation tasks due to its easy availability. Since image-level labels can only indicate the existence or absence of specific categories of objects, visualization-based techniques have been widely adopted to provide object location clues. Considering class activation maps (CAMs) can only locate the most discriminative part of objects, recent approaches usually adopt an expansion strategy to enlarge the activation area for more integral object localization. However, without proper constraints, the expanded activation will easily intrude into the background region. In this paper, we propose spatial structure constraints (SSC) for weakly supervised semantic segmentation to alleviate the unwanted object over-activation of attention expansion. Specifically, we propose a CAM-driven reconstruction module to directly reconstruct the input image from deep CAM features, which constrains the diffusion of last-layer object attention by preserving the coarse spatial structure of the image content. Moreover, we propose an activation self-modulation module to refine CAMs with finer spatial structure details by enhancing regional consistency. Without external saliency models to provide background clues, our approach achieves 72.7% and 47.0% mIoU on the PASCAL VOC 2012 and COCO datasets, respectively, demonstrating the superiority of our proposed approach. The source codes and models have been made available at https://github.com/NUST-Machine-Intelligence-Laboratory/SSC.

Quality-Aware Selective Fusion Network for V-D-T Salient Object Detection.

Depth images and thermal images contain the spatial geometry information and surface temperature information, which can act as complementary information for the RGB modality. However, the quality of the depth and thermal images is often unreliable in some challenging scenarios, which will result in the performance degradation of the two-modal based salient object detection (SOD). Meanwhile, some researchers pay attention to the triple-modal SOD task, namely the visible-depth-thermal (VDT) SOD, where they attempt to explore the complementarity of the RGB image, the depth image, and the thermal image. However, existing triple-modal SOD methods fail to perceive the quality of depth maps and thermal images, which leads to performance degradation when dealing with scenes with low-quality depth and thermal images. Therefore, in this paper, we propose a quality-aware selective fusion network (QSF-Net) to conduct VDT salient object detection, which contains three subnets including the initial feature extraction subnet, the quality-aware region selection subnet, and the region-guided selective fusion subnet. Firstly, except for extracting features, the initial feature extraction subnet can generate a preliminary prediction map from each modality via a shrinkage pyramid architecture, which is equipped with the multi-scale fusion (MSF) module. Then, we design the weakly-supervised quality-aware region selection subnet to generate the quality-aware maps. Concretely, we first find the high-quality and low-quality regions by using the preliminary predictions, which further constitute the pseudo label that can be used to train this subnet. Finally, the region-guided selective fusion subnet purifies the initial features under the guidance of the quality-aware maps, and then fuses the triple-modal features and refines the edge details of prediction maps through the intra-modality and inter-modality attention (IIA) module and the edge refinement (ER) module, respectively. Extensive experiments are performed on VDT-2048 dataset, and the results show that our saliency model consistently outperforms 13 state-of-the-art methods with a large margin. Our code and results are available at https://github.com/Lx-Bao/QSFNet.

Pixel is All You Need: Adversarial Spatio-Temporal Ensemble Active Learning for Salient Object Detection.

Although weakly-supervised techniques can reduce the labeling effort, it is unclear whether a saliency model trained with weakly-supervised data (e.g., point annotation) can achieve the equivalent performance of its fully-supervised version. This paper attempts to answer this unexplored question by proving a hypothesis: there is a point-labeled dataset where saliency models trained on it can achieve equivalent performance when trained on the densely annotated dataset. To prove this conjecture, we proposed a novel yet effective adversarial spatio-temporal ensemble active learning. Our contributions are four- fold: 1) Our proposed adversarial attack triggering uncertainty can conquer the overconfidence of existing active learning methods and accurately locate these uncertain pixels. 2) Our proposed spatio-temporal ensemble strategy not only achieves outstanding performance but significantly reduces the model's computational cost. 3) Our proposed relationship-aware diversity sampling can conquer oversampling while boosting model performance. 4) We provide theoretical proof for the existence of such a point-labeled dataset. Experimental results show that our approach can find such a point-labeled dataset, where a saliency model trained on it obtained 98%-99% performance of its fully-supervised version with only ten annotated points per image. The code is available at https://github.com/wuzhenyubuaa/ASTE-AL.

Unified Audio-Visual Saliency Model for Omnidirectional Videos With Spatial Audio

Unified Audio-Visual Saliency Model for Omnidirectional Videos With Spatial Audio

Salience and Erasure in Environmental Advertisements: An Ecolinguistic Study

Ecolinguistics is a twofold field in which ecology and language are its two major concerns. That is, this field is concerned with the way through which our thoughts, ideologies and the like influence the environment. The present study aims at analyzing (6) constructive and destructive environmental advertisements to find out how the techniques of erasure and salience operate in these types of advertisements. It studies the linguistic expressions that achieve these techniques in the constructive and destructive advertisements. The qualitative and quantitative methods are exploited in the current study. Analyzing (6) constructive and destructive environmental advertisements in accordance with Stibbe’s (2015) model of salience and erasure accomplishes the qualitative approach whereas accounting the frequencies and percentages of salience and erasure occurrences in constructive and destructive environmental advertisements accomplishes the quantitative approach. The model of the analysis used in the present study is of Stibbe’s (2015) theory of erasure and salience. The main finding shows that the technique of salience is employed more frequently in constructive advertisements than that of erasure, whereas the same techniques are equally used in destructive advertisements. The main conclusion of the present study reads that the technique of salience is appropriate for constructive advertisements since it highlights the important aspects of the environmental advertisements

SODRet: Instance retrieval using salient object detection for self-service shopping

Self-service shopping technologies have become commonplace in modern society. Although various innovative solutions have been adopted, there is still a gap in providing efficient services to consumers. Recent developments in mobile application technologies and internet-of-things devices promote information and knowledge dissemination by integrating innovative services to meet users’ needs. We argue that object retrieval applications can be used to provide effective online or self-service shopping. Therefore, to fill this technological void, this study aims to propose an object retrieval system using a fusion-based salient object detection (SOD) method. The SOD has attracted significant attention, and recently many heuristic computational models have been developed for object detection. It has been widely used in object detection and retrieval applications. This work proposes an instance retrieval system based on the SOD to find the objects from the commodity datasets. A prediction about the object’s position is made using the saliency detection system through a saliency model, and the proposed SOD-based retrieval (SODRet) framework uses saliency maps for retrieving the searched items. The method proposed in this work is evaluated on INSTRE and Flickr32 datasets. Our proposed work outperforms state-of-the-art object retrieval methods and can further be employed for large-scale self-service shopping-based points of sales.