Computational Saliency Models Research Articles

Predicting Radiologists' Gaze With Computational Saliency Models in Mammogram Reading

Previous studies have shown that there is a strong correlation between radiologists' diagnoses and their gaze when reading medical images. The extent to which gaze is attracted by content in a visual scene can be characterised as visual saliency. There is a potential for the use of visual saliency in computer-aided diagnosis in radiology. However, little is known about what methods are effective for diagnostic images, and how these methods could be adapted to address specific applications in diagnostic imaging. In this study, we investigate 20 state-of-the-art saliency models including 10 traditional models and 10 deep learning-based models in predicting radiologists' visual attention while reading 196 mammograms. We found that deep learning-based models represent the most effective type of methods for predicting radiologists' gaze in mammogram reading; and that the performance of these saliency models can be significantly improved by transfer learning. In particular, an enhanced model can be achieved by pre-training the model on a large-scale natural image saliency dataset and then fine-tuning it on the target medical image dataset. In addition, based on a systematic selection of backbone networks and network architectures, we proposed a parallel multi-stream encoded model which outperforms the state-of-the-art approaches for predicting saliency of mammograms.

Scene Complexity Captures the Detail Trace of Visual Episodic Memory

Memory research has established that human observers use schemas, mental constructs that represent scene elements and relationships between those elements, as the basis for encoding information into visual episodic memory. Newly developed computational methods now allow the extraction of this schema information from the human observer via two-dimensional maps known as Visual Memory Schema (VMS) maps (Kyle-Davidson, Bors & Evans, 2019). VMS maps reveal which regions in a scene image are responsible for a human recalling that scene. VMS Maps are highly consistent among observers and are not explained solely by computational saliency models or eye-fixations. However, previous work also suggests that humans, in addition to schemas, retain rich detail of the scene, and the presence of that detailed information aids in later recognition of the same. Findings show that scenes considered to be more detailed (scenes with man-made elements) are remembered better than less detailed scenes (natural scene). Similarly, images that have detail removed (compared to a source image) appear less memorable than images that maintain the original detail (Baddeley & Evans, 2019). We investigated whether perceptual image complexity could stand as an analogue for the detail trace in visual memory, exploring how the complexity of an image relates to that image's memorability. Our approach was to conduct a behavioural experiment gathering two-dimensional complexity information from humans for a scene dataset that already has corresponding schema information (VMS Maps), which allowed us to explore complexity and memorability from a two-dimensional perspective. Next, we trained a neural network to predict complexity scores, and investigate this relationship for another scene memory dataset, now comparing the predicted complexity scores with ground truth memory for the images. The findings indicate that both extracted, and predicted perceptual complexity of the scene, relates to how well observers remember that same scene.

Which emphasis technique to use? Perception of emphasis techniques with varying distractors, backgrounds, and visualization types.

Emphasis effects are visual changes that make data elements distinct from their surroundings. Designers may use computational saliency models to predict how a viewer’s attention will be guided by a specific effect; however, although saliency models provide a foundational understanding of emphasis perception, they only cover specific visual effects in abstract conditions. To address these limitations, we carried out crowdsourced studies that evaluate emphasis perception in a wider range of conditions than previously studied. We varied effect magnitude, distractor number and type, background, and visualization type, and measured the perceived emphasis of 12 visual effects. Our results show that there are perceptual commonalities of emphasis across a wide range of environments, but also that there are limitations on perceptibility for some effects, dependent on a visualization’s background or type. We developed a model of emphasis predictability based on simple scatterplots that can be extended to other viewing conditions. Our studies provide designers with new understanding of how viewers experience emphasis in realistic visualization settings.

What do radiologists look for? Advances and limitations of perceptual learning in radiologic search.

Supported by guidance from training during residency programs, radiologists learn clinically relevant visual features by viewing thousands of medical images. Yet the precise visual features that expert radiologists use in their clinical practice remain unknown. Identifying such features would allow the development of perceptual learning training methods targeted to the optimization of radiology training and the reduction of medical error. Here we review attempts to bridge current gaps in understanding with a focus on computational saliency models that characterize and predict gaze behavior in radiologists. There have been great strides toward the accurate prediction of relevant medical information within images, thereby facilitating the development of novel computer-aided detection and diagnostic tools. In some cases, computational models have achieved equivalent sensitivity to that of radiologists, suggesting that we may be close to identifying the underlying visual representations that radiologists use. However, because the relevant bottom-up features vary across task context and imaging modalities, it will also be necessary to identify relevant top-down factors before perceptual expertise in radiology can be fully understood. Progress along these dimensions will improve the tools available for educating new generations of radiologists, and aid in the detection of medically relevant information, ultimately improving patient health.

Mobile Interface Attentional Priority Model

Mobile devices are now the most popular computing technology for accessing the Internet. This has resulted in designers devoting more of their time to improving users’ mobile experiences. This can be achieved in part by helping users find content quickly and easily. Understanding what supports or harms a user’s visual search is key to creating displays that meet usability efficiency requirements. Broadly, searches are guided by a combination of visual salience and previous experiences. We recommend revealing these influences by employing a computational saliency model and by employing our mobile spatial convention map. The research presented extends the previous attentional priority (AP) modeling work, from web pages to mobile interfaces. Notably, we reveal that users typically search a mobile web page by using a “railroad-like” viewing pattern rather than the “F” pattern that is typically described in web page research. Also, we propose a mobile web page-specific attentional priority (AP) model. The AP model combines our experience-based spatial convention map with a saliency model map. We examined the predictive performance of a saliency model, compared to the mobile-specific convention map. It was discovered that the convention map better predicted the initial deployment of attention, and the saliency map better accounted for later selection.

VMD-DMD coupled data-driven approach for visual saliency in noisy images

Human visual system is endowed with an innate capability of distinguishing the salient regions of an image. It do so even in the presence of noise and other natural disturbances. Conventional8 computational saliency models in the literature assume that the input images are clean, though an explicit treatment of noise is missing. In this paper, we propose a coupled data-driven approach for estimating saliency map for a noisy input using Variational Mode Decomposition (VMD) and Dynamic Mode Decomposition(DMD. Variational Mode Decomposition (VMD) is a well received technique explored for denoising in the literature. VMD modes with high entropy (randomness) are removed and the residual modes are employed to generate a scalar valued saliency map. The proposed method is compared against seven state-of-the-art methods over a wide range of noise strengths. The submitted approach furnished comparable results with respect to state-of-the art methods for clean and noisy images in terms of various benchmark performance measures.

High Efficiency Video Coding Compliant Perceptual Video Coding Using Entropy Based Visual Saliency Model

In past years, several visual saliency algorithms have been proposed to extract salient regions from multimedia content in view of practical applications. Entropy is one of the important measures to extract salient regions, as these regions have high randomness and attract more visual attention. In the context of perceptual video coding (PVC), computational visual saliency models that utilize the charactertistics of the human visual system to improve the compression ratio are of paramount importance. To date, only a few PVC schemes have been reported that use the visual saliency model. In this paper, we conduct the first attempt to utilize entropy based visual saliency models within the high efficiency video coding (HEVC) framework. The visual saliency map generated for each input video frame is optimally thresholded to generate a binary saliency mask. The proposed HEVC compliant PVC scheme adjusts the quantization parameter according to visual saliency relevance at the coding tree unit (CTU) level. Efficient CTU level rate control is achieved by allocating bits to salient and non-salient CTUs by adjusting the quantization parameter values according to their perceptual weighted map. The attention based on information maximization has shown the best performance on newly created ground truth dataset, which is then incorporated in a HEVC framework. An average bitrate reduction of is achieved by the proposed HEVC compliant PVC scheme with the same perceptual quality and a nominal increase in coding complexity of when compared with HEVC reference software. Moreover, the proposed PVC scheme performs better than other HEVC based PVC schemes when encoded at low data rates.

Saliency-driven system models for cell analysis with deep learning

Background and objectives: Saliency refers to the visual perception quality that makes objects in a scene to stand out from others and attract attention. While computational saliency models can simulate the expert’s visual attention, there is little evidence about how these models perform when used to predict the cytopathologist’s eye fixations. Saliency models may be the key to instrumenting fast object detection on large Pap smear slides under real noisy conditions, artifacts, and cell occlusions. This paper describes how our computational schemes retrieve regions of interest (ROI) of clinical relevance using visual attention models. We also compare the performance of different computed saliency models as part of cell screening tasks, aiming to design a computer-aided diagnosis systems that supports cytopathologists.Method: We record eye fixation maps from cytopathologists at work, and compare with 13 different saliency prediction algorithms, including deep learning. We develop cell-specific convolutional neural networks (CNN) to investigate the impact of bottom-up and top-down factors on saliency prediction from real routine exams. By combining the eye tracking data from pathologists with computed saliency models, we assess algorithms reliability in identifying clinically relevant cells.Results:The proposed cell-specific CNN model outperforms all other saliency prediction methods, particularly regarding the number of false positives. Our algorithm also detects the most clinically relevant cells, which are among the three top salient regions, with accuracy above 98% for all diseases, except carcinoma (87%). Bottom-up methods performed satisfactorily, with saliency maps that enabled ROI detection above 75% for carcinoma and 86% for other pathologies.Conclusions:ROIs extraction using our saliency prediction methods enabled ranking the most relevant clinical areas within the image, a viable data reduction strategy to guide automatic analyses of Pap smear slides. Top-down factors for saliency prediction on cell images increases the accuracy of the estimated maps while bottom-up algorithms proved to be useful for predicting the cytopathologist’s eye fixations depending on parameters, such as the number of false positive and negative. Our contributions are: comparison among 13 state-of-the-art saliency models to cytopathologists’ visual attention and deliver a method that the associate the most conspicuous regions to clinically relevant cells.

Evaluation and Analysis of White Space in Wu Guanzhong’s Chinese Paintings

This article reports on a recent study that examines the effect of white space on perception of Chinese paintings. The authors investigate whether white space in Chinese paintings is not simply a blank background space but rather meaningful for aesthetic perception. Applying a computational saliency model to analyze the influence of white space on viewers’ visual information processing, the authors conducted an eye-tracking experiment. As a case study, they analyzed paintings by a well-known artist, Wu Guanzhong, and collected users’ subjective aesthetic ratings. Their results show that white space is not just a silent background: It is intentionally designed to convey certain information and has a significant effect on viewers’ aesthetic experience.

Improved recognition of Echinococcus granulosus protoscoleces using visual saliency and scale-invariant features

Echinococcosis—a parasitic disease caused by Echinococcus granulosus or Echinococcus multilocularis larvae—occurs in many regions in the world. This disease can pose a serious threat to public health and thus requires a convenient and cost-effective method for early detection. So, we developed a novel method based on visual saliency and scale-invariant features that detects the tapeworm parasites. This method improves upon existing bottom-up computational saliency models by introducing a visual attention mechanism. The results indicated that the proposed method offers a higher level of both accuracy and computational efficiency when detecting Echinococcus granulosus protoscoleces, which in turn could improve early detection of echinococcosis.

A Deep Spatial Contextual Long-Term Recurrent Convolutional Network for Saliency Detection.

Traditional saliency models usually adopt hand-crafted image features and human-designed mechanisms to calculate local or global contrast. In this paper, we propose a novel computational saliency model, i.e., deep spatial contextual long-term recurrent convolutional network (DSCLRCN), to predict where people look in natural scenes. DSCLRCN first automatically learns saliency related local features on each image location in parallel. Then, in contrast with most other deep network based saliency models which infer saliency in local contexts, DSCLRCN can mimic the cortical lateral inhibition mechanisms in human visual system to incorporate global contexts to assess the saliency of each image location by leveraging the deep spatial long short-term memory (DSLSTM) model. Moreover, we also integrate scene context modulation in DSLSTM for saliency inference, leading to a novel deep spatial contextual LSTM (DSCLSTM) model. The whole network can be trained end-to-end and works efficiently when testing. Experimental results on two benchmark datasets show that DSCLRCN can achieve state-of-the-art performance on saliency detection. Furthermore, the proposed DSCLSTM model can significantly boost the saliency detection performance by incorporating both global spatial interconnections and scene context modulation, which may uncover novel inspirations for studies on them in computational saliency models.

Spatio-Temporal Saliency Networks for Dynamic Saliency Prediction

Computational saliency models for still images have gained significant popularity in recent years. Saliency prediction from videos, on the other hand, has received relatively little interest from the community. Motivated by this, in this paper, we study the use of deep learning for dynamic saliency prediction and propose the so-called spatio-temporal saliency networks. The key to our models is the architecture of two-stream networks where we investigate different fusion mechanisms to integrate spatial and temporal information. We evaluate our models on the dynamic images and eye movements and University of Central Florida-Sports datasets and present highly competitive results against the existing state-of-the-art models. We also carry out some experiments on a number of still images from the MIT300 dataset by exploiting the optical flow maps predicted from these images. Our results show that considering inherent motion information in this way can be helpful for static saliency estimation.

Saliency Detection for Stereoscopic 3D Images in the Quaternion Frequency Domain

Recent studies have shown that a remarkable distinction exists between human binocular and monocular viewing behaviors. Compared with two-dimensional (2D) saliency detection models, stereoscopic three-dimensional (S3D) image saliency detection is a more challenging task. In this paper, we propose a saliency detection model for S3D images. The final saliency map of this model is constructed from the local quaternion Fourier transform (QFT) sparse feature and global QFT log-Gabor feature. More specifically, the local QFT feature measures the saliency map of an S3D image by analyzing the location of a similar patch. The similar patch is chosen using a sparse representation method. The global saliency map is generated by applying the wake edge-enhanced gradient QFT map through a band-pass filter. The results of experiments on two public datasets show that the proposed model outperforms existing computational saliency models for estimating S3D image saliency.

Utility function generated saccade strategies for robot active vision: a probabilistic approach

Humans and many animals can selectively sample necessary part of the visual scene to carry out daily activities like foraging and finding prey or mates. Selective attention allows them to efficiently use the limited resources of the brain by deploying sensory apparatus to collect data believed to be pertinent to the organisms current situation. Robots operating in dynamic environments are similarly exposed to a wide variety of stimuli, which they must process with limited sensory and computational resources. Computational saliency models inspired by biological studies have previously been used in robotic applications, but these had limited capacity to deal with dynamic environments and have no capacity to reason about uncertainty when planning their sensor placement strategy. This paper generalises the traditional model of saliency by using a Kalman filter estimator to describe an agent’s understanding of the world. The resulting modelling of uncertainty allows the agents to adopt a richer set of strategies to deploy sensory apparatus than is possible with the winner-take-all mechanism of the traditional saliency model. This paper demonstrates the use of three utility functions that are used to encapsulate the perceptual state that is valued by the agent. Each utility function thereby produces a distinct sensory deployment behaviour.

GBVS360, BMS360, ProSal: Extending existing saliency prediction models from 2D to omnidirectional images

In this paper it is studied how existing visual saliency models designed for 2D images can be extended and applied to Omnidirectional images in the equirectangular format. Three different frameworks, BMS360, GBVS360 and Projected Saliency were designed to address this task. In the particular case of BMS360 and GBVS360, the 2D models Boolean Map Saliency (BMS) and Graph-Based Visual Saliency (GBVS) were rigorously extended to integrate the specific properties of equirectangular images to model visual attention in omnidirectional images watched using Head Mounted Displays (HMDs). With the proposed extensions, the saliency prediction performance in comparison to their original design could significantly be improved. Another key result of this paper comes from the design of a framework called projected saliency which allows applying existing saliency models on equirectangular images without requiring in-depth adaptation of the prediction algorithms. The projected saliency framework enabled to study the interaction of features in the feature activation process of computational saliency models. It shows that the activation should not be done per-viewport, but should account for neighboring regions. As a consequence, we argue that the feature activation process should be performed at a global level instead of a local one. Here, the use of a local approach with a large FOV was found to be a good compromise between global activation and appropriate feature computation to directly port existing saliency models designed for rectilinear images to the domain of equirectangular images. In this paper an adaptive equatorial prior is also described. This equatorial prior allows to account for pitch shift of the horizon line. Finally, results show that the newly designed BMS360 and GVBS360 outperform even recent state-of-the-art models at head motion prediction and are competitive at head/eye saliency-map prediction. Moreover, all models and their code are made publicly available on GitHub allowing to further iterate on this work.

Superior colliculus neurons encode a visual saliency map during free viewing of natural dynamic video

Models of visual attention postulate the existence of a saliency map whose function is to guide attention and gaze to the most conspicuous regions in a visual scene. Although cortical representations of saliency have been reported, there is mounting evidence for a subcortical saliency mechanism, which pre-dates the evolution of neocortex. Here, we conduct a strong test of the saliency hypothesis by comparing the output of a well-established computational saliency model with the activation of neurons in the primate superior colliculus (SC), a midbrain structure associated with attention and gaze, while monkeys watched video of natural scenes. We find that the activity of SC superficial visual-layer neurons (SCs), specifically, is well-predicted by the model. This saliency representation is unlikely to be inherited from fronto-parietal cortices, which do not project to SCs, but may be computed in SCs and relayed to other areas via tectothalamic pathways.

Study of Saliency in Objective Video Quality Assessment.

Reliably predicting video quality as perceived by humans remains challenging and is of high practical relevance. A significant research trend is to investigate visual saliency and its implications for video quality assessment. Fundamental problems regarding how to acquire reliable eye-tracking data for the purpose of video quality research and how saliency should be incorporated in objective video quality metrics (VQMs) are largely unsolved. In this paper, we propose a refined methodology for reliably collecting eye-tracking data, which essentially eliminates bias induced by each subject having to view multiple variations of the same scene in a conventional experiment. We performed a large-scale eye-tracking experiment that involved 160 human observers and 160 video stimuli distorted with different distortion types at various degradation levels. The measured saliency was integrated into several best known VQMs in the literature. With the assurance of the reliability of the saliency data, we thoroughly assessed the capabilities of saliency in improving the performance of VQMs, and devised a novel approach for optimal use of saliency in VQMs. We also evaluated to what extent the state-of-the-art computational saliency models can improve VQMs in comparison to the improvement achieved by using "ground truth" eye-tracking data. The eye-tracking database is made publicly available to the research community.

Neural activities in V1 create the bottom-up saliency map of natural scenes.

A saliency map is the bottom-up contribution to the deployment of exogenous attention. It, as well as its underlying neural mechanism, is hard to identify because of the influence of top-down signals. A recent study showed that neural activities in V1 could create a bottom-up saliency map (Zhang et al. in Neuron 73(1):183-192, 2012). In this paper, we tested whether their conclusion can generalize to complex natural scenes. In order to avoid top-down influences, each image was presented with a low contrast for only 50ms and was followed by a high contrast mask, which rendered the whole image invisible to participants (confirmed by a forced-choice test). The Posner cueing paradigm was adopted to measure the spatial cueing effect (i.e., saliency) by an orientation discrimination task. A positive cueing effect was found, and the magnitude of the cueing effect was consistent with the saliency prediction of a computational saliency model. In a following fMRI experiment, we used the same masked natural scenes as stimuli and measured BOLD signals responding to the predicted salient region (relative to the background). We found that the BOLD signal in V1, but not in other cortical areas, could well predict the cueing effect. These results suggest that the bottom-up saliency map of natural scenes could be created in V1, providing further evidence for the V1 saliency theory (Li in Trends Cogn Sci 6(1):9-16, 2002).

Spatial-Aware Object-Level Saliency Prediction by Learning Graphlet Hierarchies

To fill the semantic gap between the predictive power of computational saliency models and human behavior, this paper proposes to predict where people look at using spatial-aware object-level cues. While object-level saliency has been recently suggested by psychophysics experiments and shown effective with a few computational models, the spatial relationship between the objects has not yet been explored in this context. We in this work for the first time explicitly model such spatial relationship, as well as leveraging semantic information of an image to enhance object-level saliency modeling. The core computational module is a graphlet-based (i.e., graphlets are moderate-sized connected subgraphs) deep architecture, which hierarchically learns a saliency map from raw image pixels to object-level graphlets (oGLs) and further to spatial-level graphlets (sGLs). Eye tracking data are also used to leverage human experience in saliency prediction. Experimental results demonstrate that the proposed oGLs and sGLs well capture object-level and spatial-level cues relating to saliency, and the resulting saliency model performs competitively compared with the state-of-the-art.

Predicting human gaze beyond pixels

A large body of previous models to predict where people look in natural scenes focused on pixel-level image attributes. To bridge the semantic gap between the predictive power of computational saliency models and human behavior, we propose a new saliency architecture that incorporates information at three layers: pixel-level image attributes, object-level attributes, and semantic-level attributes. Object- and semantic-level information is frequently ignored, or only a few sample object categories are discussed where scaling to a large number of object categories is not feasible nor neurally plausible. To address this problem, this work constructs a principled vocabulary of basic attributes to describe object- and semantic-level information thus not restricting to a limited number of object categories. We build a new dataset of 700 images with eye-tracking data of 15 viewers and annotation data of 5,551 segmented objects with fine contours and 12 semantic attributes (publicly available with the paper). Experimental results demonstrate the importance of the object- and semantic-level information in the prediction of visual attention.