Saliency Computation Research Articles

Scale-Adaptive Salience Supervision and Dynamic Token Filtering for Small Object Detection in Remote Sensing Images

Abstract Recently, DETR-like detectors, which have shown remarkable performance in general object detection, face limitations when dealing with remote sensing images primarily containing small objects. Mainstream two-stage DETR-like models employ a pipeline that selects and processes a small portion of informative tokens, which enhances performance but also shows a high dependency on token selection. The current static token selection strategies lead to inconsistencies between the static selection criteria and dynamic token updates. Additionally, in remote sensing images, the limited information available for small objects and their inherent sensitivity to pixel shifts further degrade detection performance. To address this, we propose Scale-Adaptive Salience DETR (SAS DETR), a two-stage DETR-like method. SAS DETR incorporates dynamic token filtering, which uses a global threshold predictor to determine the token filtering ratio for each layer of the encoder. This approach selects an appropriate filtering ratio for different network layers while maintaining consistency between the foreground confidence map and token updates. Furthermore, we introduce a novel scale-adaptive salience supervision mechanism that adaptively scales the salience computation area based on object size, ensuring the model more effectively supervises small objects and utilizes the information within tokens without compromising the detection performance for objects of other sizes. Finally, we employ Scale-Adaptive Intersection over Union to reduce the impact of pixel shifts on small objects. With these improvements, our proposed SAS DETR achieves 25.2% AP on the AI-TOD-V2 dataset with 24 training epochs and 50.4% AP on the COCO 2017 dataset with 12 training epochs.

Airborne LiDAR Point Cloud Filtering Algorithm Based on Supervoxel Ground Saliency

Abstract. Airborne laser scanning (ALS) is able to penetrate sparse vegetation to obtain highly accurate height information on the ground surface. LiDAR point cloud filtering is an important prerequisite for downstream tasks such as digital terrain model (DTM) extraction and point cloud classification. Aiming at the problem that existing LiDAR point cloud filtering algorithms are prone to errors in complex terrain environments, an ALS point cloud filtering method based on supervoxel ground saliency (SGSF) is proposed in this paper. Firstly, a boundary-preserving TBBP supervoxel algorithm is utilized to perform supervoxel segmentation of ALS point clouds, and multi-directional scanning strip delineation and ground saliency computation are carried out for the clusters of supervoxel point clouds. Subsequently, the energy function is constructed by introducing the ground saliency and the optimal filtering plane of the supervoxel is solved using the semi-global optimization idea to realize the effective distinction between ground and non-ground points. Experimental results on the ALS point cloud filtering dataset openGF indicate that, compared to state-of-the-art surface-based filtering methods, the SGSF algorithm achieves the highest average values across various terrain conditions for multiple evaluation metrics. It also addresses the issue of recessed structures in buildings being prone to misclassification as ground points.

Predicting Radiologists' Gaze With Computational Saliency Models in Mammogram Reading

Previous studies have shown that there is a strong correlation between radiologists' diagnoses and their gaze when reading medical images. The extent to which gaze is attracted by content in a visual scene can be characterised as visual saliency. There is a potential for the use of visual saliency in computer-aided diagnosis in radiology. However, little is known about what methods are effective for diagnostic images, and how these methods could be adapted to address specific applications in diagnostic imaging. In this study, we investigate 20 state-of-the-art saliency models including 10 traditional models and 10 deep learning-based models in predicting radiologists' visual attention while reading 196 mammograms. We found that deep learning-based models represent the most effective type of methods for predicting radiologists' gaze in mammogram reading; and that the performance of these saliency models can be significantly improved by transfer learning. In particular, an enhanced model can be achieved by pre-training the model on a large-scale natural image saliency dataset and then fine-tuning it on the target medical image dataset. In addition, based on a systematic selection of backbone networks and network architectures, we proposed a parallel multi-stream encoded model which outperforms the state-of-the-art approaches for predicting saliency of mammograms.

A Critique of the Attentional Window Account of Capture Failures.

There has been a lengthy debate about whether salient stimuli have the power to automatically capture attention, even when entirely task irrelevant. Theeuwes (2022) has suggested that an attentional window account could explain why capture is observed in some studies, but not others. According to this account, when search is difficult, participants narrow their attentional window, and this prevents the salient distractor from generating a saliency signal. In turn, this causes the salient distractor to fail to capture attention. In the present commentary, we describe two major problems with this account. First, the attentional window account proposes that attention must be focused so narrowly that featural information from the salient distractor will be filtered prior to saliency computations. However, many previous studies observing no capture provided evidence that featural processing was sufficiently detailed to guide attention toward the target shape. This indicates that the attentional window was sufficiently broad to allow featural processing. Second, the attentional window account proposes that capture should occur more readily in easy search tasks than difficult search tasks. We review previous studies that violate this basic prediction of the attentional window account. A more parsimonious account of the data is that control over feature processing can be exerted proactively to prevent capture, at least under certain conditions.

Deep Learning Assists Surveillance Experts: Toward Video Data Prioritization

Video summarization (VS) suppresses high-dimensional (HD) video data by only extracting only the important information. However, prior research has not focused on the need for surveillance VS, that is used for many applications to assist video surveillance experts, including video retrieval and data storage. In addition, mainstream techniques commonly use 2D deep models for VS, ignoring event occurrences. Accordingly, we present a two-fold 3D deep learning-assisted VS framework. First, we employ an inflated 3D ConvNet model to extract temporal features; these features are optimized using a proposed encoder mechanism. The input video is temporally segmented using a feature comparison technique for selecting a single frame from each video segment. The segmented shots are evaluated using our novel shot segmentation evaluation scheme and are input into a saliency computation mechanism for keyframe selection in a second fold. Qualitative and quantitative analyses over VS benchmarks and surveillance videos demonstrate the superior performance of our framework, with 0.3- and 4.2-unit increases in the F1 scores for YouTube and TVSum datasets, respectively. Along with accurate VS, a key contribution of our study is the novel shot segmentation criterion prior to VS, which can be used as a benchmark in future research to effectively prioritize HD visual data.

Structural Landmark Salience Computation in Compact Urban Districts with 3D Node-Landmark Grid Analysis Model: A Case Study on Two Sample Districts in Changsha, China

Mastering the relationship between urban landmarks and urban space morphology in urban planning, landscape planning, and architectural design helps maintain the intelligibility of compact urban districts. The objective of the present study was to numerically determine the structural salience of various landmarks in an urban environment and use it to interpret the intelligibility of the city. Combining the measurement method of 3D visibility and the related principles of space syntax, this study develops a new 3D Node–Landmark Grid Analysis Model (3D NL GAM) for structural salience computation of urban landmarks. In this study, a numerical approach is used to construct a 3D simulation model. Firstly, the visibility of each decision node to landmarks in an urban environment, using a 3D digital model, is measured using the 3D isovist component of Rhinoceros and Grasshopper software. Secondly, links among wayfinding decision nodes and landmarks are established to form a 3D NL GAM. The normalized angular integration of decision nodes and the normalized angular choice of landmarks are computed using the principle of space syntax. Thirdly, the structural salience of landmarks is determined with a function of landmark visibility, spatial properties of landmarks, and wayfinding decision nodes. Finally, a case study was carried out by using a 3D NL GAM to analyze three types of urban areas located in Changsha. The results indicated that large-scale natural landscapes have a higher structural salience among the types of landmarks. The structural salience of architectural landmarks in the combined spatial form of combining tall and low building groups has a clear advantage over the form dominated by high-rise building groups. Raising the height of landmark buildings can modify the structure of the grid analysis model and improve the people aggregation of urban space. The 3D NL GAM can quantify the spatial properties and landmark structural salience of a city and can effectively assist in the evaluation of the intelligibility of built or future urban environments.

Scene Complexity Captures the Detail Trace of Visual Episodic Memory

Memory research has established that human observers use schemas, mental constructs that represent scene elements and relationships between those elements, as the basis for encoding information into visual episodic memory. Newly developed computational methods now allow the extraction of this schema information from the human observer via two-dimensional maps known as Visual Memory Schema (VMS) maps (Kyle-Davidson, Bors & Evans, 2019). VMS maps reveal which regions in a scene image are responsible for a human recalling that scene. VMS Maps are highly consistent among observers and are not explained solely by computational saliency models or eye-fixations. However, previous work also suggests that humans, in addition to schemas, retain rich detail of the scene, and the presence of that detailed information aids in later recognition of the same. Findings show that scenes considered to be more detailed (scenes with man-made elements) are remembered better than less detailed scenes (natural scene). Similarly, images that have detail removed (compared to a source image) appear less memorable than images that maintain the original detail (Baddeley & Evans, 2019). We investigated whether perceptual image complexity could stand as an analogue for the detail trace in visual memory, exploring how the complexity of an image relates to that image's memorability. Our approach was to conduct a behavioural experiment gathering two-dimensional complexity information from humans for a scene dataset that already has corresponding schema information (VMS Maps), which allowed us to explore complexity and memorability from a two-dimensional perspective. Next, we trained a neural network to predict complexity scores, and investigate this relationship for another scene memory dataset, now comparing the predicted complexity scores with ground truth memory for the images. The findings indicate that both extracted, and predicted perceptual complexity of the scene, relates to how well observers remember that same scene.

Where to look at the movies: Analyzing visual attention to understand movie editing.

In the process of making a movie, directors constantly care about where the spectator will look on the screen. Shot composition, framing, camera movements, or editing are tools commonly used to direct attention. In order to provide a quantitative analysis of the relationship between those tools and gaze patterns, we propose a new eye-tracking database, containing gaze-pattern information on movie sequences, as well as editing annotations, and we show how state-of-the-art computational saliency techniques behave on this dataset. In this work, we expose strong links between movie editing and spectators gaze distributions, and open several leads on how the knowledge of editing information could improve human visual attention modeling for cinematic content. The dataset generated and analyzed for this study is available at https://github.com/abruckert/eye_tracking_filmmaking.

Directional Preference in Avian Midbrain Saliency Computing Nucleus Reflects a Well-Designed Receptive Field Structure.

Simple SummaryDirectional preference neurons has been found in many vertebrate sensory systems. The isthmi pars magnocellularis (Imc) in avian midbrain, playing a key role in visual selective attention, shows impressive motion directional preference, but little is known about the physiological basis of this phenomenon. Herein, artificial visual stimuli, statistical analyses, and a neural computational model were used to unravel this mystery. This study deepens the understanding of the relationship between the directional preference and special receptive field structure of pigeon’s (Columba livia) Imc neuron.Neurons responding sensitively to motions in several rather than all directions have been identified in many sensory systems. Although this directional preference has been demonstrated by previous studies to exist in the isthmi pars magnocellularis (Imc) of pigeon (Columba livia), which plays a key role in the midbrain saliency computing network, the dynamic response characteristics and the physiological basis underlying this phenomenon are unclear. Herein, dots moving in 16 directions and a biologically plausible computational model were used. We found that pigeon Imc’s significant responses for objects moving in preferred directions benefit the long response duration and high instantaneous firing rate. Furthermore, the receptive field structures predicted by a computational model, which captures the actual directional tuning curves, agree with the real data collected from population Imc units. These results suggested that directional preference in Imc may be internally prebuilt by elongating the vertical axis of the receptive field, making predators attack from the dorsal-ventral direction and conspecifics flying away in the ventral-dorsal direction, more salient for avians, which is of great ecological and physiological significance for survival.

Human lesions and animal studies link the claustrum to perception, salience, sleep and pain.

The claustrum is the most densely interconnected region in the human brain. Despite the accumulating data from clinical and experimental studies, the functional role of the claustrum remains unknown. Here, we systematically review claustrum lesion studies and discuss their functional implications. Claustral lesions are associated with an array of signs and symptoms, including changes in cognitive, perceptual and motor abilities; electrical activity; mental state; and sleep. The wide range of symptoms observed following claustral lesions do not provide compelling evidence to support prominent current theories of claustrum function such as multisensory integration or salience computation. Conversely, the lesions studies support the hypothesis that the claustrum regulates cortical excitability. We argue that the claustrum is connected to, or part of, multiple brain networks that perform both fundamental and higher cognitive functions. As a multifunctional node in numerous networks, this may explain the manifold effects of claustrum damage on brain and behaviour.

A compact deep architecture for real-time saliency prediction

Saliency computation models aim to imitate the attention mechanism in the human visual system. The application of deep neural networks for saliency prediction has led to a drastic improvement over the last few years. However, deep models have a high number of parameters which makes them less suitable for real-time applications. Here we propose a compact yet fast model for real-time saliency prediction. Our proposed model consists of a modified U-net architecture, a location-dependent fully-connected layer, and central difference convolutional layers. The modification we made on U-Net architecture, promoted compactness and efficiency of the final model. Our location-dependent fully-connected layer facilitates implicit capturing of the location-dependent information. Using the central difference convolutional layers at different scales enables capturing more robust and biologically motivated features. We compare our model with state-of-the-art saliency models using traditional saliency scores as well as our newly devised scheme. Experimental results over four challenging saliency benchmark datasets demonstrate the effectiveness of our approach in striking a balance between accuracy and speed. Our model can be run in real-time which makes it appealing for edge devices and video processing.

Parallel Advantage: Further Evidence for Bottom-up Saliency Computation by Human Primary Visual Cortex.

Finding a target among uniformly oriented non-targets is typically faster when this target is perpendicular, rather than parallel, to the non-targets. The V1 Saliency Hypothesis (V1SH), that neurons in the primary visual cortex (V1) signal saliency for exogenous attentional attraction, predicts exactly the opposite in a special case: each target or non-target comprises two equally sized disks displaced from each other by 1.2 disk diameters center-to-center along a line defining its orientation. A target has two white or two black disks. Each non-target has one white disk and one black disk, and thus, unlike the target, activates V1 neurons less when its orientation is parallel rather than perpendicular to the neurons’ preferred orientations. When the target is parallel, rather than perpendicular, to the uniformly oriented non-targets, the target’s evoked V1 response escapes V1’s iso-orientation surround suppression, making the target more salient. I present behavioral observations confirming this prediction.

Which emphasis technique to use? Perception of emphasis techniques with varying distractors, backgrounds, and visualization types.

Emphasis effects are visual changes that make data elements distinct from their surroundings. Designers may use computational saliency models to predict how a viewer’s attention will be guided by a specific effect; however, although saliency models provide a foundational understanding of emphasis perception, they only cover specific visual effects in abstract conditions. To address these limitations, we carried out crowdsourced studies that evaluate emphasis perception in a wider range of conditions than previously studied. We varied effect magnitude, distractor number and type, background, and visualization type, and measured the perceived emphasis of 12 visual effects. Our results show that there are perceptual commonalities of emphasis across a wide range of environments, but also that there are limitations on perceptibility for some effects, dependent on a visualization’s background or type. We developed a model of emphasis predictability based on simple scatterplots that can be extended to other viewing conditions. Our studies provide designers with new understanding of how viewers experience emphasis in realistic visualization settings.

Detecting Salient Image Objects Using Color Histogram Clustering for Region Granularity.

Salient object detection represents a novel preprocessing stage of many practical image applications in the discipline of computer vision. Saliency detection is generally a complex process to copycat the human vision system in the processing of color images. It is a convoluted process because of the existence of countless properties inherent in color images that can hamper performance. Due to diversified color image properties, a method that is appropriate for one category of images may not necessarily be suitable for others. The selection of image abstraction is a decisive preprocessing step in saliency computation and region-based image abstraction has become popular because of its computational efficiency and robustness. However, the performances of the existing region-based salient object detection methods are extremely hooked on the selection of an optimal region granularity. The incorrect selection of region granularity is potentially prone to under- or over-segmentation of color images, which can lead to a non-uniform highlighting of salient objects. In this study, the method of color histogram clustering was utilized to automatically determine suitable homogenous regions in an image. Region saliency score was computed as a function of color contrast, contrast ratio, spatial feature, and center prior. Morphological operations were ultimately performed to eliminate the undesirable artifacts that may be present at the saliency detection stage. Thus, we have introduced a novel, simple, robust, and computationally efficient color histogram clustering method that agglutinates color contrast, contrast ratio, spatial feature, and center prior for detecting salient objects in color images. Experimental validation with different categories of images selected from eight benchmarked corpora has indicated that the proposed method outperforms 30 bottom-up non-deep learning and seven top-down deep learning salient object detection methods based on the standard performance metrics.

Saliency Computation for Virtual Cinematography in 360° Videos.

Recent advances in virtual reality cameras have contributed to a phenomenal growth of 360$^{\circ }$∘ videos. Estimating regions likely to attract user attention is critical for efficiently streaming and rendering 360$^{\circ }$∘ videos. In this article, we present a simple, novel, GPU-driven pipeline for saliency computation and virtual cinematography in 360$^{\circ }$∘ videos using spherical harmonics (SH). We efficiently compute the 360$^{\circ }$∘ video saliency through the spectral residual of the SH coefficients between multiple bands at over 60FPS for 4K resolution videos. Further, our interactive computation of spherical saliency can be used for saliency-guided virtual cinematography in 360$^{\circ }$∘ videos.

Voluntary choice tasks increase control settings and reduce capture

ABSTRACT In their article, Luck et al. ([2021]. Progress toward resolving the attentional capture debate. Visual Cognition, 29(1), 1–21. https://doi.org/10.1080/13506285.2020.1848949) outline alternative hypotheses regarding the cognitive mechanisms that modulate attention capture. This commentary addresses how control signals can be used to modulate feature gain control prior to saliency computations, in addition to the implicitly learned effects of alternate selection histories. We suggest that voluntary choice paradigms provide much stronger task-induced attentional control than pre-trial cueing, and allow a comparison of the same search stimuli under different control states. We argue that voluntary task choice results in an attenuation of salience signals, eliminating attention capture, although response time distractor costs remain.

Multi-scale Feature Fusion for 3D Saliency Detection

3D saliency detection aims to take advantage of the disparity map, depth map and color information to automatically detect informative objects from natural scenes. Although studies have concentrated on this issue in recent years, there are challenges such as how to leverage disparity map or depth map effectively to compute depth-induced saliency, and how to fuse optimally multiple visual features and cues. A novel 3D saliency detection approach is proposed, which fuses local contrast, region contrast, texture feature, depth cue, and location cue into a unified saliency computation framework. Results show that the proposed approach achieves significant and consistent improvements on other advanced methods in the RGBD1000 datasets.

What do radiologists look for? Advances and limitations of perceptual learning in radiologic search.

Supported by guidance from training during residency programs, radiologists learn clinically relevant visual features by viewing thousands of medical images. Yet the precise visual features that expert radiologists use in their clinical practice remain unknown. Identifying such features would allow the development of perceptual learning training methods targeted to the optimization of radiology training and the reduction of medical error. Here we review attempts to bridge current gaps in understanding with a focus on computational saliency models that characterize and predict gaze behavior in radiologists. There have been great strides toward the accurate prediction of relevant medical information within images, thereby facilitating the development of novel computer-aided detection and diagnostic tools. In some cases, computational models have achieved equivalent sensitivity to that of radiologists, suggesting that we may be close to identifying the underlying visual representations that radiologists use. However, because the relevant bottom-up features vary across task context and imaging modalities, it will also be necessary to identify relevant top-down factors before perceptual expertise in radiology can be fully understood. Progress along these dimensions will improve the tools available for educating new generations of radiologists, and aid in the detection of medically relevant information, ultimately improving patient health.

Mobile Interface Attentional Priority Model

Mobile devices are now the most popular computing technology for accessing the Internet. This has resulted in designers devoting more of their time to improving users’ mobile experiences. This can be achieved in part by helping users find content quickly and easily. Understanding what supports or harms a user’s visual search is key to creating displays that meet usability efficiency requirements. Broadly, searches are guided by a combination of visual salience and previous experiences. We recommend revealing these influences by employing a computational saliency model and by employing our mobile spatial convention map. The research presented extends the previous attentional priority (AP) modeling work, from web pages to mobile interfaces. Notably, we reveal that users typically search a mobile web page by using a “railroad-like” viewing pattern rather than the “F” pattern that is typically described in web page research. Also, we propose a mobile web page-specific attentional priority (AP) model. The AP model combines our experience-based spatial convention map with a saliency model map. We examined the predictive performance of a saliency model, compared to the mobile-specific convention map. It was discovered that the convention map better predicted the initial deployment of attention, and the saliency map better accounted for later selection.

Detecting semantic regions of construction site images by transfer learning and saliency computation

Effective use of massive construction site images and videos requires an efficient storage and retrieval method. However, significant portions of the image regions contain little useful information to project engineers and managers. To reduce resource waste in data storage and retrieval, we developed a new semantic region detection approach using transfer learning and modified saliency computation method without the need to specify targeted objects. In the new approach, the saliency matrix is generated using labelled bounding boxes, and the semantic regions are selected using a developed algorithm. The proposed method was applied to case studies based on two image datasets. The case studies suggest that the proposed method can efficiently detect semantic regions in site images and detect construction events from other image datasets without a modifying or re-training process. The research contributes to construction image analytics academically by advancing the context-based semantic region detection method and practically by facilitating the effective storage and processing of the massive site images and videos.