Saliency Prediction Research Articles

A Trained Humanoid Robot can Perform Human-Like Crossmodal Social Attention and Conflict Resolution

To enhance human-robot social interaction, it is essential for robots to process multiple social cues in a complex real-world environment. However, incongruency of input information across modalities is inevitable and could be challenging for robots to process. To tackle this challenge, our study adopted the neurorobotic paradigm of crossmodal conflict resolution to make a robot express human-like social attention. A behavioural experiment was conducted on 37 participants for the human study. We designed a round-table meeting scenario with three animated avatars to improve ecological validity. Each avatar wore a medical mask to obscure the facial cues of the nose, mouth, and jaw. The central avatar shifted its eye gaze while the peripheral avatars generated sound. Gaze direction and sound locations were either spatially congruent or incongruent. We observed that the central avatar’s dynamic gaze could trigger crossmodal social attention responses. In particular, human performance was better under the congruent audio-visual condition than the incongruent condition. Our saliency prediction model was trained to detect social cues, predict audio-visual saliency, and attend selectively for the robot study. After mounting the trained model on the iCub, the robot was exposed to laboratory conditions similar to the human experiment. While the human performance was overall superior, our trained model demonstrated that it could replicate attention responses similar to humans.

Loop closure detection with patch-level local features and visual saliency prediction

Loop closure detection (LCD) is essential in the field of visual Simultaneous Localization and Mapping (vSLAM). In the LCD system, geometrical verification based on image matching plays a crucial role in avoiding erroneous detections. This paper focuses on adopting patch-level local features for image matching to compute the similarity score between the current query image and the candidate images. However, an important factor that may reduce the robustness is that some distracting and dynamic regions in a scene (e.g., the sky, cars, pedestrians, the ground, etc.) are not helpful and may seriously harm the performance. To address this challenge, we first use a newly designed patch descriptor loss to optimize the distance relationship between the patch-level local features. In this way, the patch-level local features extracted from the query/candidate images are more suitable for performing image matching. Moreover, we mimic the visual attention mechanism and propose a patch matching with saliency strategy, which enables local patches in salient regions to play crucial roles in image matching by assigning suitable weights to them. Finally, experiments on several public datasets demonstrate that the proposed LCD system can achieve encouraging improvements over the state-of-the-art approaches regarding recall rates under 100% precision.

Global–local–global context-aware network for salient object detection in optical remote sensing images

For the salient object detection in optical remote sensing images (ORSI-SOD), many existing methods are trapped in a local–global mode, i.e., CNN-based encoder binds with a specific global context-aware module, struggling to deal with the challenging ORSIs with complex background and scale-variant objects. To solve this issue, we explore the synergy of the global-context-aware and local-context-aware modeling and construct a preferable global–local–global context-aware network (GLGCNet). In the GLGCNet, a transformer-based encoder is adopted to extract global representations, combining with local-context-aware features gathered from three saliency-up modules for comprehensive saliency modeling, and an edge assignment module is additionally employed to refine the preliminary detection. Specifically, the saliency-up module involves two components, one for global–local context-aware transfer towards pixel-wise dynamic convolution parameters prediction, the other for dynamically local-context aware modeling. The corresponding position-sensitive filter is aware of its previous global-wise focus, thus enhancing the spatial compactness of salient objects and encouraging the feature upsampling achievement for multi-scale feature combinations. The edge assignment module enhances the robustness of preliminary saliency prediction and assigns the semantic attributes of preliminary saliency cues to the shallow-level edge feature to obtain final complete salient objects in a spatially and semantically global manner. Extensive experiments demonstrate that the proposed GLGCNet surpasses 23 state-of-the-art methods on three popular datasets.

Toward Visual Behavior and Attention Understanding for Augmented 360 Degree Videos

Augmented reality (AR) overlays digital content onto reality. In an AR system, correct and precise estimations of user visual fixations and head movements can enhance the quality of experience by allocating more computational resources for analyzing, rendering, and 3D registration on the areas of interest. However, there is inadequate research to help in understanding the visual explorations of the users when using an AR system or modeling AR visual attention. To bridge the gap between the saliency prediction on real-world scenes and on scenes augmented by virtual information, we construct the ARVR saliency dataset. The virtual reality (VR) technique is employed to simulate the real-world. Annotations of object recognition and tracking as augmented contents are blended into omnidirectional videos. The saliency annotations of head and eye movements for both original and augmented videos are collected and together constitute the ARVR dataset. We also design a model that is capable of solving the saliency prediction problem in AR. Local block images are extracted to simulate the viewport and offset the projection distortion. Conspicuous visual cues in the local block images are extracted to constitute the spatial features. The optical flow information is estimated as an important temporal feature. We also consider the interplay between virtual information and reality. The composition of the augmentation information is distinguished, and the joint effects of adversarial augmentation and complementary augmentation are estimated. The Markov chain is constructed with block images as graph nodes. In the determination of the edge weights, both the characteristics of the viewing behaviors and the visual saliency mechanisms are considered. The order of importance for block images is estimated through the state of equilibrium of the Markov chain. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.

Simulating Urban Element Design with Pedestrian Attention: Visual Saliency as Aid for More Visible Wayfinding Design

Signs, landmarks, and other urban elements should attract attention to or harmonize with the environment for successful landscape design. These elements also provide information during navigation—particularly for people with cognitive difficulties or those unfamiliar with the geographical area. Nevertheless, some urban components are less eye-catching than intended because they are created and positioned irrespective of their surroundings. While quantitative measures such as eye tracking have been introduced, they help the initial or final stage of the urban design process and they involve expensive experiments. We introduce machine-learning-predicted visual saliency as iterative feedback for pedestrian attention during urban element design. Our user study focused on wayfinding signs as part of urban design and revealed that providing saliency prediction promoted a more efficient and helpful design experience without compromising usability. The saliency-guided design practice also contributed to producing more eye-catching and aesthetically pleasing urban elements. The study demonstrated that visual saliency can lead to an improved urban design experience and outcome, resulting in more accessible cities for citizens, visitors, and people with cognitive impairments.

Quality-Driven Dual-Branch Feature Integration Network for Video Salient Object Detection

Video salient object detection has attracted growing interest in recent years. However, some existing video saliency models often suffer from the inappropriate utilization of spatial and temporal cues and the insufficient aggregation of different level features, leading to remarkable performance degradation. Therefore, we propose a quality-driven dual-branch feature integration network majoring in the adaptive fusion of multi-modal cues and sufficient aggregation of multi-level spatiotemporal features. Firstly, we employ the quality-driven multi-modal feature fusion (QMFF) module to combine the spatial and temporal features. Particularly, the quality scores estimated from each level’s spatial and temporal cues are not only used to weigh the two modal features but also to adaptively integrate the coarse spatial and temporal saliency predictions into the guidance map, which further enhances the two modal features. Secondly, we deploy the dual-branch-based multi-level feature aggregation (DMFA) module to integrate multi-level spatiotemporal features, where the two branches including the progressive decoder branch and the direct concatenation branch sufficiently explore the cooperation of multi-level spatiotemporal features. In particular, in order to provide an adaptive fusion for the outputs of the two branches, we design the dual-branch fusion (DF) unit, where the channel weight of each output can be learned jointly from the two outputs. The experiments conducted on four video datasets clearly demonstrate the effectiveness and superiority of our model against the state-of-the-art video saliency models.

STI-Net: Spatiotemporal integration network for video saliency detection

Image saliency detection, to which much effort has been devoted in recent years, has advanced significantly. In contrast, the community has paid little attention to video saliency detection. Especially, existing video saliency models are very likely to fail in videos with difficult scenarios such as fast motion, dynamic background, and nonrigid deformation. Furthermore, performing video saliency detection directly using image saliency models that ignore video temporal information is inappropriate. To alleviate this issue, this study proposes a novel end-to-end spatiotemporal integration network (STI-Net) for detecting salient objects in videos. Specifically, our method is made up of three key steps: feature aggregation, saliency prediction, and saliency fusion, which are used sequentially to generate spatiotemporal deep feature maps, coarse saliency predictions, and the final saliency map. The key advantage of our model lies in the comprehensive exploration of spatial and temporal information across the entire network, where the two features interact with each other in the feature aggregation step, are used to construct boundary cue in the saliency prediction step, and also serve as the original information in the saliency fusion step. As a result, the generated spatiotemporal deep feature maps can precisely and completely characterize the salient objects, and the coarse saliency predictions have well-defined boundaries, effectively improving the final saliency map's quality. Furthermore, “shortcut connections” are introduced into our model to make the proposed network easy to train and obtain accurate results when the network is deep. Extensive experimental results on two publicly available challenging video datasets demonstrate the effectiveness of the proposed model, which achieves comparable performance to state-of-the-art saliency models.

Specificity-preserving RGB-D saliency detection

Salient object detection (SOD) in RGB and depth images has attracted increasing research interest. Existing RGB-D SOD models usually adopt fusion strategies to learn a shared representation from RGB and depth modalities, while few methods explicitly consider how to preserve modality-specific characteristics. In this study, we propose a novel framework, the specificity-preserving network (SPNet), which improves SOD performance by exploring both the shared information and modality-specific properties. Specifically, we use two modality-specific networks and a shared learning network to generate individual and shared saliency prediction maps. To effectively fuse cross-modal features in the shared learning network, we propose a cross-enhanced integration module (CIM) and propagate the fused feature to the next layer to integrate cross-level information. Moreover, to capture rich complementary multi-modal information to boost SOD performance, we use a multi-modal feature aggregation (MFA) module to integrate the modality-specific features from each individual decoder into the shared decoder. By using skip connections between encoder and decoder layers, hierarchical features can be fully combined. Extensive experiments demonstrate that our SPNet outperforms cutting-edge approaches on six popular RGB-D SOD and three camouflaged object detection benchmarks. The project is publicly available at https://github.com/taozh2017/SPNet.

Advertising Image Saliency Prediction Method Based on Score Level Fusion

At present, visual saliency prediction algorithms have been developed more and more mature, but most of the current saliency prediction algorithms are aimed at natural images. Due to the inconsistency of elements and features between natural images and advertising images, the existing saliency prediction algorithms show poor robustness and low inference speed to advertising images, which severely limits its commercial application in advertising design and evaluation. In view of this, a saliency prediction algorithm for advertisement images is proposed in this paper. In the feature extraction stage, two text candidate regions based on intensity feature and improved MESR algorithm are first obtained and further integrated to produce a two-dimensional text confidence score. Meanwhile, a saliency confidence score is also obtained by an improved natural image saliency prediction network. Then, the score level fusion strategy was adopted to fuse the two confidence scores to get the final saliency prediction map. The experimental results show that the proposed model has good accuracy and robustness in advertising images, as well as the most remarkable inference speed, which can meet the demand for real-time performance of advertising image saliency prediction, leading to great practical and commercial value.

An Energy-Based Prior for Generative Saliency.

We propose a novel generative saliency prediction framework that adopts an informative energy-based model as a prior distribution. The energy-based prior model is defined on the latent space of a saliency generator network that generates the saliency map based on a continuous latent variables and an observed image. Both the parameters of saliency generator and the energy-based prior are jointly trained via Markov chain Monte Carlo-based maximum likelihood estimation, in which the sampling from the intractable posterior and prior distributions of the latent variables are performed by Langevin dynamics. With the generative saliency model, we can obtain a pixel-wise uncertainty map from an image, indicating model confidence in the saliency prediction. Different from existing generative models, which define the prior distribution of the latent variables as a simple isotropic Gaussian distribution, our model uses an energy-based informative prior which can be more expressive in capturing the latent space of the data. With the informative energy-based prior, we extend the Gaussian distribution assumption of generative models to achieve a more representative distribution of the latent space, leading to more reliable uncertainty estimation. We apply the proposed frameworks to both RGB and RGB-D salient object detection tasks with both transformer and convolutional neural network backbones. We further propose an adversarial learning algorithm and a variational inference algorithm as alternatives to train the proposed generative framework. Experimental results show that our generative saliency model with an energy-based prior can achieve not only accurate saliency predictions but also reliable uncertainty maps that are consistent with human perception.

Saliency Prediction in Uncategorized Videos Based on Audio-Visual Correlation

Substantial research has been done in saliency modeling to make intelligent machines that can perceive and interpret their surroundings and focus only on the salient regions in a visual scene. But existing spatio-temporal saliency models either treat videos as merely image sequences excluding any audio information or are unable to cope with inherently varying content. Based on the hypothesis that an audiovisual saliency model will perform better than traditional spatio-temporal saliency models, this work aims to provide a generic preliminary audio/video saliency model. This is achieved by augmenting visual saliency map with an audio saliency map computed by synchronizing low-level audio and visual features. The proposed model was evaluated using different criteria against eye fixations data for a publicly available video dataset DIEM. The evaluation results show that the model outperforms two state-of-the-art visual spatio-temporal saliency models supporting our hypothesis that an audiovisual model performs better in comparison to a visual model for natural uncategorized videos.

S $^3$ Net: Self-Supervised Self-Ensembling Network for Semi-Supervised RGB-D Salient Object Detection

RGB-D salient object detection aims to detect visually distinctive objects or regions from a pair of the RGB image and the depth image. State-of-the-art RGB-D saliency detectors are mainly based on convolutional neural networks but almost suffer from an intrinsic limitation relying on the labeled data, thus degrading detection accuracy in complex cases. In this work, we present a self-supervised self-ensembling network (S $^3$ Net) for semi-supervised RGB-D salient object detection by leveraging the unlabeled data and exploring a self-supervised learning mechanism. To be specific, we first build a self-guided convolutional neural network (SG-CNN) as a baseline model by developing a series of three-layer cross-model feature fusion (TCF) modules to leverage complementary information among depth and RGB modalities and formulating an auxiliary task that predicts a self-supervised image rotation angle. After that, to further explore the knowledge from unlabeled data, we assign SG-CNN to a student network and a teacher network, and encourage the saliency predictions and self-supervised rotation predictions from these two networks to be consistent on the unlabeled data. Experimental results on seven widely-used benchmark datasets demonstrate that our network quantitatively and qualitatively outperforms the state-of-the-art methods.

Exploring viewport features for semi-supervised saliency prediction in omnidirectional images

Exploring viewport features for semi-supervised saliency prediction in omnidirectional images

Spatio-Temporal Self-Attention Network for Video Saliency Prediction

3D convolutional neural networks have achieved promising results for video tasks in computer vision, including video saliency prediction that is explored in this paper. However, 3D convolution encodes visual representation merely on fixed local spacetime according to its kernel size, while human attention is always attracted by relational visual features at different time. To overcome this limitation, we propose a novel Spatio-Temporal Self-Attention 3D Network (STSANet) for video saliency prediction, in which multiple Spatio-Temporal Self-Attention (STSA) modules are employed at different levels of 3D convolutional backbone to directly capture long-range relations between spatio-temporal features of different time steps. Besides, we propose an Attentional Multi-Scale Fusion (AMSF) module to integrate multi-level features with the perception of context in semantic and spatio-temporal subspaces. Extensive experiments demonstrate the contributions of key components of our method, and the results on DHF1K, Hollywood-2, UCF, and DIEM benchmark datasets clearly prove the superiority of the proposed model compared with all state-of-the-art models.

Viewing Bias Matters in 360° Videos Visual Saliency Prediction

360° video has been applied to many areas such as immersive contents, virtual tours, and surveillance systems. Compared to the field of view prediction on planar videos, the explosive amount of information contained in the omni-directional view on the entire sphere poses an additional challenge in predicting high-salient regions in 360° videos. In this work, we propose a visual saliency prediction model that directly takes 360° video in the equirectangular format. Unlike previous works that often adopted recurrent neural network (RNN) architecture for the saliency detection task, in this work, we utilize 3D convolution to a spatial-temporal encoder and generalize SphereNet kernels to construct a spatial-temporal decoder. We further study the statistical properties of viewing biases present in 360° datasets across various video types, which provides us with insights into the design of a fusing mechanism that incorporates the predicted saliency map with the viewing bias in an adaptive manner. The proposed model yields state-of-the-art performance, as evidenced by empirical results over renowned 360° visual saliency datasets such as Salient360!, PVS, and Sport360.

A geospatial image based eye movement dataset for cartography and GIS

ABSTRACT Eye movement is a new type of data for cartography and geographic information science (GIS) research. However, previous studies rarely built eye movement datasets with geospatial images. In this paper, we firstly proposed a geospatial image-based eye movement dataset called GeoEye, a publicly shared, widely available eye movement dataset. This dataset consists of 110 college-aged participants who freely viewed 500 images, including thematic maps, remote sensing images, and street view images. In addition, we used the dataset for geospatial image saliency prediction and map user identification. Results demonstrated the scientific benefits and applications of the proposed dataset. GeoEye dataset will not only promote the application of eye-tracking data in cartography and GIS research but also intelligence and customization of geographic information services.

Intra- and Inter-Reasoning Graph Convolutional Network for Saliency Prediction on 360° Images

Cubic projection can be utilized to divide 360° images into multiple rectilinear images, with little distortion. However, the existing saliency prediction models fail to integrate semantic information of these images. In this paper, we address this by proposing an intra- and inter-reasoning graph convolutional network for saliency prediction on 360° images (SalReGCN360). The whole framework contains six sub-networks, each of which contains two branches. In the training phase, after utilizing Multiple Cubic Projection (MCP), six rectilinear images are simultaneously put into corresponding sub-networks. In one of the branches, the global features of a single rectilinear image are extracted by the intra-graph inference module to finely predict local saliency of 360° images. In the other branch, the contextual features are extracted by the inter-graph inference module to effectively integrate semantic information of six rectilinear images. Finally, the feature maps are generated by the two branches fusion, and six corresponding rectilinear saliency maps are predicted. Extensive experiments on two popular saliency datasets illustrate the superiority of the proposed model, especially the improvement in KLD metric.

Two‐stage local attention network for salient object detection in remote sensing images

With the development of earth observation satellites, it is very meaningful to study methods for intelligently processing remote sensing images (RSIs). As an important pretreatment technique, salient object detection (SOD) can produce accurate saliency predictions in natural scenes. However, due to the complex object types, high noise levels, and different imaging angles in RSIs, it is still difficult to generate accurate saliency predictions in such images. This paper proposes a SOD method for RSIs, namely, the two-stage local attention network (TLANet). It uses two-stage attention (TSA) to capture the relationships between the pixels of features at different scales and uses an adjacent semantic refinement (ASR) module to enhance the flow of information across scales. In addition, a cascade structure is introduced to the decoder so that more semantic features can guide the feature decoding process. Moreover, a new loss function is proposed to improve the prediction effects obtained for salient edges. Through comprehensive experiments conducted on two RSIs datasets, it is demonstrated that the TLANet is superior to the state-of-the-art methods in both qualitative and quantitative comparisons.

Attention-based pyramid decoder network for salient object detection in remote sensing images

Salient object detection (SOD) in remote sensing images (RSIs) is a highly practical task. However, scale variations of salient objects and the diversity of salient objects in RSIs pose challenges for detection. To address these issues, an attention-based pyramid decoder network (APDNet) is proposed for SOD in RSIs. The APDNet consists of three key components. First, a multiscale attention block is constructed to extract multiscale information and relations between salient objects, suppressing the distraction of variations of object types and scales. Second, a pyramid decoder structure is designed to take full advantage of multilevel features by gradually fusing features from two adjacent layers for result prediction. This feature fusion allows APDNet to efficiently exploit multilevel feature information, thus enabling a better feature representation. Finally, a bidirectional residual refinement module is proposed to enhance the structural integrity and boundary retention of initial saliency predictions. Extensive experiments on two public datasets demonstrate the superiority and effectiveness of the proposed APDNet against other compared state-of-the-art methods.

Describing UI Screenshots in Natural Language

Being able to describe any user interface (UI) screenshot in natural language can promote understanding of the main purpose of the UI, yet currently it cannot be accomplished with state-of-the-art captioning systems. We introduce XUI, a novel method inspired by the global precedence effect to create informative descriptions of UIs, starting with an overview and then providing fine-grained descriptions about the most salient elements. XUI builds upon computational models for topic classification, visual saliency prediction, and natural language generation (NLG). XUI provides descriptions with up to three different granularity levels that, together, describe what is in the interface and what the user can do with it. We found that XUI descriptions are highly readable, are perceived to accurately describe the UI, and score similarly to human-generated UI descriptions. XUI is available as open-source software.