Saliency Prediction Model Research Articles

Audio-visual Saliency Prediction Model with Implicit Neural Representation

Audio-visual Saliency Prediction Model with Implicit Neural Representation

An egocentric video and eye-tracking dataset for visual search in convenience stores

We introduce an egocentric video and eye-tracking dataset, comprised of 108 first-person videos of 36 shoppers searching for three different products (orange juice, KitKat chocolate bars, and canned tuna) in a convenience store, along with the frame-centered eye fixation locations for each video frame. The dataset also includes demographic information about each participant in the form of an 11-question survey. The paper describes two applications using the dataset — an analysis of eye fixations during search in the store, and a training of a clustered saliency model for predicting saliency of viewers engaged in product search in the store. The fixation analysis shows that fixation duration statistics are very similar to those found in image and video viewing, suggesting that similar visual processing is employed during search in 3D environments and during viewing of imagery on computer screens. A clustering technique was applied to the questionnaire data, which resulted in two clusters being detected. Based on these clusters, personalized saliency prediction models were trained on the store fixation data, which provided improved performance in prediction saliency on the store video data compared to state-of-the art universal saliency prediction methods.

SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation

Recent advancements in deep convolutional neural networks have significantly improved the performance of saliency prediction. However, the manual configuration of the neural network architectures requires domain knowledge expertise and can still be time-consuming and error-prone. To solve this, we propose a new Neural Architecture Search (NAS) framework for saliency prediction with two contributions. Firstly, a supernet for saliency prediction is built with a weight-sharing network containing all candidate architectures, by integrating a dynamic convolution into the encoder–decoder in the supernet, termed SalNAS. Secondly, despite the fact that SalNAS is highly efficient (20.98 million parameters), it can suffer from the lack of generalization. To solve this, we propose a self-knowledge distillation approach, termed Self-KD, that trains the student SalNAS with the weighted average information between the ground truth and the prediction from the teacher model. The teacher model, while sharing the same architecture, contains the best-performing weights chosen by cross-validation. Self-KD can generalize well without the need to compute the gradient in the teacher model, enabling an efficient training system. By utilizing Self-KD, SalNAS outperforms other state-of-the-art saliency prediction models in most evaluation rubrics across seven benchmark datasets while being a lightweight model. The code will be available at https://github.com/chakkritte/SalNAS.

Progressive Adapting and Pruning: Domain-Incremental Learning for Saliency Prediction

Saliency prediction (SAP) plays a crucial role in simulating the visual perception function of human beings. In practical situations, humans can quickly grasp saliency extraction in new image domains. However, current SAP methods mainly concentrate on training models in single domains, which do not effectively handle diverse content and styles present in real-world images. As a result, it would be of great significance if SAP models could efficiently adjust to new image domains. To this end, this article aims to design SAP models that can imitate the incremental learning ability of human beings on multiple image domains and name domain-incremental saliency prediction (DISAP). To make a tradeoff between preventing the forgetting of historical domains and achieving high performance on new domains, we propose a progressively updated domain incremental encoder. This encoder consists of a domain-sharing branch and a domain-specific branch. The domain-sharing branch includes a feature selection mechanism to preserve crucial parameters after fine-tuning the model on each current domain. The remaining parameters are reserved to absorb knowledge from future domains. Furthermore, to capture the unique characteristics of each domain with relatively low computational overhead, we introduce a lightweight design to construct the domain-specific branch, enabling effective adaptation to new domains. Extensive experiments are conducted on multiple domain-incremental learning settings formed by four saliency prediction datasets, including Salicon, MIT1003, the art subset of CAT2000, and WebSal. The results demonstrate that our method outperforms existing methods significantly. The code is available at https://github.com/KaIi-github/DIL4SAP .

3D Pop-Ups: Omnidirectional image visual saliency prediction based on crowdsourced eye-tracking data in VR

The prediction of the visual saliency of omnidirectional images in VR is valuable for understanding visual behaviors. However, the equipment cost, software setups, hardware operation, and other constraints in acquiring eye-tracking data of omnidirectional images for visual saliency prediction would lead to a low training efficiency and prediction performance. Therefore, this paper proposed a crowdsourcing method based on recall fixations, which was used to collect and construct an omnidirectional image with eye-tracking dataset called CrowdSourcing360, which contained 16,200 pieces of data on 180 images. Using this dataset, a visual saliency prediction model CSnet360 was trained. Experiments demonstrated that the visual saliency prediction performance of the CSnet360 outperformed most existing models even without using actual gaze fixations. Finally, a VR interior design assistance prototype system was built and the preliminary study results indicated that the system could help designers to improve the quality of their design solutions.

Video saliency prediction for First-Person View UAV videos: Dataset and benchmark

Visual saliency prediction plays a crucial role in Unmanned Aerial Vehicle (UAV) video analysis tasks. In this paper, an eye-tracking dataset of the immersive viewing of videos captured from a First-Person View (FPV) of UAVs is developed, which consists of 200 video clips captured by DJI FPV drones, with a resolution of 4K QHD. The videos cover six different genres and fourteen unique scenes. To study human visual attention in watching FPV videos, fixation points are recorded using an eye tracker integrated into a VR headset. Based on the dataset, a simple yet effective FPV UAV video Saliency prediction model (FUAVSal) is proposed as a baseline, considering spatial–temporal feature, camera motion information and FPV prior. To establish benchmarks for saliency prediction in immersive FPV UAV video viewing, sixteen computational models are evaluated on this dataset. Detailed quantitative and qualitative comparisons are provided. The developed dataset and benchmarks aim to facilitate research on visual saliency prediction for First-Person View UAV videos.

How is Visual Attention Influenced by Text Guidance? Database and Model.

The analysis and prediction of visual attention have long been crucial tasks in the fields of computer vision and image processing. In practical applications, images are generally accompanied by various text descriptions, however, few studies have explored the influence of text descriptions on visual attention, let alone developed visual saliency prediction models considering text guidance. In this paper, we conduct a comprehensive study on text-guided image saliency (TIS) from both subjective and objective perspectives. Specifically, we construct a TIS database named SJTU-TIS, which includes 1200 text-image pairs and the corresponding collected eye-tracking data. Based on the established SJTU-TIS database, we analyze the influence of various text descriptions on visual attention. Then, to facilitate the development of saliency prediction models considering text influence, we construct a benchmark for the established SJTU-TIS database using state-of-the-art saliency models. Finally, considering the effect of text descriptions on visual attention, while most existing saliency models ignore this impact, we further propose a text-guided saliency (TGSal) prediction model, which extracts and integrates both image features and text features to predict the image saliency under various text-description conditions. Our proposed model significantly outperforms the state-of-the-art saliency models on both the SJTU-TIS database and the pure image saliency databases in terms of various evaluation metrics. The SJTU-TIS database and the code of the proposed TGSal model will be released at: https://github.com/IntMeGroup/TGSal.

From Discrete Representation to Continuous Modeling: A Novel Audio-Visual Saliency Prediction Model With Implicit Neural Representations

From Discrete Representation to Continuous Modeling: A Novel Audio-Visual Saliency Prediction Model With Implicit Neural Representations

Spatial memory-augmented visual navigation based on hierarchical deep reinforcement learning in unknown environments

Visual navigation in unknown environments poses significant challenges due to the presence of many obstacles and low-texture scenes. These factors may cause frequent collisions and tracking failure of feature-based visual Simultaneous Localization and Mapping (vSLAM). To avoid these issues, this paper proposes a spatial memory-augmented visual navigation system that combines a vSLAM module, a conventional global planner module, and a Hierarchical Reinforcement Learning (HRL)-based local planner module. Firstly, a real-time vSLAM named Salient-SLAM is proposed to augment the performance of visual navigation. Salient-SLAM creates a navigation mapping thread by combining a saliency prediction model to build a navigation map that categorizes environmental regions as occupied, explored, or noticeable. Spatial memory that contains spatial abstraction and saliency information of the environment can be further formed by encoding navigation maps, which helps the agent determine an optimal path towards its destination. An open-sourced saliency dataset is proposed to train the saliency prediction model by mimicking the visual attention mechanism. Secondly, a HRL method is proposed to automatically decompose local planning into a high-level policy selector and several low-level policies, where the latter produces actions to interact with the environment. We maximize entropy and minimize option correlation in learning low-level policies, aiming at acquiring diverse and independent behaviors. The simulation results show that the proposed HRL method outperforms competitive baselines by 6.29–10.85 % on Success Rate (SR) and 3.87–11.1 % on Success weighted by Path Length (SPL) metrics. By incorporating the spatial memory, SR, and SPL metrics can be augmented by an average of 9.85 % and 10.89 %, respectively.

Global semantic-guided network for saliency prediction

The human visual system effectively analyzes scenes based on local, global and semantic properties. Deep learning-based saliency prediction models adopted two-stream networks, leveraged prior knowledge of global semantics, or added long-range dependency modeling structures like transformers to incorporate global saliency information. However, they either brought high complexity to learning local and global features or neglected the design for enhancing local features. In this paper, we propose a Global Semantic-Guided Network (GSGNet), which first enriches global semantics through a modified transformer block and then incorporates semantic information into visual features from local and global perspectives in an efficient way. Multi-head self-attention in transformers captures global features, but lacks information communication within and between feature subspaces (heads) when computing the similarity matrix. To learn global representations and enhance interactions of the subspaces, we propose a Channel-Squeeze Spatial Attention (CSSA) module to emphasize channel-relevant information in a compression manner and learn global spatial relationships. To better fuse local and global contextual information, we propose a hybrid CNN-Transformer block called local–global fusion block (LGFB) for aggregating semantic features simply and efficiently. Experimental results on four public datasets demonstrate that our model achieves compelling performance compared with the state-of-the-art saliency prediction models on various evaluation metrics.

Transformer-Based Multi-Scale Feature Integration Network for Video Saliency Prediction

Most cutting-edge video saliency prediction models rely on spatiotemporal features extracted by 3D convolutions due to its local contextual cues acquirement ability. However, the shortage of 3D convolutions is that it cannot effectively capture long-term spatiotemporal dependencies in videos. To address this limitation, we propose a novel Transformer-based Multi-scale Feature Integration Network (TMFI-Net) for video saliency prediction, where the proposed TMFI-Net consists of a semantic-guided encoder and a hierarchical decoder. Firstly, embarking on the Transformer-based multi-level spatiotemporal features, the semantic-guided encoder enhances the features by inserting the high-level feature into each level feature via a top-down pathway and a longitudinal connection, which endows the multi-level spatiotemporal features with rich contextual information. In this way, the features are steered to give more concerns to saliency regions. Secondly, the hierarchical decoder employs a multi-dimensional attention (MA) module to elevate features along channel, temporal, and spatial dimensions jointly. Successively, the hierarchical decoder deploys a progressive decoding block to conduct an initial saliency prediction, which provides a coarse localization of saliency regions. Lastly, considering the complementarity of different saliency predictions, we integrate all initial saliency prediction results into the final saliency map. Comprehensive experimental results on four video saliency datasets firmly demonstrate that our model achieves superior performance when compared with the state-of-the-art video saliency models. The code is available at https://github.com/wusonghe/TMFI-Net.

Poster graphic design with your Eyes: An approach to automatic textual layout design based on visual perception

The layout of graphic design can betedious, especially for non-professional users who require additional time. Therefore, this paper proposes a graphic design layout method based on visual perception. First, a graphic design dataset is trained based on a fully convolutional network, and a visual saliency prediction model is established to predict the visual saliency of the background image. Second, a probability density map of the background image is obtained using a diffusion equation. Third, based on a generation algorithm for a text anchor box, anchor boxes of different positions, sizes, and lengths are obtained. The experimental results show that, compared with previous work, the text position and size obtained by this method have higher correlations with the baseline.

Three-Dimensional Drone Exploration with Saliency Prediction in Real Unknown Environments

In this paper, we propose a three-dimensional autonomous drone exploration system (ADES) with a lightweight and low-latency saliency prediction model to explore unknown environments. Several studies have applied saliency prediction in drone exploration. However, these studies are not sufficiently mature. For example, the computational complexity and the size of the developed prediction models have not been considered. In addition, some studies have only proposed saliency prediction models without actually applying them to drones. The ADES system proposed in this paper has a small and fast saliency prediction model and uses a novel drone exploration approach based on visual-inertial odometry to solve the practical problems encountered during drone exploration, such as collisions with and the repeated exploration of salient objects. The proposed ADES system performs comparably to the state-of-the-art, multiple-discontinuous-image saliency prediction network TA-MSNet and enables drones to explore unknown environments with high efficiency.

Saliency prediction based on multi-channel models of visual processing

Visual attention is one of the most significant characteristics for selecting and understanding the outside redundancy world. The human vision system cannot process all information simultaneously due to the visual information bottleneck. In order to reduce the redundant input of visual information, the human visual system mainly focuses on dominant parts of scenes. This is commonly known as visual saliency map prediction. This paper proposed a new psychophysical oriented saliency prediction architecture, which inspired by multi-channel model of visual cortex functioning in humans. The model consists of opponent color channels, wavelet transform, wavelet energy map, and contrast sensitivity function for extracting low-level image features and providing a maximum approximation to the low-level human visual system. The proposed model is evaluated using several datasets, including the MIT1003, MIT300, TORONTO, SID4VAM, and UCF Sports datasets. We also quantitatively and qualitatively compare the saliency prediction performance with that of other state-of-the-art models. Our model achieved strongly stable and better performance with different metrics on natural images, psychophysical synthetic images and dynamic videos. Additionally, we suggested that Fourier and spectral-inspired saliency prediction models outperformed other state-of-the-art non-neural network and even deep neural network models on psychophysical synthetic images. In the meantime, we suggest that deep neural networks need specific architectures and goals to be able to predict salient performance on psychophysical synthetic images better and more reliably. Finally, the proposed model could be used as a computational model of primate low-level vision system and help us understand mechanism of primate low-level vision system. The project page can be available at: https://sinodanishspain.github.io/HVS_SaliencyModel/ .

ADINet: Attention-guided dual-branch integration network for atypical visual saliency prediction

Compared with typically developed people, patients with autism spectrum disorder (ASD) have many differences, including visual attention. Recently, researchers have employed saliency prediction to conduct early diagnosis for patients with ASD by observing whether the fixation maps of the patients are consistent with prediction results. Though numerous saliency models have been designed, which promotes the performance of saliency prediction significantly, the efforts major in atypical attention prediction are fewer. Therefore, we propose a novel atypical saliency model, namely attention-guided dual-branch integration network, to perform atypical visual saliency prediction. Generally, our model is a U-shaped architecture including an encoder and a decoder. In the encoder, we first deploy the dual-branch fusion module to extract multiscale deep features, where the dual branches consist of a coarse branch and a fine branch. Particularly, the multiscale features of each branch are integrated in a bilateral way. Then, we employ the multiattention module consisting of dilated convolution, channel attention, and spatial attention to further elevate the high-level semantic cues. Particularly, the attention units are constructed in a cascaded way. Besides, to give an effective representation for the differences between atypical saliency estimation and the typical saliency estimation, we design a new discriminative mask to compute the loss function. Extensive experiments are performed on challenging datasets, and the results show the superiority and effectiveness of our atypical saliency model when compared with the state-of-the-art saliency prediction models.

Fast saliency prediction based on multi-channels activation optimization

The saliency prediction precision has improved rapidly with the development of deep learning technology, but the inference speed is slow due to the continuous deepening of networks. Hence, this paper proposes a fast saliency prediction model. Concretely, the siamese network backbone based on tailored EfficientNetV2 accelerates the inference speed while maintaining high performance. The shared parameters strategy further curbs parameter growth. Furthermore, we add multi-channel activation maps to optimize the fine features considering different channels and low-level visual features, which improves the interpretability of the model. Extensive experiments show that the proposed model achieves competitive performance on the standard benchmark datasets, and prove the effectiveness of our method in striking a balance between prediction accuracy and inference speed. Moreover, the small model size allows our method to be applied in edge devices. The code is available at: https://github.com/lscumt/fast-fixation-prediction.

Atypical Salient Regions Enhancement Network for visual saliency prediction of individuals with Autism Spectrum Disorder

Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder and can result in atypical visual perception towards stimuli. Existing atypical saliency prediction models for individuals with ASD are highly inspired by typical saliency prediction models, while neglecting their special visual traits and preferences which are different from neurotypical individuals. In this paper, we propose an Atypical Salient Regions Enhancement Network (ASD-ASRENet) based on an encoder–decoder architecture to predict the atypical visual saliency of individuals with ASD. Concretely, the output of the encoder is treated as the initial prediction, then four Atypical Salient Regions Enhancement (ASRE) modules, which are specially designed for individuals with ASD, are deployed at different stages of the decoder to emphasize the atypical salient regions and further complete the prediction in a progressive manner. Besides, considering the problem that the semantic information from high levels is gradually diluted while the effect of noises contained in low levels is increasingly stronger during the top-down transmission in the decoder, we further design a Global Semantics Flow (GSF) module, which captures the global interdependencies from both spatial perspective and channel perspective, to guide each integration stage in the decoder. Extensive experiments demonstrate the effectiveness of the proposed modules and our ASD-ASRENet achieves superior performance compared with all state-of-the-art models on the Saliency4ASD benchmark.

A Trained Humanoid Robot can Perform Human-Like Crossmodal Social Attention and Conflict Resolution

To enhance human-robot social interaction, it is essential for robots to process multiple social cues in a complex real-world environment. However, incongruency of input information across modalities is inevitable and could be challenging for robots to process. To tackle this challenge, our study adopted the neurorobotic paradigm of crossmodal conflict resolution to make a robot express human-like social attention. A behavioural experiment was conducted on 37 participants for the human study. We designed a round-table meeting scenario with three animated avatars to improve ecological validity. Each avatar wore a medical mask to obscure the facial cues of the nose, mouth, and jaw. The central avatar shifted its eye gaze while the peripheral avatars generated sound. Gaze direction and sound locations were either spatially congruent or incongruent. We observed that the central avatar’s dynamic gaze could trigger crossmodal social attention responses. In particular, human performance was better under the congruent audio-visual condition than the incongruent condition. Our saliency prediction model was trained to detect social cues, predict audio-visual saliency, and attend selectively for the robot study. After mounting the trained model on the iCub, the robot was exposed to laboratory conditions similar to the human experiment. While the human performance was overall superior, our trained model demonstrated that it could replicate attention responses similar to humans.

Viewing Bias Matters in 360° Videos Visual Saliency Prediction

360° video has been applied to many areas such as immersive contents, virtual tours, and surveillance systems. Compared to the field of view prediction on planar videos, the explosive amount of information contained in the omni-directional view on the entire sphere poses an additional challenge in predicting high-salient regions in 360° videos. In this work, we propose a visual saliency prediction model that directly takes 360° video in the equirectangular format. Unlike previous works that often adopted recurrent neural network (RNN) architecture for the saliency detection task, in this work, we utilize 3D convolution to a spatial-temporal encoder and generalize SphereNet kernels to construct a spatial-temporal decoder. We further study the statistical properties of viewing biases present in 360° datasets across various video types, which provides us with insights into the design of a fusing mechanism that incorporates the predicted saliency map with the viewing bias in an adaptive manner. The proposed model yields state-of-the-art performance, as evidenced by empirical results over renowned 360° visual saliency datasets such as Salient360!, PVS, and Sport360.

Intra- and Inter-Reasoning Graph Convolutional Network for Saliency Prediction on 360° Images

Cubic projection can be utilized to divide 360° images into multiple rectilinear images, with little distortion. However, the existing saliency prediction models fail to integrate semantic information of these images. In this paper, we address this by proposing an intra- and inter-reasoning graph convolutional network for saliency prediction on 360° images (SalReGCN360). The whole framework contains six sub-networks, each of which contains two branches. In the training phase, after utilizing Multiple Cubic Projection (MCP), six rectilinear images are simultaneously put into corresponding sub-networks. In one of the branches, the global features of a single rectilinear image are extracted by the intra-graph inference module to finely predict local saliency of 360° images. In the other branch, the contextual features are extracted by the inter-graph inference module to effectively integrate semantic information of six rectilinear images. Finally, the feature maps are generated by the two branches fusion, and six corresponding rectilinear saliency maps are predicted. Extensive experiments on two popular saliency datasets illustrate the superiority of the proposed model, especially the improvement in KLD metric.