Saliency Model Research Articles

Pixel Is All You Need: Adversarial Trajectory-Ensemble Active Learning for Salient Object Detection

Although weakly-supervised techniques can reduce the labeling effort, it is unclear whether a saliency model trained with weakly-supervised data (e.g., point annotation) can achieve the equivalent performance of its fully-supervised version. This paper attempts to answer this unexplored question by proving a hypothesis: there is a point-labeled dataset where saliency models trained on it can achieve equivalent performance when trained on the densely annotated dataset. To prove this conjecture, we proposed a novel yet effective adversarial trajectory-ensemble active learning (ATAL). Our contributions are three-fold: 1) Our proposed adversarial attack triggering uncertainty can conquer the overconfidence of existing active learning methods and accurately locate these uncertain pixels. 2) Our proposed trajectory-ensemble uncertainty estimation method maintains the advantages of the ensemble networks while significantly reducing the computational cost. 3) Our proposed relationship-aware diversity sampling algorithm can conquer oversampling while boosting performance. Experimental results show that our ATAL can find such a point-labeled dataset, where a saliency model trained on it obtained 97%-99% performance of its fully-supervised version with only 10 annotated points per image.

Visual saliency assistance mechanism based on visually impaired navigation systems

Visually impaired people face significant challenges in their work, studies, and daily lives. Nowadays, numerous visually impaired navigation devices are available to help visually impaired people solve daily life problems. These devices typically include modules for target recognition, distance measurement, and text reading aloud. They are designed to help visually impaired people avoid obstacles and understand the presence of things around them through text reading aloud. Due to the need to avoid all potential obstacles, the target recognition algorithms embedded in these devices must recognize a wide range of targets. However, the text reading aloud module cannot read all of them. Therefore, we designed a visual saliency assistance mechanism that simulates the regions that humans may pay the most attention to in the whole picture. The output of the visual saliency assistance mechanism is overlaid with the target recognition result, which can greatly reduce the target number of text reading aloud. This way, the visually impaired navigation device can not only help to avoid obstacles but also help visually impaired people understand the interest targets of most people in the whole picture. The visual saliency assistance mechanism we designed consists of three components: a spatio-temporal feature extraction (STFE) module, a spatio-temporal feature fusion (STFF) module, and a multi-scale feature fusion (MSFF) module. The STFF module fuses long-term spatio-temporal features and improves the temporal memory between frames. The MSFF module fully integrates information at different scales to improve the accuracy of saliency prediction. Therefore, this proposed visual saliency model can assist in the efficient operation of visually impaired navigation systems. The area under roc curve judd (AUC-J) metric of the proposed model was 93.9%, 93.8%, and 91.5% on three widely used saliency datasets: Holly-wood2, UCF Sports, and DHF1K, respectively. The results show that our proposed model outperforms the current state-of-the-art models.

Object identification in cerebral visual impairment characterized by gaze behavior and image saliency analysis

Individuals with cerebral visual impairment (CVI) have difficulties identifying common objects, especially when presented as cartoons or abstract images. In this study, participants were shown a series of images of ten common objects, each from five possible categories ranging from abstract black & white line drawings to color photographs. Fifty individuals with CVI and 50 neurotypical controls verbally identified each object and success rates and reaction times were collected. Visual gaze behavior was recorded using an eye tracker to quantify the extent of visual search area explored and number of fixations. A receiver operating characteristic (ROC) analysis was also carried out to compare the degree of alignment between the distribution of individual eye gaze patterns and image saliency features computed by the graph-based visual saliency (GBVS) model. Compared to controls, CVI participants showed significantly lower success rates and longer reaction times when identifying objects. In the CVI group, success rate improved moving from abstract black & white images to color photographs, suggesting that object form (as defined by outlines and contours) and color are important cues for correct identification. Eye tracking data revealed that the CVI group showed significantly greater visual search areas and number of fixations per image, and the distribution of eye gaze patterns in the CVI group was less aligned with the high saliency features of the image compared to controls. These results have important implications in helping to understand the complex profile of visual perceptual difficulties associated with CVI.

Quantum prices

Many retailers practice an extreme form of discrete pricing defined as quantum prices: differentiated products are priced using few and sparse price buckets. To show this, online data was collected for 350,000 products from over 65 fashion retailers in the U.S. and the U.K. This pricing strategy is observed within categories and across categories (i.e., similar or even disparately distinct products like jeans and bags have an identical price), as well as in product introductions, where new products come in at previous price buckets. Normalized indices indicate substantial price clustering after controlling for popular prices, convenient prices, assortment size, or digit endings. Quantum prices have implications for price adjustments through product shares, markdown prices, and for the law-of-one-price. A behavioral model of price salience and recall is discussed.

ADINet: Attention-guided dual-branch integration network for atypical visual saliency prediction

Compared with typically developed people, patients with autism spectrum disorder (ASD) have many differences, including visual attention. Recently, researchers have employed saliency prediction to conduct early diagnosis for patients with ASD by observing whether the fixation maps of the patients are consistent with prediction results. Though numerous saliency models have been designed, which promotes the performance of saliency prediction significantly, the efforts major in atypical attention prediction are fewer. Therefore, we propose a novel atypical saliency model, namely attention-guided dual-branch integration network, to perform atypical visual saliency prediction. Generally, our model is a U-shaped architecture including an encoder and a decoder. In the encoder, we first deploy the dual-branch fusion module to extract multiscale deep features, where the dual branches consist of a coarse branch and a fine branch. Particularly, the multiscale features of each branch are integrated in a bilateral way. Then, we employ the multiattention module consisting of dilated convolution, channel attention, and spatial attention to further elevate the high-level semantic cues. Particularly, the attention units are constructed in a cascaded way. Besides, to give an effective representation for the differences between atypical saliency estimation and the typical saliency estimation, we design a new discriminative mask to compute the loss function. Extensive experiments are performed on challenging datasets, and the results show the superiority and effectiveness of our atypical saliency model when compared with the state-of-the-art saliency prediction models.

Assessing saliency models of observers' visual attention on acquired facial differences.

Saliency models that predict observers' visual attention to facial differences could enable psychosocial interventions to help patients and their families anticipate staring behaviors. The purpose of this study was to assess the ability of existing saliency models to predict observers' visual attention to acquired facial differences arising from head and neck cancer and its treatment. Saliency maps predicted by graph-based visual saliency (GBVS), an artificial neural network (ANN), and a face-specific model were compared to observer fixation maps generated from eye-tracking of lay observers presented with clinical facial photographs of patients with a visible or functional impairment manifesting in the head and neck region. We used a linear mixed-effects model to investigate observer and stimulus factors associated with the saliency models' accuracy. The GBVS model predicted many irrelevant regions (e.g., shirt collars) as being salient. The ANN model underestimated observers' attention to facial differences relative to the central region of the face. Compared with GBVS and ANN, the face-specific saliency model was more accurate on this task; however, the face-specific model underestimated the saliency of deviations from the typical structure of human faces. The linear mixed-effects model revealed that the location of the facial difference (midface versus periphery) was significantly associated with saliency model performance. Model performance was also significantly impacted by interobserver variability. Existing saliency models are not adequate for predicting observers' visual attention to facial differences. Extensions of face-specific saliency models are needed to accurately predict the saliency of acquired facial differences arising from head and neck cancer and its treatment.

Comprehension of metaphors in patients with mild cognitive impairment: Evidence from behavioral and ERP data

The ability of metaphor comprehension reflects the cognitive status of elders. This study explored the ability of Chinese aMCI patients to access metaphorical meaning based on linguistic models of metaphor processing. ERPs were recorded from 30 aMCI patients and 30 control participants when judging the meaningfulness of literal sentences, conventional metaphors, novel metaphors, and anomalous expressions. The lower accuracy of the aMCI group revealed an impairment in metaphoric comprehension ability, but this difference was not reflected in ERPs data. In all participants, anomalous endings to sentences evoked the most negative N400 amplitude, whereas conventional metaphors evoked the smallest N400 amplitude. The LPC amplitude might be masked by a metaphor rebound effect when processing novel metaphors, which was consistent with the Graded Salience Model that novel metaphors needed further semantic integration. The results suggest that the aMCI patients may suffer an impairment in metaphorical meaning recognition, which the declined working memory may cause.

Global–local–global context-aware network for salient object detection in optical remote sensing images

For the salient object detection in optical remote sensing images (ORSI-SOD), many existing methods are trapped in a local–global mode, i.e., CNN-based encoder binds with a specific global context-aware module, struggling to deal with the challenging ORSIs with complex background and scale-variant objects. To solve this issue, we explore the synergy of the global-context-aware and local-context-aware modeling and construct a preferable global–local–global context-aware network (GLGCNet). In the GLGCNet, a transformer-based encoder is adopted to extract global representations, combining with local-context-aware features gathered from three saliency-up modules for comprehensive saliency modeling, and an edge assignment module is additionally employed to refine the preliminary detection. Specifically, the saliency-up module involves two components, one for global–local context-aware transfer towards pixel-wise dynamic convolution parameters prediction, the other for dynamically local-context aware modeling. The corresponding position-sensitive filter is aware of its previous global-wise focus, thus enhancing the spatial compactness of salient objects and encouraging the feature upsampling achievement for multi-scale feature combinations. The edge assignment module enhances the robustness of preliminary saliency prediction and assigns the semantic attributes of preliminary saliency cues to the shallow-level edge feature to obtain final complete salient objects in a spatially and semantically global manner. Extensive experiments demonstrate that the proposed GLGCNet surpasses 23 state-of-the-art methods on three popular datasets.

Inequality and identity salience

This paper provides a simple model of identity salience that is applied to the phenomenon of the recent rise in right-wing populism in the Western world. Trade and capital flows, skill-biased technological change, and migration have led to declining employment and wages in these economies and a parallel rise in economic and cultural populism, tapping into nativist sentiments. We argue that when long-term income stagnation for most of the population and decline for some go together with high rates of income growth at the very top, one has zero-sum economics and that naturally raises the possibility of using various kinds of social identities to claim a bigger share of a fixed sized pie. We show that in ethnically or racially polarized societies this naturally leads to the salience of social identities that enable majority ethnic groups to vote for policies that exclude minority groups so that they get a greater share of a dwindling surplus. In contrast, in more ethnically and racially homogeneous societies, this would instead lead to the demand for more pro-redistribution policies that involve greater provision of public goods.

An improved saliency model of visual attention dependent on image content.

Many visual attention models have been presented to obtain the saliency of a scene, i.e., the visually significant parts of a scene. However, some mechanisms are still not taken into account in these models, and the models do not fit the human data accurately. These mechanisms include which visual features are informative enough to be incorporated into the model, how the conspicuity of different features and scales of an image may integrate to obtain the saliency map of the image, and how the structure of an image affects the strategy of our attention system. We integrate such mechanisms in the presented model more efficiently compared to previous models. First, besides low-level features commonly employed in state-of-the-art models, we also apply medium-level features as the combination of orientations and colors based on the visual system behavior. Second, we use a variable number of center-surround difference maps instead of the fixed number used in the other models, suggesting that human visual attention operates differently for diverse images with different structures. Third, we integrate the information of different scales and different features based on their weighted sum, defining the weights according to each component's contribution, and presenting both the local and global saliency of the image. To test the model's performance in fitting human data, we compared it to other models using the CAT2000 dataset and the Area Under Curve (AUC) metric. Our results show that the model has high performance compared to the other models (AUC = 0.79 and sAUC = 0.58) and suggest that the proposed mechanisms can be applied to the existing models to improve them.

A Novel Lightweight Audio-visual Saliency Model for Videos

Audio information has not been considered an important factor in visual attention models regardless of many psychological studies that have shown the importance of audio information in the human visual perception system. Since existing visual attention models only utilize visual information, their performance is limited but also requires high-computational complexity due to the limited information available. To overcome these problems, we propose a lightweight audio-visual saliency (LAVS) model for video sequences. To the best of our knowledge, this article is the first trial to utilize audio cues for an efficient deep-learning model for the video saliency estimation. First, spatial-temporal visual features are extracted by the lightweight receptive field block (RFB) with the bidirectional ConvLSTM units. Then, audio features are extracted by using an improved lightweight environment sound classification model. Subsequently, deep canonical correlation analysis (DCCA) aims at capturing the correspondence between audio and spatial-temporal visual features, thus obtaining a spatial-temporal auditory saliency. Lastly, the spatial-temporal visual and auditory saliency are fused to obtain the audio-visual saliency map. Extensive comparative experiments and ablation studies validate the performance of the LAVS model in terms of effectiveness and complexity.

Pixel-Domain Just Noticeable Difference Modeling with Heterogeneous Color Features.

With the rapidly emerging user-generated images, perception compression for color image is an inevitable mission. Whilst in existing just noticeable difference (JND) models, color-oriented features are not fully taken into account for coinciding with HVS perception characteristics, such as sensitivity, attention, and masking. To fully imitate the color perception process, we extract color-related feature parameters as local features, including color edge intensity and color complexity, as well as region-wise features, including color area proportion, color distribution position and color distribution dispersion, and inherent feature irrelevant to color content called color perception difference. Then, the potential interaction among them is analyzed and modeled as color contrast intensity. To utilize them, color uncertainty and color saliency are envisaged to emanate from feature integration in the information communication framework. Finally, color and uncertainty saliency models are applied to improve the conventional JND model, taking the masking and attention effect into consideration. Subjective and objective experiments validate the effectiveness of the proposed model, delivering superior noise concealment capacity compared with start-of-the-art works.

Salient Objects in Clutter.

In this paper, we identify and address a serious design bias of existing salient object detection (SOD) datasets, which unrealistically assume that each image should contain at least one clear and uncluttered salient object. This design bias has led to a saturation in performance for state-of-the-art SOD models when evaluated on existing datasets. However, these models are still far from satisfactory when applied to real-world scenes. Based on our analyses, we propose a new high-quality dataset and update the previous saliency benchmark. Specifically, our dataset, called Salient Objects in Clutter (SOC), includes images with both salient and non-salient objects from several common object categories. In addition to object category annotations, each salient image is accompanied by attributes that reflect common challenges in common scenes, which can help provide deeper insight into the SOD problem. Further, with a given saliency encoder, e.g., the backbone network, existing saliency models are designed to achieve mapping from the training image set to the training ground-truth set. We therefore argue that improving the dataset can yield higher performance gains than focusing only on the decoder design. With this in mind, we investigate several dataset-enhancement strategies, including label smoothing to implicitly emphasize salient boundaries, random image augmentation to adapt saliency models to various scenarios, and self-supervised learning as a regularization strategy to learn from small datasets. Our extensive results demonstrate the effectiveness of these tricks. We also provide a comprehensive benchmark for SOD, which can be found in our repository: https://github.com/DengPingFan/SODBenchmark.

Learning How to Detect Salient Objects in Nighttime Scenes

The detection of salient objects in nighttime scene settings is an essential research issue in computer vision. None of the known approaches can accurately anticipate salient objects in the nighttime scenes. Due to the lack of visible light, spatial visual information cannot be accurately perceived by traditional and deep network models. This paper proposed a Mountain Basin Network (MBNet) to identify salient objects for distinguishing the pixel-level saliency of low-light images. To improve the objects localizations and pixel classification performances, the proposed model incorporated a High-Low Feature Aggregation Module (HLFA) to synchronize the information from a high-level branch (named Bal-Net) and a low-level branch (called Mol-Net) to fuse the global and local context, and a Hierarchical Supervision Module (HSM) was embedded to aid in obtaining accurate salient objects, particularly the small ones. In addition, a multi-supervised integration technique was explored to optimize the structure and borders of salient objects. In the meantime, to facilitate more investigation into nighttime scenes and assessment of visual saliency models, we created a new nighttime dataset consisting of thirteen categories and a total of one thousand low-light images. Our experimental results demonstrated that the suggested MBNet model outperforms seven current state-of-the-art methods for salient object detection in nighttime scenes.

Quality-Driven Dual-Branch Feature Integration Network for Video Salient Object Detection

Video salient object detection has attracted growing interest in recent years. However, some existing video saliency models often suffer from the inappropriate utilization of spatial and temporal cues and the insufficient aggregation of different level features, leading to remarkable performance degradation. Therefore, we propose a quality-driven dual-branch feature integration network majoring in the adaptive fusion of multi-modal cues and sufficient aggregation of multi-level spatiotemporal features. Firstly, we employ the quality-driven multi-modal feature fusion (QMFF) module to combine the spatial and temporal features. Particularly, the quality scores estimated from each level’s spatial and temporal cues are not only used to weigh the two modal features but also to adaptively integrate the coarse spatial and temporal saliency predictions into the guidance map, which further enhances the two modal features. Secondly, we deploy the dual-branch-based multi-level feature aggregation (DMFA) module to integrate multi-level spatiotemporal features, where the two branches including the progressive decoder branch and the direct concatenation branch sufficiently explore the cooperation of multi-level spatiotemporal features. In particular, in order to provide an adaptive fusion for the outputs of the two branches, we design the dual-branch fusion (DF) unit, where the channel weight of each output can be learned jointly from the two outputs. The experiments conducted on four video datasets clearly demonstrate the effectiveness and superiority of our model against the state-of-the-art video saliency models.

STI-Net: Spatiotemporal integration network for video saliency detection

Image saliency detection, to which much effort has been devoted in recent years, has advanced significantly. In contrast, the community has paid little attention to video saliency detection. Especially, existing video saliency models are very likely to fail in videos with difficult scenarios such as fast motion, dynamic background, and nonrigid deformation. Furthermore, performing video saliency detection directly using image saliency models that ignore video temporal information is inappropriate. To alleviate this issue, this study proposes a novel end-to-end spatiotemporal integration network (STI-Net) for detecting salient objects in videos. Specifically, our method is made up of three key steps: feature aggregation, saliency prediction, and saliency fusion, which are used sequentially to generate spatiotemporal deep feature maps, coarse saliency predictions, and the final saliency map. The key advantage of our model lies in the comprehensive exploration of spatial and temporal information across the entire network, where the two features interact with each other in the feature aggregation step, are used to construct boundary cue in the saliency prediction step, and also serve as the original information in the saliency fusion step. As a result, the generated spatiotemporal deep feature maps can precisely and completely characterize the salient objects, and the coarse saliency predictions have well-defined boundaries, effectively improving the final saliency map's quality. Furthermore, “shortcut connections” are introduced into our model to make the proposed network easy to train and obtain accurate results when the network is deep. Extensive experimental results on two publicly available challenging video datasets demonstrate the effectiveness of the proposed model, which achieves comparable performance to state-of-the-art saliency models.

Multi-stakeholder’s sustainable requirement analysis for smart manufacturing systems based on the stakeholder value network approach

Sustainable development is of great significance to all Smart Manufacturing Systems (SMSs). Complex multi-stakeholder requirements influence the design and implementation of SMSs. Capturing multi-stakeholder requirements of SMSs are very important to manufacturer for construction or business process reengineering (BPR) phase of SMSs. However, there is a lack of the research on SMSs requirements (SMSs-R) list from the sustainable value view of multi-stakeholder and a lack of illustrating the complex relationship between SMSs and multi-stakeholder. Therefore, this study provided a systematic method to capture the SMSs-R list and explain the complex relationship between the SMSs and their multi-stakeholder. The contribution of this research study offers the comprehensive approach to capture SMSs-R based on stakeholder salience model and stakeholder value network. Secondly, the quantification analysis of SMS-R is proposed to determine the urgency and importance through the comprehensive fuzzy Kano model. Thirdly, SMSs-R list is obtained through the systematic evaluation methods, including graph theory, dependency relationship matrix, and network statistical. Finally, a case study in Chinese company is used to check the feasibility of the proposed approach. The results show that our method can provide a visual and effective analysis for traditional manufacturing enterprises, and improve the operability and accuracy of SMSs-R list in the preliminary design or BPR stage.

CACFNet: Fabric defect detection via context-aware attention cascaded feedback network

Fabric defect detection plays an irreplaceable role in the quality control of the textile manufacturing industry, but it is still a challenging task due to the diversity and complexity of defects and environmental factors. Visual saliency models imitating the human vision system can quickly determine the defect regions from the complex texture background. However, most visual saliency-based methods still suffer from incomplete predictions owing to the variability of fabric defects and low contrast with the background. In this paper, we develop a context-aware attention cascaded feedback network for fabric defect detection to achieve more accurate predictions, in which a parallel context extractor is designed to characterize the multi-scale contextual information. Moreover, a top-down attention cascaded feedback module was devised adaptively to select the important multi-scale complementary information and then transmit it to an adjacent shallower layer to compensate for the inconsistency of information among layers for accurate location. Finally, a multi-level loss function is applied to guide our model for generating more accurate prediction results via optimizing multiple side-output predictions. Experimental results on the two fabric datasets built under six widely used evaluation metrics demonstrate that our proposed framework outperforms state-of-the-art models remarkably.

Method of gaze extraction in bionic vision

Aiming at the low efficiency of information processing in a robot vision system, our work studies how to extract fixed points during tasks so that significant and interesting target areas in the scene can be located. First, the architecture of a gaze extraction model of bionic vision is proposed, which consists of a spatial visual saliency model and a task-driven fixed point extraction model. The task-driven fixed point extraction process is represented as a closed-loop control problem. To couple exploration with the sampling process, a Q learning algorithm based on a random strategy is adopted, which is a top-down task-driven gaze extraction in the time dimension. It is fused with the spatial-based visual saliency model to form a spatiotemporal hybrid gaze extraction model to determine the fixed point in the final image. Finally, our qualitative image visualization experiment indicates that the fixed point extraction results of the model in two consecutive frames are close to the truth value obtained by an eye tracker. The quantitative area under the curve and average angular error data confirms the effectiveness of the model in predicting and extracting fixed points.

Saliency Prediction in Uncategorized Videos Based on Audio-Visual Correlation

Substantial research has been done in saliency modeling to make intelligent machines that can perceive and interpret their surroundings and focus only on the salient regions in a visual scene. But existing spatio-temporal saliency models either treat videos as merely image sequences excluding any audio information or are unable to cope with inherently varying content. Based on the hypothesis that an audiovisual saliency model will perform better than traditional spatio-temporal saliency models, this work aims to provide a generic preliminary audio/video saliency model. This is achieved by augmenting visual saliency map with an audio saliency map computed by synchronizing low-level audio and visual features. The proposed model was evaluated using different criteria against eye fixations data for a publicly available video dataset DIEM. The evaluation results show that the model outperforms two state-of-the-art visual spatio-temporal saliency models supporting our hypothesis that an audiovisual model performs better in comparison to a visual model for natural uncategorized videos.