Saliency Model Research Articles

LFRNet: Localizing, Focus, and Refinement Network for Salient Object Detection of Surface Defects

Salient object detection of surface defects is one of the surface defect detection tasks, which aims at highlighting the defect regions from the surface of strip steel, magnetic tale, road, and so on. However, the performance of existing methods degrades dramatically when dealing with complex scenarios such as low-contrast of defect regions and various defect shapes. Therefore, in this paper, we propose a novel saliency model, namely localizing, focus, and refinement network (LFRNet), which consists of the semantic-guided localizing module, the context-driven focus module, and the edge-aware refinement module. Firstly, the semantic-guided localizing module deploys the graph reasoning (GR) unit and global attention (GA) unit to localize the potential defect regions from a global view. Secondly, the context-driven focus module employs the split context (SC) unit and the mutual attention (MA) unit to perform the identification process via the introduction of spatial detail features. Lastly, to further improve the accuracy of the detection results, we deploy the edge-aware refinement (ER) module, which introduces the boundary cues via the edge generation (EG) unit and aggregates the localizing result, the focus results, and the edge information into the high-quality detection map. Extensive experiments on four public defect datasets clearly show the effectiveness and superiority of the proposed LFRNet, where the LFRNet obtains an improvement of 4.1%, 5.7%, 1.0%, and 0.8% on FM, WF, EM, and SM respectively when compared with the top-level method AEP.

Stakeholder Mapping For Social Entrepreneurship Projects

The article discusses stakeholder theory and stakeholder definitions. The methods of stakeholder analysis known in the world practice are investigated: power-influence-interest grid, stakeholder cube, salience model. A step-by-step stakeholders mapping outline is given. The methodology for categorizing stakeholders has been developed by the author, based on a resource approach. Based on three types of resources (administrative, financial and informational), an integral indicator of the importance of stakeholders is calculated and mapped. The stakeholder mapping allows you to optimally allocate time and attention to stakeholders, as well as develop effective communication plans for the company and stakeholder engagement strategy.

Transformer-Based Efficient Salient Instance Segmentation Networks With Orientative Query

Salient instance segmentation (SIS) can be consid-ered as the next generation task for saliency detection com-munity. As a new challenging task, most of the existing state-of-the-art methods are built on the mainstream Mask R-CNN architecture. However, this mechanism relies heavily on hand-designed anchors and NMS post-processing. In this paper, we provide a one stage SIS framework with transformers, termed Orientative Query Transformer (OQTR). To leverage long-range dependencies of transformer, a cross fusion module is designed to efficiently fuse the global features in encoder and salient query features for salient mask prediction. Furthermore, derived from the center prior in traditional saliency models, we propose an orientative query that is considered as the initial salient object query to accelerate convergence. In addition, to mitigate the issue of lacking large-scale dataset with salient instance labels, we collect a new SIS dataset (SIS10K) containing over 10K images elaborately annotated with both object- and instance-level labels to promote the community. Without any post-processing, our end-to-end OQTR framework significantly surpass the top-1 RDPNet by an average of 13.1% AP scores across all three challenging datasets, showing a strong performance of the proposed OQTR. The code and the dataset proposed in this work are available at: https://github.com/ssecv/OQTR.

What Catch Your Attention in SAR Images: Saliency Detection Based on Soft-Superpixel Lacunarity Cue

In existing superpixel-wise saliency detection algorithms, superpixel generation often is an isolated preprocessing step. The performance of saliency maps is determined by the accuracy of superpixels to a certain extent. However, it is still a challenge to develop a stable superpixel generation method. In this article, we attempt to incorporate the superpixel generation and saliency calculation steps into an end-to-end trainable deep network. First, we employ a recently proposed differentiable superpixel generation method to over-segment the synthetic aperture radar (SAR) images, which outputs the possibility that the pixels assigned to neighbor superpixels (soft superpixel). In saliency calculation part, as one of our main contributions, we propose a differentiable and computationally simple saliency model, i.e., lacunarity cue. It is inspired by the fact that generally the backscattering intensity of regions of interest (ROIs) in SAR images irregularly fluctuates, while the areas with consistent pixels are often ignored as the clusters. We improve the pixelwise box differential dimension algorithm to measure the irregularity of scattering points in a superpixel. The superpixel generation and saliency calculation can be implemented under a unified deep network. Hence, the shapes of the superpixels can be iteratively adjusted according to the saliency maps until the ROIs are correctly detected. Experiments on real SAR images with different sizes and scenes show that the saliency maps can effectively highlight the target areas, thus outperforming the state-of-the-art saliency detection models.

An Energy-Based Prior for Generative Saliency.

We propose a novel generative saliency prediction framework that adopts an informative energy-based model as a prior distribution. The energy-based prior model is defined on the latent space of a saliency generator network that generates the saliency map based on a continuous latent variables and an observed image. Both the parameters of saliency generator and the energy-based prior are jointly trained via Markov chain Monte Carlo-based maximum likelihood estimation, in which the sampling from the intractable posterior and prior distributions of the latent variables are performed by Langevin dynamics. With the generative saliency model, we can obtain a pixel-wise uncertainty map from an image, indicating model confidence in the saliency prediction. Different from existing generative models, which define the prior distribution of the latent variables as a simple isotropic Gaussian distribution, our model uses an energy-based informative prior which can be more expressive in capturing the latent space of the data. With the informative energy-based prior, we extend the Gaussian distribution assumption of generative models to achieve a more representative distribution of the latent space, leading to more reliable uncertainty estimation. We apply the proposed frameworks to both RGB and RGB-D salient object detection tasks with both transformer and convolutional neural network backbones. We further propose an adversarial learning algorithm and a variational inference algorithm as alternatives to train the proposed generative framework. Experimental results show that our generative saliency model with an energy-based prior can achieve not only accurate saliency predictions but also reliable uncertainty maps that are consistent with human perception.

RSSFormer: Foreground Saliency Enhancement for Remote Sensing Land-Cover Segmentation.

High spatial resolution (HSR) remote sensing images contain complex foreground-background relationships, which makes the remote sensing land cover segmentation a special semantic segmentation task. The main challenges come from the large-scale variation, complex background samples and imbalanced foreground-background distribution. These issues make recent context modeling methods sub-optimal due to the lack of foreground saliency modeling. To handle these problems, we propose a Remote Sensing Segmentation framework (RSSFormer), including Adaptive TransFormer Fusion Module, Detail-aware Attention Layer and Foreground Saliency Guided Loss. Specifically, from the perspective of relation-based foreground saliency modeling, our Adaptive Transformer Fusion Module can adaptively suppress background noise and enhance object saliency when fusing multi-scale features. Then our Detail-aware Attention Layer extracts the detail and foreground-related information via the interplay of spatial attention and channel attention, which further enhances the foreground saliency. From the perspective of optimization-based foreground saliency modeling, our Foreground Saliency Guided Loss can guide the network to focus on hard samples with low foreground saliency responses to achieve balanced optimization. Experimental results on LoveDA datasets, Vaihingen datasets, Potsdam datasets and iSAID datasets validate that our method outperforms existing general semantic segmentation methods and remote sensing segmentation methods, and achieves a good compromise between computational overhead and accuracy. Our code is available at https://github.com/Rongtao-Xu/RepresentationLearning/tree/main/RSSFormer-TIP2023.

A context-aware progressive attention aggregation network for fabric defect detection

Fabric defect detection plays a critical role for measuring quality control in the textile manufacturing industry. Deep learning-based saliency models can quickly spot the most interesting regions that attract human attention from the complex background, which have been successfully applied in fabric defect detection. However, most of the previous methods mainly adopted multi-level feature aggregation yet ignored the complementary relationship among different features, and thus resulted in poor representation capability for the tiny and slender defects. To remedy these issues, we propose a novel saliency-based fabric defect detection network, which can exploit the complementary information between different layers to enhance the representation features ability and discrimination of defects. Specifically, a multi-scale feature aggregation unit (MFAU) is proposed to effectively characterize the multi-scale contextual features. Besides, a feature fusion refinement module (FFR) composed of an attention fusion unit (AFU) and an auxiliary refinement unit (ARU) is designed to exploit complementary important information and further refine the input features for enhancing the discriminative ability of defect features. Finally, a multi-level deep supervision (MDS) is adopted to guide the model to generate more accurate saliency maps. Under different evaluation metrics, our proposed method outperforms most state-of-the-art methods on our developed fabric datasets.

Image Perceptual Hashing for Content Authentication Based on Geometric Invariant Vector Distance

Image perceptual hashing is broadly applied in image content authentication, recognition, retrieval, and social media hotspot event detection. An image authentication algorithm is put forward based on the Itti visual saliency model and geometric invariant vector distance. To begin with, the image is preprocessed and weighted by the Itti model and contourlet transform. After that, the weighted image is randomly divided into blocks, and the image feature vector is constructed by calculating the geometric invariant vector distance on both Hu invariant moment vector and maximum singular value vector of the random blocks. In the end, the feature vector is quantized and encrypted to generate the ultimate hash. Experimental results illustrate that when the threshold T = 70, the true positive rate P T P R for duplicate images stands at 0.96574, while the false rate P F P R of different images is merely 0.0224, with the total error rate reaching the minimum value (0.0566). Furthermore, the AUC value of the proposed algorithm is 0.9951, which is higher than that of the comparison algorithms, indicating that the algorithm has better performance than other state-of-the-art algorithms in terms of various visual content-preserving attacks.

Detecting dynamic visual attention in augmented reality aided navigation environment based on a multi-feature integration fully convolutional network

ABSTRACT Visual attention detection, as an important concept for human visual behavior research, has been widely studied. However, previous studies seldom considered the feature integration mechanism to detect visual attention and rarely considered the differences due to different geographical scenes. In this paper, we use an augmented reality aided (AR-aided) navigation experimental dataset to study human visual behavior in a dynamic AR-aided environment. Then, we propose a multi-feature integration fully convolutional network (M-FCN) based on a self-adaptive environment weight (SEW) to integrate RGB-D, semantic, optical flow and spatial neighborhood features to detect human visual attention. The result shows that the M-FCN performs better than other state-of-the-art saliency models. In addition, the introduction of feature integration mechanism and the SEW can improve the accuracy and robustness of visual attention detection. Meanwhile, we find that RGB-D and semantic features perform best in different road routes and road types, but with the increase in road type complexity, the expressiveness of these two features weakens, and the expressiveness of optical flow and spatial neighborhood features increases. The research is helpful for AR-device navigation tool design and urban spatial planning.

Evaluating the faithfulness of saliency maps in explaining deep learning models using realistic perturbations

Deep Learning has reached human-level performance in several medical tasks including classification of histopathological images. Continuous effort has been made at finding effective strategies to interpret these types of models, among them saliency maps, which depict the weights of the pixels on the classification as an heatmap of intensity values, have been by far the most used for image classification. However, there is a lack of tools for the systematic evaluation of saliency maps, and existing works introduce non-natural noise such as random or uniform values. To address this issue, we propose an approach to evaluate the faithfulness of the saliency maps by introducing natural perturbations in the image, based on oppose-class substitution, and studying their impact on evaluation metrics adapted from saliency models. We validate the proposed approach on a breast cancer metastases detection dataset PatchCamelyon with 327,680 patches of histopathological images of sentinel lymph node sections. Results show that GradCAM, Guided-GradCAM and gradient-based saliency map methods are sensitive to natural perturbations and correlate to the presence of tumor evidence in the image. Overall, this approach proves to be a solution for the validation of saliency map methods without introducing confounding variables and shows potential for application on other medical imaging tasks.

Boundary-Aware Salient Object Detection in Optical Remote-Sensing Images

Different from the traditional natural scene images, optical remote-sensing images (RSIs) suffer from diverse imaging orientations, cluttered backgrounds, and various scene types. Therefore, the object-detection methods salient to optical RSIs require effective localization and segmentation to deal with complex scenarios, especially small targets, serious occlusion, and multiple targets. However, the existing models’ experimental results are incapable of distinguishing salient objects and backgrounds using clear boundaries. To tackle this problem, we introduce boundary information to perform salient object detection in optical RSIs. Specifically, we first combine the encoder’s low-level and high-level features (i.e., abundant local spatial and semantic information) via a feature-interaction operation, yielding boundary information. Then, the boundary cues are introduced into each decoder block, where the decoder features are directed to focus more on the boundary details and objects simultaneously. In this way, we can generate high-quality saliency maps which can highlight salient objects from optical RSIs completely and accurately. Extensive experiments are performed on a public dataset (i.e., ORSSD dataset), and the experimental results demonstrate the effectiveness of our model when compared with the cutting-edge saliency models.

SRI-Net: Similarity retrieval-based inference network for light field salient object detection

The cutting-edge RGB saliency models are prone to fail for some complex scenes, while RGB-D saliency models are often affected by inaccurate depth maps. Fortunately, light field images can provide a sufficient spatial layout depiction of 3D scenes. Therefore, this paper focuses on salient object detection of light field images, where a Similarity Retrieval-based Inference Network (SRI-Net) is proposed. Due to various focus points, not all focal slices extracted from light field images are beneficial for salient object detection, thus, the key point of our model lies in that we attempt to select the most valuable focal slice, which can contribute more complementary information for the RGB image. Specifically, firstly, we design a focal slice retrieval module (FSRM) to choose an appropriate focal slice by measuring the foreground similarity between the focal slice and RGB image. Secondly, in order to combine the original RGB image and the selected focal slice, we design a U-shaped saliency inference module (SIM), where the two-stream encoder is used to extract multi-level features, and the decoder is employed to aggregate multi-level deep features. Extensive experiments are conducted on two widely used light field datasets, and the results firmly demonstrate the superiority and effectiveness of the proposed SRI-Net.

Can a saliency model using feature sets derived cityscapes predict cultural differences in visual search asymmetry?

Can a saliency model using feature sets derived cityscapes predict cultural differences in visual search asymmetry?

Meaning maps detect the removal of local scene content but deep saliency models do not

Meaning maps detect the removal of local scene content but deep saliency models do not

Scene Complexity Captures the Detail Trace of Visual Episodic Memory

Memory research has established that human observers use schemas, mental constructs that represent scene elements and relationships between those elements, as the basis for encoding information into visual episodic memory. Newly developed computational methods now allow the extraction of this schema information from the human observer via two-dimensional maps known as Visual Memory Schema (VMS) maps (Kyle-Davidson, Bors & Evans, 2019). VMS maps reveal which regions in a scene image are responsible for a human recalling that scene. VMS Maps are highly consistent among observers and are not explained solely by computational saliency models or eye-fixations. However, previous work also suggests that humans, in addition to schemas, retain rich detail of the scene, and the presence of that detailed information aids in later recognition of the same. Findings show that scenes considered to be more detailed (scenes with man-made elements) are remembered better than less detailed scenes (natural scene). Similarly, images that have detail removed (compared to a source image) appear less memorable than images that maintain the original detail (Baddeley & Evans, 2019). We investigated whether perceptual image complexity could stand as an analogue for the detail trace in visual memory, exploring how the complexity of an image relates to that image's memorability. Our approach was to conduct a behavioural experiment gathering two-dimensional complexity information from humans for a scene dataset that already has corresponding schema information (VMS Maps), which allowed us to explore complexity and memorability from a two-dimensional perspective. Next, we trained a neural network to predict complexity scores, and investigate this relationship for another scene memory dataset, now comparing the predicted complexity scores with ground truth memory for the images. The findings indicate that both extracted, and predicted perceptual complexity of the scene, relates to how well observers remember that same scene.

Dense Attentive Feature Enhancement for Salient Object Detection

Attention mechanisms have been proven highly effective for salient object detection. Most previous works utilize attention as a self-gated module to reweigh the feature maps at different levels independently. However, they are limited to certain-level guidance and could not satisfy the need of both accurately detecting intact objects and maintaining their detailed boundaries. In this paper, we build dense attention upon features from multiple levels simultaneously and propose a novel Dense Attentive Feature Enhancement (DAFE) module for efficient feature enhancement in saliency detection. DAFE stacks several attentional units and densely connects attentive feature output from current unit to its all subsequent units. This allows feature maps at deep units to absorb attentive information from shallow units, thus more discriminative information can be efficiently selected at the final output. Note that DAFE is plug and play, which can be effortlessly inserted into any saliency or video saliency models for their performance improvements. We further instantiate a highly effective Dense Attentive Feature Enhancement Network (DAFE-Net) for accurate salient object detection. DAFE-Net constructs DAFE over the aggregation feature that contains both semantics and saliency details, the entire salient objects and their boundaries can be well retained through dense attentions. Extensive experiments demonstrate that the proposed DAFE module is highly effective, and the DAFE-Net performs favorably compared with state-of-the-art approaches.

Equipment noise evaluation based on auditory saliency map

This study focuses on noise evaluation and targeted noise reduction efforts. Goals of this study are to propose a noise annoyance evaluation (D) by developing the auditory saliency model. D replaces complex subjective measurements with concrete objective values. This noise evaluation method improves the accuracy of sound quality evaluation by avoiding the bias caused by different emphasis in traditional noise evaluation schemes and the huge workload caused by using questionnaires. Whether it is equipment operation noise or environmental noise, analyzing the noise frequency structure is an important part of achieving targeted noise reduction. Through the EA model (Early Acoustic Model), which is more suitable for the actual working conditions of the human ear, this study has combined with the energy algorithm to establish the noise annoyance factor (BP), which can accurately identify the main frequencies in the noise signal that affect the sound quality. Meanwhile, they also have a vital role in analyzing the frequency structure of the movement. This research verifies the accuracy of noise annoyance evaluation (D) by analyzing the simulation signal, vacuum cleaner working signal, and fan heat dissipation signal. The noise annoyance factor (BP) is used to locate the center frequency range of a group of acoustic signals that produce the most significant noise annoyance to achieve the purpose of targeted noise reduction. This study provided a new solution for sound quality evaluation and targeted noise reduction.

Exploring Stakeholder Salience Perspective of Firm Reputation: Evidence From Malaysia

This paper aims to explore the components of a firm reputation based on public information disclosure and its relationship with the stakeholder salience model. The firm reputation components are analysed using content analysis of the firm's annual report. The study collects data on the firm reputation components from 200 annual reports of Malaysian public listed companies over a period of 4 years (2016-2019). The results of the study found that there are seven (7) components that can be used to reflect on the firm reputational aspects as disclosed in the firm's annual report. The components include performance, product and service quality, innovation, workplace, governance, citizenship/CSR, and leadership. The findings also reveal three highest components which are product and service quality, performance, and workplace, are supported by the stakeholder salience theory. Our analysis provides evidence that power, legitimacy, and urgency become crucial attributes in determining the key stakeholders of the firms (customers, employees, and shareholders) when the firms have different types of stakeholders. Operatively, our evidence can help regulators, investors and researchers to better understand firm reputational components as per information disclosed by the firms, especially for the firms operating in developing and emerging markets. Further, this study is among the earliest that provide the interrelation between firm reputational dimension and stakeholder salience.

Stakeholder roles in artificial intelligence projects

Algorithmic decision-making implemented through artificial intelligence (AI) projects is augmenting or replacing human decision-making across numerous industries. Although AI systems may impact individuals and society in life-and-death situations, project organizations or plans may ignore the concerns of the passive stakeholders. This research describes the components, lifecycle, and characteristics of AI projects. It employs stakeholder theory and a systematic literature review with thematic analysis to identify and classify individuals, groups, and organizations into six stakeholder project roles. It configures the stakeholder salience model with a harm attribute to identify passive stakeholders—individuals affected by AI systems but powerless to affect the project—and their nexus to the project. The study contributes a novel method for identifying passive stakeholders and highlights the need to engage developers, operators, and representatives of passive stakeholders to achieve moral, ethical, and sustainable development.

Cognitive Science of PowerPoint Part II: The Power of Attention

The most common pedagogical visual tool used in engineering classrooms are slides, such as those generated by Google Slides, Keynote, Prezi or PowerPoint. Unfortunately, when viewed through the lens of current models describing human information processing, many slides are poorly designed. That is, they either contain too much, poorly organized, or distracting information. Given the complexity of engineering content, it is essential that slides be used to help the student focus on key elements to increase learning, rather than simply act as a data dump or cue card. The paper will first provide an overview of human attention processes and how these impact working memory and learning. Then, the paper will provide an overview of selected theories from cognitive psychology, including top-down vs. bottom-up processing, focused vs. divided attention, and salience models (e.g. perception, gaze, and motion). Finally, using an authentic example of an engineering classroom slide, this paper will demonstrate how the practical application of these cognitive theories of attention can increase focus on (and thus retention) of the relevant content. This paper aims to be a “why-to” as well as a “how-to” guide for improving visual aids, specifically slides, in the engineering classroom. Note, this paper builds on our previous paper that focused on the cognitive load with respect to slide design.