Salient Pixels Research Articles

Leveraging CAM Algorithms for Explaining Medical Semantic Segmentation

Convolutional neural networks (CNNs) achieve prevailing results in segmentation tasks nowadays and represent the state-of-the-art for image-based analysis. However, the under- standing of the accurate decision-making process of a CNN is rather unknown. The research area of explainable artificial intelligence (xAI) primarily revolves around understanding and interpreting this black-box behavior. One way of interpreting a CNN is the use of class ac- tivation maps (CAMs) that represent heatmaps to indicate the importance of image areas for the prediction of the CNN. For classification tasks, a variety of CAM algorithms exist. But for segmentation tasks, only one CAM algorithm for the interpretation of the output of a CNN exist. We propose a transfer between existing classification- and segmentation- based methods for more detailed, explainable, and consistent results which show salient pixels in semantic segmentation tasks. The resulting Seg-HiRes-Grad CAM is an exten- sion of the segmentation-based Seg-Grad CAM with the transfer to the classification-based HiRes CAM. Our method improves the previously-mentioned existing segmentation-based method by adjusting it to recently published classification-based methods. Especially for medical image segmentation, this transfer solves existing explainability disadvantages. The code is available at <a href='https://github.com/TillmannRheude/SegHiResGrad_CAM'>https://github.com/TillmannRheude/SegHiResGrad_CAM</a>

Efficient fusion of spatio-temporal saliency for frame wise saliency identification

<p>Video saliency detection is a rapidly growing subject that has seen very few contributions. The most common technique used nowadays is to perform frame-by-frame saliency detection. The modified Spatio-temporal fusion method presented in this paper offers a novel approach to saliency detection and mapping. It uses frame-wise overall motion color saliency as well as pixel-based consistent Spatio-temporal diffusion for its temporal uniformity. Additionally, a variety of techniques is advocated as a way to increase the saliency maps' overall accuracy and precision. The video is divided into groups of frames, and each frame temporarily goes through diffusion and integration in order to compute the color saliency mapping, as covered in the proposed method section. Then, with the aid of a permutation matrix, the inter-group frame is used to format the pixel-based saliency fusion, after which the features, or the fusion of pixel saliency and color information, direct the diffusion of the spatiotemporal saliency. The result is tested using five publicly accessible global saliency evaluation metrics, and it is determined that the proposed algorithm outperforms numerous saliency detection techniques with an improvement in accuracy margin. The robustness, dependability, adaptability, and precision are all demonstrated by the results.</p>

A deep learning based relative clarity classification method for infrared and visible image fusion

Infrared and visible image fused(IVIF) results normally suffer from detail loss, noise occurrence, low contrast, and blurred edges. In this paper, a new method is proposed to address the detail loss, low contrast, and blurring issue in IVIF. Specifically, visible images are enhanced by guided filter and high dynamic range compression. Infrared images are normalized by a linear transformation. Then we use blur and clear discrimination to detect salient pixels between infrared and visible images. A fully weight shared multi-path residual neural network is proposed for blur discrimination between infrared and visible image pixels in the same position. Clear pixels are treated as salient pixels, which contribute more to fused images than blur pixels. The output of our proposed network is a binary classification map for blur and clear discrimination, which is treated as our fusion weight map in the fusion stage. To deal with the discontinuous problem, we compute the two distance-transformed maps of the binary classification map and its complementary map. The two distance-transformed maps are used as weight maps to fuse the enhanced infrared and visible images. Finally, we use single scale retinex (SSR) to further enhance our fused images. The experimental results in public IVIF datasets demonstrate the superior performance of our proposed approach over other state-of-the-art methods in terms of both subjective visual quality and objective metrics. The source code is available in https://github.com/eyob12/Multi_path_residual_neural_network_based_IVIF

Video saliency-recognition by applying custom spatio temporal fusion technique

<span lang="EN-US">Video saliency detection is a major growing field with quite few contributions to it. The general method available today is to conduct frame wise saliency detection and this leads to several complications, including an incoherent pixel-based saliency map, making it not so useful. This paper provides a novel solution to saliency detection and mapping with its custom spatio-temporal fusion method that uses frame wise overall motion colour saliency along with pixel-based consistent spatio-temporal diffusion for its temporal uniformity. In the proposed method section, it has been discussed how the video is fragmented into groups of frames and each frame undergoes diffusion and integration in a temporary fashion for the colour saliency mapping to be computed. Then the inter group frame are used to format the pixel-based saliency fusion, after which the features, that is, fusion of pixel saliency and colour information, guide the diffusion of the spatio temporal saliency. With this, the result has been tested with 5 publicly available global saliency evaluation metrics and it comes to conclusion that the proposed algorithm performs better than several state-of-the-art saliency detection methods with increase in accuracy with a good value margin. All the results display the robustness, reliability, versatility and accuracy.</span>

Context-aware Optimization for Bandwidth-Efficient Image Analytics Offloading

Convolutional Neural Networks (CNN) have given rise to numerous visual analytics applications at the edge of the Internet. The image is typically captured by cameras and then live-streamed to edge servers for analytics due to the prohibitive cost of running CNN on computation-constrained end devices. A critical component to ensure low-latency and accurate visual analytics offloading over low bandwidth networks is image compression which minimizes the amount of visual data to offload and maximizes the decoding quality of salient pixels for analytics. Despite the wide adoption, JPEG standards and traditional image compression techniques do not address the accuracy of analytics tasks, leading to ineffective compression for visual analytics offloading. Although recent machine-centric image compression techniques leverage sophisticated neural network models or hardware architecture to support the accuracy-bandwidth trade-off, they introduce excessive latency in the visual analytics offloading pipeline. This paper presents CICO, a Context-aware Image Compression Optimization framework to achieve low-bandwidth and low-latency visual analytics offloading. CICO contextualizes image compression for offloading by employing easily-computable low-level image features to understand the importance of different image regions for a visual analytics task. Accordingly, CICO can optimize the trade-off between compression size and analytics accuracy. Extensive real-world experiments demonstrate that CICO reduces the bandwidth consumption of existing compression methods by up to 40% under comparable analytics accuracy. Regarding the low-latency support, CICO achieves up to a 2x speedup over state-of-the-art compression techniques.

Multi-scale color constancy based on salient varying local spatial statistics

The human visual system unconsciously determines the color of the objects by “discounting” the effects of the illumination, whereas machine vision systems have difficulty performing this task. Color constancy algorithms assist computer vision pipelines by removing the effects of the illuminant, which in the end enables these pipelines to perform better on high-level vision tasks based on the color features of the scene. Due to its benefits, numerous color constancy algorithms have been developed, and existing techniques have been improved. Combining different strategies and investigating new methods might help us design simple yet effective algorithms. Thereupon, we present a color constancy algorithm based on the outcomes of our previous works. Our algorithm is built upon the biological findings that the human visual system might be discounting the illuminant based on the highest luminance patches and space-average color. We find the illuminant estimate based on the idea that if the world is gray on average, the deviation of the brightest pixels from the achromatic value should be caused by the illuminant. Our approach utilizes multi-scale operations by only considering the salient pixels. It relies on varying surface orientations by adopting a block-based approach. We show that our strategy outperforms learning-free algorithms and provides competitive results compared to the learning-based methods. Moreover, we demonstrate that using parts of our strategy can significantly improve the performance of several learning-free methods. We also briefly present an approach to transform our global color constancy method into a multi-illuminant color constancy approach.

Video saliency-detection using custom spatiotemporal fusion method

There have been several researches done in the field of image saliency but not as much as in video saliency. In order to increase precision and accuracy during compression, reduce coding complexity and time consumption along with memory allocation problems with our proposed solution. It is a modified high-definition video compression (HEVC) pixel based consistent spatiotemporal diffusion with temporal uniformity. It involves taking apart the video into groups of frames, computing colour saliency, integrate temporal fusion, pixel saliency fusion is conducted and then colour information guides the diffusion process for the spatiotemporal mapping with the help of permutation matrix. The proposed solution is tested on a publicly available extensive dataset with five global saliency valuation metrics and is compared with several other state-of-the-art saliency detection methods. The results display and overall best performance amongst all other candidates.

OISE: Optimized Input Sampling Explanation with a Saliency Map Based on the Black-Box Model

With the development of artificial intelligence technology, machine learning models are becoming more complex and accurate. However, the explainability of the models is decreasing, and much of the decision process is still unclear and difficult to explain to users. Therefore, we now often use Explainable Artificial Intelligence (XAI) techniques to make models transparent and explainable. For an image, the ability to recognize its content is one of the major contributions of XAI techniques to image recognition. Visual methods for describing classification decisions within an image are usually expressed in terms of salience to indicate the importance of each pixel. In some approaches, explainability is achieved by deforming and integrating white-box models, which limits the use of specific network architectures. Therefore, in contrast to white-box model-based approaches that use weights or other internal network states to estimate pixel saliency, we propose the Optimized Input Sampling Explanation (OISE) technique based on black-box models. OISE uses masks to generate saliency maps that reflect the importance of each pixel to the model predictions, and employs black-box models to empirically infer the importance of each pixel. We evaluate our method using deleted/inserted pixels, and extensive experiments on several basic datasets show that OISE achieves better visual performance and fairness in explaining the decision process compared to the performance of other methods. This approach makes the decision process clearly visible, makes the model transparent and explainable, and serves to explain it to users.

SAGE: Steering the Adversarial Generation of Examples With Accelerations

To generate image adversarial examples, state-of-the-art black-box attacks usually require thousands of queries. However, massive queries will introduce additional costs and exposure risks in the real world. Towards improving the attack efficiency, we carefully design an acceleration framework SAGE for existing black-box methods, which is composed of sLocator (initial point optimization) and sRudder (search process optimization). The core idea of SAGE in terms of 1) saliency map can guide the perturbations towards the most adversarial direction and 2) exploiting bounding box (bbox) to capture those salient pixels in the black-box attack. Meanwhile, we provide a series of observations and experiments that demonstrate bbox holds model invariance and process invariance. We extensively evaluate SAGE on four state-of-the-art black-box attacks involving three popular datasets (MNIST, CIFAR10, and ImageNet). The results show that SAGE could present fundamental improvements even against robust models that use adversarial training. Specifically, SAGE could reduce >20% of queries and improve the success rate of attacks to 95%~100%. Compared with the other acceleration framework, SAGE fulfills the more significant effect in a flexible, stable, and low-overhead manner. Moreover, our practical evaluation (Google Cloud Vision API) shows SAGE can be applied to real-world scenarios.

Ship detection in SAR images by saliency analysis of multiscale superpixels

ABSTRACT This paper proposes a new ship detection algorithm via saliency analysis on multiscale superpixels. First, Synthetic Aperture Radar (SAR) imagery is over-segmented into many homogenous regions, called superpixels. In contrast to prior work, these superpixels are generated with various scales to handle size variation of ships in the SAR imagery. At each scale, local contrast and ship geometrical characteristics are considered to enhance the saliency of ship pixels. Then, the final saliency map is computed by integrating saliency values of all scales. Experimental results on two real SAR images demonstrate the efficiency of the proposed algorithm.

Graph ranking based butterfly segmentation in ecological images

An accurate cultural insect detection and recognition relies mainly on a proper automatic segmentation. This paper deals with butterfly segmentation in ecological images characterized by several artifacts like the complexity of environmental decors and cluttered backgrounds. The distractors contained in the rich ecological environment and the huge difference between butterfly species complicate severely the segmentation and make it a challenging task. As butterflies appears to be well contrasted from their surrounding, we suggest to explore the saliency property to delineate accurately the butterfly boundaries. In this vein, we perform a graph ranking process with high level guidance according to foreground and background cues to improve the quality of segmentation. The ranking accuracy is improved through a weighting scheme that combines accurately color, texture and spatial information. The contribution of each used feature is controlled according to its relevance in highlighting butterfly regions. After that, we initialize foreground seeds from most salient pixels and background seeds from less salient pixels as an input for a Graph-cut algorithm to extract the butterfly from the background. Comparative evaluation has shown that our segmentation scheme outperforms some existing segmentation methods that provide high segmentation scores.

GSM-HM: Generation of Saliency Maps for Black-Box Object Detection Model Based on Hierarchical Masking

Interpretability of DNN-based object detection has been a rising concern for the research community. The first step towards this goal is a saliency map that visualizes the importance (saliency) of pixels in an image for the object detected by a specific model. Black-box based methods generate a saliency map without the need to look into the internals of a model, thus applicable to all models without the need of adaptation. In addition, they provide more reliable evaluation on the saliency of pixels than white-box methods by means of the absence of these pixels from the image. However, with current black-box methods, the absence of pixels is produced by random image masks. Despite the need of a great number of random masks for sufficient coverage, the quality of the pixel saliency is not assured to be satisfactory. In this work, we propose a more effective black-box framework with hierarchical masking. In this framework, called GSM-HM, pixel saliency is evaluated at multiple levels, with each lower level performing a refinement on the saliency information of the upper level. This hierarchical framework significantly reduces the masking efforts on less valuable pixels, thus it can produce saliency maps with higher qualities. In our experiments, the quality of a generated saliency map is evaluated with four different metrics: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">deletion</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">insertion</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">convergence</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RAM</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">the ratio of average to maximum</i> ). Compared with D-RISE, a recent black-block method, GSM-HM generates more accurate saliency maps evaluated by these metrics.

Explainable nucleus classification using Decision Tree Approximation of Learned Embeddings.

Nucleus detection, segmentation and classification are fundamental to high-resolution mapping of the tumor microenvironment using whole-slide histopathology images. The growing interest in leveraging the power of deep learning to achieve state-of-the-art performance often comes at the cost of explainability, yet there is general consensus that explainability is critical for trustworthiness and widespread clinical adoption. Unfortunately, current explainability paradigms that rely on pixel saliency heatmaps or superpixel importance scores are not well-suited for nucleus classification. Techniques like Grad-CAM or LIME provide explanations that are indirect, qualitative and/or nonintuitive to pathologists. In this article, we present techniques to enable scalable nuclear detection, segmentation and explainable classification. First, we show how modifications to the widely used Mask R-CNN architecture, including decoupling the detection and classification tasks, improves accuracy and enables learning from hybrid annotation datasets like NuCLS, which contain mixtures of bounding boxes and segmentation boundaries. Second, we introduce an explainability method called Decision Tree Approximation of Learned Embeddings (DTALE), which provides explanations for classification model behavior globally, as well as for individual nuclear predictions. DTALE explanations are simple, quantitative, and can flexibly use any measurable morphological features that make sense to practicing pathologists, without sacrificing model accuracy. Together, these techniques present a step toward realizing the promise of computational pathology in computer-aided diagnosis and discovery of morphologic biomarkers. Relevant code can be found at github.com/CancerDataScience/NuCLS. Supplementary data are available at Bioinformatics online.

Ranking-Based Salient Object Detection and Depth Prediction for Shallow Depth-of-Field.

Shallow depth-of-field (DoF), focusing on the region of interest by blurring out the rest of the image, is challenging in computer vision and computational photography. It can be achieved either by adjusting the parameters (e.g., aperture and focal length) of a single-lens reflex camera or computational techniques. In this paper, we investigate the latter one, i.e., explore a computational method to render shallow DoF. The previous methods either rely on portrait segmentation or stereo sensing, which can only be applied to portrait photos and require stereo inputs. To address these issues, we study the problem of rendering shallow DoF from an arbitrary image. In particular, we propose a method that consists of a salient object detection (SOD) module, a monocular depth prediction (MDP) module, and a DoF rendering module. The SOD module determines the focal plane, while the MDP module controls the blur degree. Specifically, we introduce a label-guided ranking loss for both salient object detection and depth prediction. For salient object detection, the label-guided ranking loss comprises two terms: (i) heterogeneous ranking loss that encourages the sampled salient pixels to be different from background pixels; (ii) homogeneous ranking loss penalizes the inconsistency of salient pixels or background pixels. For depth prediction, the label-guided ranking loss mainly relies on multilevel structural information, i.e., from low-level edge maps to high-level object instance masks. In addition, we introduce a SOD and depth-aware blur rendering method to generate shallow DoF images. Comprehensive experiments demonstrate the effectiveness of our proposed method.

Saliency Detection Using Deep Features and Affinity-Based Robust Background Subtraction

Most existing saliency methods measure fore- ground saliency by using the contrast of a foreground region to its local context, or boundary priors and spatial compactness. These methods are not powerful enough to extract a precise salient region from noisy and cluttered backgrounds. To evaluate the contrast of salient and background regions effectively, we consider high-level features from both supervised and unsupervised methods. We propose an affinity-based robust background subtraction technique and maximum attention map using a pre-trained convolution neural network. This affinity-based technique uses pixel similarities to propagate the values of salient pixels among foreground and background regions and their union. The salient pixel value controls the foreground and background information by using multiple pixel affinities. The maximum attention map is derived from the convolution neural network using features of the Pooling and Relu layers. This method can detect salient regions from images that have noisy and cluttered backgrounds. Our experimental results demonstrate the effectiveness of the proposed approach on six different saliency data sets and benchmarks and show that it improves the quality of detection beyond current saliency detection methods.

IHS-GTF: A Fusion Method for Optical and Synthetic Aperture Radar Data

Optical and Synthetic Aperture Radar (SAR) fusion is addressed in this paper. Intensity–Hue–Saturation (IHS) is an easily implemented fusion method and can separate Red–Green–Blue (RGB) images into three independent components; however, using this method directly for optical and SAR images fusion will cause spectral distortion. The Gradient Transfer Fusion (GTF) algorithm is proposed firstly for infrared and gray visible images fusion, which formulates image fusion as an optimization problem and keeps the radiation information and spatial details simultaneously. However, the algorithm assumes that the spatial details only come from one of the source images, which is inconsistent with the actual situation of optical and SAR images fusion. In this paper, a fusion algorithm named IHS-GTF for optical and SAR images is proposed, which combines the advantages of IHS and GTF and considers the spatial details from the both images based on pixel saliency. The proposed method was assessed by visual analysis and ten indices and was further tested by extracting impervious surface (IS) from the fused image with random forest classifier. The results show the good preservation of spatial details and spectral information by our proposed method, and the overall accuracy of IS extraction is 2% higher than that of using optical image alone. The results demonstrate the ability of the proposed method for fusing optical and SAR data effectively to generate useful data.

Multi-Modal Image Fusion via Convolutional Morphological Component Analysis and Guided Filter

In feature-level image fusion, deep learning technology, particularly convolutional sparse representation (SR) theory, has emerged as a new topic over the past three years. This paper proposes an effective image fusion method based on convolution SR, namely, convolutional sparsity-based morphological component analysis and guided filter (CS-MCA-GF). The guided filter operator and choose-max coefficient fusion scheme introduced in this method can effectively eliminate the artifacts generated by the morphological components in the linear fusion, and maintain the pixel saliency of the source images. Experiments show that the proposed method can achieve an excellent performance in multi-modal image fusion, which includes medical image fusion.

Saliency-dependent adaptive remeshing for cloth simulation

We propose a method for simulating cloth with meshes dynamically refined according to visual saliency. It is a common belief that it is preferable for the regions of an image being viewed to have more details than others. For a certain scene, a low-resolution cloth mesh is first simulated and rendered into images in the preview stage. Pixel saliency values of these images are predicted according to a pre-trained saliency prediction model. These pixel saliencies are then translated to a vertex saliency of the corresponding meshes. Vertex saliency, together with camera positions and a number of geometric features of surfaces, guides the dynamic remeshing for simulation in the production stage. To build the saliency prediction model, images extracted from various videos of clothing scenes were used as training data. Participants were asked to watch these videos and their eye motion was tracked. A saliency map is generated from the eye motion data for each extracted video frame image. Image feature vectors and map labels are sent to a Support Vector Machine for training to obtain a saliency prediction model. Our method greatly reduces the number of vertices and faces in the clothing model, and generates a speed-up of more than 3 × for scenes with single dressed character, while for multi-character scenes the speed-up is increased to more than 5×. The proposed technique can work together with view-dependency for offline simulation.

Anomaly Detection Using Local Kernel Density Estimation and Context-Based Regression

Current local density-based anomaly detection methods are limited in that the local density estimation and the neighborhood density estimation are not accurate enough for complex and large databases, and the detection performance depends on the size parameter of the neighborhood. In this paper, we propose a new kernel function to estimate samples’ local densities and propose a weighted neighborhood density estimation to increase the robustness to changes in the neighborhood size. We further propose a local kernel regression estimator and a hierarchical strategy for combining information from the multiple scale neighborhoods to refine anomaly factors of samples. We apply our general anomaly detection method to image saliency detection by regarding salient pixels in objects as anomalies to the background regions. Local density estimation in the visual feature space and kernel-based saliency score propagation in the image enable the assignment of similar saliency values to homogenous object regions. Experimental results on several benchmark datasets demonstrate that our anomaly detection methods overall outperform several state-of-art anomaly detection methods. The effectiveness of our image saliency detection method is validated by comparison with several state-of-art saliency detection methods.

Least Square Saliency Transformation of Capsule Endoscopy Images for PDF Model Based Multiple Gastrointestinal Disease Classification

Wireless capsule endoscopy (WCE) is an effective video technology to diagnose gastrointestinal (GI) diseases, such as bleeding, ulcer, and tumor. In order to avoid a tedious manual review process of long duration WCE video recordings, automatic disease detection schemes have received significant attention from the researchers. In particular, instead of the conventional approach of dealing with a single disease, developing a unified scheme, which is capable of detecting multiple GI diseases, is getting more importance while being challenging. In this paper, a unified computer-aided scheme is developed for detecting multiple GI diseases from WCE videos based on a proposed least-square saliency transformation (LSST) followed by a probabilistic model-fitting approach. Commonly in the training phase, image-level labeling of images is used, as pixel-level annotations are available only for a small number of images. In view of utilizing the knowledge from pixel-level annotated diseased images, an LSST scheme is proposed to extract a set of optimum prior coefficient-vectors which is later used to capture the salient pixels of interest (POI) in a larger WCE image dataset that do not have pixel-annotations. The intensity distributions of salient POI are modeled by a suitable probability density function (PDF) and the fitted PDF parameters are utilized as features in the proposed supervised hierarchical classification scheme. A large number of WCE images obtained from publicly available WCE videos are used for performance evaluation and it is found that the result obtained by the proposed method outperforms the results obtained in case of some state-of-the-art methods.