Extract Salient Objects Research Articles

Intelligent extraction of salient objects from fuzzy distorted images based on multi-scale convolution

ABSTRACT To solve the problem of how to obtain important information from fuzzy distorted images, an intelligent method for extracting salient objects from fuzzy distorted images based on multi-scale convolution is proposed. A multi-scale deformation feature convolution network is established to extract the salient features of fuzzy distorted images. Among them, the multi-scale convolution network VGG-16 extracts the shallow features of the fuzzy distortion image through the convolution layer operation, uses the multi-scale convolution kernel to extract the fuzzy distortion image in parallel, and introduces the self-attention mechanism to make the extracted features more relevant, Input the fused features into the proposed area extraction network, Through the target area detection network, the proposed target area of the fuzzy distortion image is classified and position regression is performed to obtain the final salient target of the fuzzy distortion image. The experimental results show that this method can accurately recognize the contour of salient objects, and extract salient objects from fuzzy distorted images; The extracted salient objects have high accuracy, low mean square error and similarity of 92.9% with the original image; For the salient target extraction in various scenarios, it shows high advantages.

Frequency-aware feature aggregation network with dual-task consistency for RGB-T salient object detection

RGB-Thermal salient object detection (SOD) aims to merge two spectral images to segment visually appealing objects. Current methods primarily extract salient object information in the pixel perspective. However, biological and psychological research indicates notable frequency sensitivity of the human visual system (HVS). The high-frequency (HF) and low-frequency (LF) information in images are processed by different neural channels, which has been overlooked in SOD. In this study, we argue that the objective of RGB-T SOD is not only to enhance feature representation in the pixel-aware but also to emulate human visual mechanisms. To our best knowledge, we explore RGB-T SOD from the frequency perspective for the first time. Specifically, we first present a frequency-aware multi-spectral feature aggregation module (FMFA) to exploit the separability and complementarity of frequency-aware features, generating and making full use of LF and HF cues. FMFA improves the feature representation of RGB-T from the frequency perspective and provides stronger frequency cues for boundary auxiliary tasks. Then, we develop an HF-guided signed distance map prediction module (HF-SDM) with dual-task consistency to effectively alleviate the coarse mask caused by blur boundary. HF-SDM employs the geometric relationship of objects which boosts the interaction between salient regions and boundaries. As a result, the model can gain sharper boundaries for salient objects. Finally, we propose a frequency-aware feature aggregation network (FFANet) incorporated with dual-task learning. Extensive experiments on RGB-T SOD datasets demonstrate that our proposed method outperforms other state-of-the-art methods. Ablation studies and visualizations further verify the effectiveness and interpretability of our method.

Salient detection via the fusion of background-based and multiscale frequency-domain features

Salient object detection is a fundamental problem in image processing and computer vision. Many saliency detection algorithms based on the background and frequency-domain are used to extract salient object clues. However, the former causes the real object to be submerged in the detected object areas, especially in complex or small object scenes. While the latter will lead to the loss of some object information when detecting large objects. To solve these problems and achieve better object detection results, we propose a fusion framework for salient object detection by fusing background and frequency-domain features. The background features of the image are extracted by an improved background model. This model represents the spatial layout of the image area with respect to the image boundaries. Meanwhile, we present a new frequency-domain processing method to obtain multiscale frequency-domain features and mark the saliency of the object at different scales. Within our framework, inspired by human visual attention, we use the idea of a self-attention mechanism to capture the intrinsic relation between background and multiscale frequency-domain features. In addition, this fusion framework provides a three-dimensional Gaussian convolution kernel, which expands two-dimensional local information to three dimensions for feature fusion, thus producing more accurate salient objects. Experiment results demonstrate that the proposed method consistently outperforms eleven state-of-the-art methods on five challenging and complicated datasets in terms of four evaluation metrics.

Infrared and visible image fusion via salient object extraction and low-light region enhancement

The infrared and visible image fusion aims to integrate complementary information from the input image to generate an information-rich fused image, which has been widely used to improve the performance of various surveillance systems and high-level vision tasks. In this paper, we propose a novel infrared and visible image fusion via salient object extraction and low-light region enhancement. The proposed method can accurately extract salient objects from the source image and preserve the visual background. For infrared images, the most widely distributed pixels are utilized as seed points to measure intensity saliency. Not only that, since the spatial distribution of objects in images also affects visual attention, we design a central deviation model to measure spatial distribution saliency. The infrared salient object can be extracted by combining the intensity and spatial distribution saliency. For visible images, different from existing fusion methods, the direction uniformity instead of the gradient magnitude is utilized to extract salient objects. Finally, because the fusion task is always performed under low-light conditions or complex environments, we reduce the low-light region to improve the visual quality of visible images and then use the enhanced visible image as the background to reconstruct the fused image. Experimental results demonstrate that the proposed fusion method outperforms nine state-of-the-art image fusion methods in a series of qualitative and quantitative evaluations, and also has excellent visual effects.

An Improved Saliency Detection Algorithm Based on Edge Boxes and Bayesian Model

Visual saliency detection aims to extract salient objects from the original image, making it less complicated to process the image. This paper combines an edge box algorithm with Bayesian theory to detect salient objects. The proposed saliency detection algorithm transforms the process of traditional detection method, and prioritizes the positioning of significant objects. Firstly, the Harris corners of the original image were calculated, and clustered by the improved clustering algorithm, yielding the number of salient objects in the image. Then, all possible positions of salient objects in the image were framed by the edge box algorithm, and the boxes were sorted in descending order of the score. According to the number N of clusters of the image corners, the N top-ranking boxes were selected to determine the salient regions. In this way, the position and number of salient objects were clarified. Based on the selected salient regions, the final saliency map was calculated by improved geodesic distance and Bayesian model. Experimental results show that our approach performed better than 11 existing algorithms in both simple and relatively complex scenes. In terms of objective performance, the accuracy and recall of our algorithm on MSRA10k, ECSSD, DUT-OMRON and SED2 datasets were higher than that of the other algorithms.

A Robust Movement Quantification Algorithm of Hyperactivity Detection for ADHD Children Based on 3D Depth Images.

Attention deficit hyperactivity disorder (ADHD) is one of the most common childhood mental disorders. Hyperactivity is a typical symptom of ADHD in children. Clinicians diagnose this symptom by evaluating the children's activities based on subjective rating scales and clinical experience. In this work, an objective system is proposed to quantify the movements of children with ADHD automatically. This system presents a new movement detection and quantification method based on depth images. A novel salient object extraction method is proposed to segment body regions. In movement detection, we explore a new local search algorithm to detect any potential motions of children based on three newly designed evaluation metrics. In the movement quantification, two parameters are investigated to quantify the participation degree and the displacements of each body part in the movements. This system is tested by a depth dataset of children with ADHD. The movement detection results of this dataset mainly range from 91.0% to 95.0%. The movement quantification results of children are consistent with the clinical observations. The public MSR Action 3D dataset is tested to validate the performance of this system.

Smart Glass System Using Deep Learning for the Blind and Visually Impaired

Individuals suffering from visual impairments and blindness encounter difficulties in moving independently and overcoming various problems in their routine lives. As a solution, artificial intelligence and computer vision approaches facilitate blind and visually impaired (BVI) people in fulfilling their primary activities without much dependency on other people. Smart glasses are a potential assistive technology for BVI people to aid in individual travel and provide social comfort and safety. However, practically, the BVI are unable move alone, particularly in dark scenes and at night. In this study we propose a smart glass system for BVI people, employing computer vision techniques and deep learning models, audio feedback, and tactile graphics to facilitate independent movement in a night-time environment. The system is divided into four models: a low-light image enhancement model, an object recognition and audio feedback model, a salient object detection model, and a text-to-speech and tactile graphics generation model. Thus, this system was developed to assist in the following manner: (1) enhancing the contrast of images under low-light conditions employing a two-branch exposure-fusion network; (2) guiding users with audio feedback using a transformer encoder–decoder object detection model that can recognize 133 categories of sound, such as people, animals, cars, etc., and (3) accessing visual information using salient object extraction, text recognition, and refreshable tactile display. We evaluated the performance of the system and achieved competitive performance on the challenging Low-Light and ExDark datasets.

Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks.

The use of RGB-D information for salient object detection (SOD) has been extensively explored in recent years. However, relatively few efforts have been put toward modeling SOD in real-world human activity scenes with RGB-D. In this article, we fill the gap by making the following contributions to RGB-D SOD: 1) we carefully collect a new Salient Person (SIP) data set that consists of ~1 K high-resolution images that cover diverse real-world scenes from various viewpoints, poses, occlusions, illuminations, and background s; 2) we conduct a large-scale (and, so far, the most comprehensive) benchmark comparing contemporary methods, which has long been missing in the field and can serve as a baseline for future research, and we systematically summarize 32 popular models and evaluate 18 parts of 32 models on seven data sets containing a total of about 97k images; and 3) we propose a simple general architecture, called deep depth-depurator network (D3Net). It consists of a depth depurator unit (DDU) and a three-stream feature learning module (FLM), which performs low-quality depth map filtering and cross-modal feature learning, respectively. These components form a nested structure and are elaborately designed to be learned jointly. D3Net exceeds the performance of any prior contenders across all five metrics under consideration, thus serving as a strong model to advance research in this field. We also demonstrate that D3Net can be used to efficiently extract salient object masks from real scenes, enabling effective background-changing application with a speed of 65 frames/s on a single GPU. All the saliency maps, our new SIP data set, the D3Net model, and the evaluation tools are publicly available at https://github.com/DengPingFan/D3NetBenchmark.

Active Contour Model Using Fast Fourier Transformation for Salient Object Detection

The active contour model is a comprehensive research technique used for salient object detection. Most active contour models of saliency detection are developed in the context of natural scenes, and their role with synthetic and medical images is not well investigated. Existing active contour models perform efficiently in many complexities but facing challenges on synthetic and medical images due to the limited time like, precise automatic fitted contour and expensive initialization computational cost. Our intention is detecting automatic boundary of the object without re-initialization which further in evolution drive to extract salient object. For this, we propose a simple novel derivative of a numerical solution scheme, using fast Fourier transformation (FFT) in active contour (Snake) differential equations that has two major enhancements, namely it completely avoids the approximation of expansive spatial derivatives finite differences, and the regularization scheme can be generally extended more. Second, FFT is significantly faster compared to the traditional solution in spatial domain. Finally, this model practiced Fourier-force function to fit curves naturally and extract salient objects from the background. Compared with the state-of-the-art methods, the proposed method achieves at least a 3% increase of accuracy on three diverse set of images. Moreover, it runs very fast, and the average running time of the proposed methods is about one twelfth of the baseline.

Automatic Salient Object Extraction Based on Locally Adaptive Thresholding to Generate Tactile Graphics

Automatic extraction of salient regions is beneficial for various computer vision applications, such as image segmentation and object recognition. The salient visual information across images is very useful and plays a significant role for the visually impaired in identifying tactile information. In this paper, we introduce a novel saliency cuts method using local adaptive thresholding to obtain four regions from a given saliency map. First, we produced four regions for image segmentation using a saliency map as an input image and local adaptive thresholding. Second, the four regions were used to initialize an iterative version of the GrabCuts algorithm and to produce a robust and high-quality binary mask with a full resolution. Finally, salient objects’ outer boundaries and inner edges were detected using the solution from our previous research. Experimental results showed that local adaptive thresholding using integral images can produce a more robust binary mask compared to the results from previous works that make use of global thresholding techniques for salient object segmentation. The proposed method can extract salient objects with a low-quality saliency map, achieving a promising performance compared to existing methods. The proposed method has advantages in extracting salient objects and generating simple, important edges from natural scene images efficiently for delivering visually salient information to the visually impaired.

Saliency From Growing Neural Gas: Learning Pre-Attentional Structures for a Flexible Attention System.

Artificial visual attention has been an active research area for over two decades. Especially, the concept of saliency has been implemented in many different ways. Early approaches aimed at closely modeling saliency processing with concepts from biological attention to provide (at least in the long run) general-purpose attention for technical systems. More recent approaches have departed from this agenda, turning to more specific attention-guided tasks, most notably the accurate extraction of salient objects, for which large-scale ground truth datasets make it possible to quantify progress. While the first type of models is troubled by weak performance in these specific tasks, the second type, as we show with a new benchmark, has lost the ability to predict saliency in the original sense, which may be an important factor for future general-purpose attention systems. Here, we describe a new approach using growing neural gas to obtain pre-attentional structures for a scene at an early processing stage. On this basis, traditional saliency concepts can be applied while at the same time they can be linked to mechanisms that make models successful in salient object detection. The model shows high performance at predicting traditional saliency and makes substantial progress toward salient object detection, although it cannot reach the top-level performance of some specialized methods. We discuss the important implications of our findings.

Saliency detection based on superpixel correlation and cosine window filtering

With the development of computer vision, image salient region detection plays an important role in the field of image processing. However, existing saliency detection methods can only generate saliency maps without achieving accuracy, thereby ignoring the integrity of the significant target. The present study proposed a saliency detection algorithm based on the correlation of superpixel and cosine window filtering (SPC) algorithm that introduced a variable weight superpixel segmentation method to keep the edge of information and enhance the integration of the salient target. Space distance and color distance are used to judge the correlation between two superpixels in the proposed algorithm, and based on the superpixel correlation, the saliency map is obtained by all superpixels’ voting. When the saliency map is corrected by a cosine window, we call this operation a cosine window filtering. Finally, we used the OSTU method for binarization and obtained the target position and size, using connected domain detection. The experiment results showed that the SPC algorithm can describe superpixels’ significance accurately and extract salient object. As compared with other algorithms, our algorithm offers higher accurate results, regarding the accurate position and size of salient targets.

Direction Selective Contour Detection for Salient Objects

The active contour model is a widely used technique for automatic object contour extraction. Existing methods based on this model can perform with high accuracy, even in the case of complex contours, but challenging issues remain, like the need for precise contour initialization for high curvature boundary segments or the handling of cluttered backgrounds. To deal with such issues, this paper presents a salient object extraction method, the first step of which is the introduction of an improved edge map that incorporates edge direction as a feature. The direction information in the small neighborhoods of image feature points is extracted, and the images’ prominent orientations are defined for direction-selective edge extraction. Using such improved edge information, we provide a highly accurate shape contour representation, which we also combine with texture features. The principle of the paper is to interpret an object as the fusion of its components: its extracted contour and its inner texture. Our goal in fusing textural and structural information is twofold: it is applied for automatic contour initialization, and it is also used to establish an improved external force field. This fusion then produces highly accurate salient object extractions. We performed extensive evaluations, which confirm that the presented object extraction method outperforms parametric active contour models and achieves higher efficiency than the majority of the evaluated automatic saliency methods.

Eye Fixation Assisted Video Saliency Detection via Total Variation-based Pairwise Interaction.

As human visual attention is naturally biased towards foreground objects in a scene, it can be used to extract salient objects in video clips. In this work, we proposed a weakly supervised learning based video saliency detection algorithm utilizing eye fixations information from multiple subjects. Our main idea is to extend eye fixations to saliency regions step by step. First, visual seeds are collected using multiple color space geodesic distance based seed region mapping with filtered and extended eye fixations. This operation helps raw fixation points spread to the most likely salient regions, namely, visual seed regions. Second, in order to seize the essential scene structure from video sequences, we introduce the total variance based pairwise interaction model to learn the potential pairwise relationship between foreground and background within a frame or across video frames. In this vein, visual seed regions eventually grow into salient regions. Compared with previous approaches the generated saliency maps has two most outstanding properties: integrity and purity, which are conductive to segment the foreground and significant to the follow-up tasks. Extensive quantitative and qualitative experiments on various video sequences demonstrate that the proposed method outperforms state-of-theart image and video saliency detection algorithms.

An effective fusion model for image retrieval

In the past decade, the popular Bag of Visual Words approach has been applied to many computer vision tasks, including image classification, video search, robot localization, and texture recognition. Unfortunately, most approaches use intensity features and discard color information, an important characteristic of any image that is motivated by human vision. Besides, if background colors are higher than foreground ones, Dominant Color Descriptor (DCD) retrieves images that contain similar background colors correctly. On the other hand, just color feature extraction is not sufficient for similar objects with different color descriptors (e.g. white dog vs. black dog). To solve these problems, a new Salient DCD (SDCD) color descriptor is proposed to extract foreground color and add semantic information into DCD based on the color distances and salient object extraction methods. Besides, a new fusion model is presented to fuse SDCD histogram and PHOW MSDSIFT histogram. Performance evaluation on several datasets proves that the new approach outperforms other existing, state-of-the-art methods.

Reprint of “Two-stage sparse coding of region covariance via Log-Euclidean kernels to detect saliency”

In this paper, we present a novel bottom-up saliency detection algorithm from the perspective of covariance matrices on a Riemannian manifold. Each superpixel is described by a region covariance matrix on Riemannian Manifolds. We carry out a two-stage sparse coding scheme via Log-Euclidean kernels to extract salient objects efficiently. In the first stage, given background dictionary on image borders, sparse coding of each region covariance via Log-Euclidean kernels is performed. The reconstruction error on the background dictionary is regarded as the initial saliency of each superpixel. In the second stage, an improvement of the initial result is achieved by calculating reconstruction errors of the superpixels on foreground dictionary, which is extracted from the first stage saliency map. The sparse coding in the second stage is similar to the first stage, but is able to effectively highlight the salient objects uniformly from the background. Finally, three post-processing methods–highlight-inhibition function, context-based saliency weighting, and the graph cut–are adopted to further refine the saliency map. Experiments on four public benchmark datasets show that the proposed algorithm outperforms the state-of-the-art methods in terms of precision, recall and mean absolute error, and demonstrate the robustness and efficiency of the proposed method.

Two-stage sparse coding of region covariance via Log-Euclidean kernels to detect saliency

In this paper, we present a novel bottom-up saliency detection algorithm from the perspective of covariance matrices on a Riemannian manifold. Each superpixel is described by a region covariance matrix on Riemannian Manifolds. We carry out a two-stage sparse coding scheme via Log-Euclidean kernels to extract salient objects efficiently. In the first stage, given background dictionary on image borders, sparse coding of each region covariance via Log-Euclidean kernels is performed. The reconstruction error on the background dictionary is regarded as the initial saliency of each superpixel. In the second stage, an improvement of the initial result is achieved by calculating reconstruction errors of the superpixels on foreground dictionary, which is extracted from the first stage saliency map. The sparse coding in the second stage is similar to the first stage, but is able to effectively highlight the salient objects uniformly from the background. Finally, three post-processing methods–highlight-inhibition function, context-based saliency weighting, and the graph cut–are adopted to further refine the saliency map. Experiments on four public benchmark datasets show that the proposed algorithm outperforms the state-of-the-art methods in terms of precision, recall and mean absolute error, and demonstrate the robustness and efficiency of the proposed method.

Extended quantum cuts for unsupervised salient object extraction

In this manuscript, an unsupervised salient object extraction algorithm is proposed for RGB and RGB-Depth images. Saliency estimation is formulated as a foreground detection problem. To this end, Q...

仮想超画素と大域的特徴による領域分割と背景と目立対象物の抽出

仮想超画素と大域的特徴による領域分割と背景と目立対象物の抽出

An infrared salient object stereo matching algorithm based on epipolar rectification

Due to the higher noise and less details in infrared images, general matching algorithms are prone to obtaining unsatisfying results. Combining the idea of salient object, we propose a novel infrared stereo matching algorithm which applies to unconstrained stereo rigs. Firstly, we present an epipolar rectification method introducing particle swarm optimization and K-nearest neighbor to deal with the problem of epipolar constraint. Then we make use of transition region to extract salient object in the rectified infrared image pairs. Finally, disparity map is generated by matching salient regions. Experiments show that our algorithm deals with the infrared stereo matching of unconstrained stereo rigs with better accuracy and higher speed.