Salient Region Extraction Research Articles

Multiscale Feature Aggregation Network for Salient Object Detection in Optical Remote Sensing Images

Optical remote sensing images (ORSI) have various applications in different fields, and salient target detection (ORSI-SOD) of ORSI has become an important research topic in recent years. However, ORSI-SOD is a challenging problem due to the variable and complex backgrounds, large differences in levels, mixed backgrounds, and diverse topological shapes of ORSI. In this paper, we propose a novel model called multi-scale feature refinement aggregation network (MFANet), which consists of a multi-scale feature refinement module (MFR) and a context aggregation module (CFA). The MFR module extracts semantic information of ORSI across different dimensions in the multi-scale feature extraction stage. In the feature refinement stage, we use the proposed self-refinement module under the guidance of attention and reverse attention to progressively refine the prediction results. The CFA module introduces the hybrid attention module to gradually aggregate and extract salient regions from the context extraction module. To adapt to dense scenes, we develop a hybrid loss function that enables network optimization of multi-scale objectives in a self-adaptive manner. Our method outperforms most state-of-the-art salient object detection methods proposed in recent years in terms of accuracy.

MNC-Net: Multi-task graph structure learning based on node clustering for early Parkinson’s disease diagnosis

Purpose:The identification of early-stage Parkinson’s disease (PD) is important for the effective management of patients, affecting their treatment and prognosis. Recently, structural brain networks (SBNs) have been used to diagnose PD. However, how to mine abnormal patterns from high-dimensional SBNs has been a challenge due to the complex topology of the brain. Meanwhile, the existing prediction mechanisms of deep learning models are often complicated, and it is difficult to extract effective interpretations. In addition, most works only focus on the classification of imaging and ignore clinical scores in practical applications, which limits the ability of the model. Inspired by the regional modularity of SBNs, we adopted graph learning from the perspective of node clustering to construct an interpretable framework for PD classification. Methods:In this study, a multi-task graph structure learning framework based on node clustering (MNC-Net) is proposed for the early diagnosis of PD. Specifically, we modeled complex SBNs into modular graphs that facilitated the representation learning of abnormal patterns. Traditional graph neural networks are optimized through graph structure learning based on node clustering, which identifies potentially abnormal brain regions and reduces the impact of irrelevant noise. Furthermore, we employed a regression task to link clinical scores to disease classification, and incorporated latent domain information into model training through multi-task learning. Results:We validated the proposed approach on the Parkinsons Progression Markers Initiative dataset. Experimental results showed that our MNC-Net effectively separated the early-stage PD from healthy controls(HC) with an accuracy of 95.5%. The t-SNE figures have showed that our graph structure learning method can capture more efficient and discriminatory features. Furthermore, node clustering parameters were used as important weights to extract salient task-related brain regions(ROIs). These ROIs are involved in the development of mood disorders, tremors, imbalances and other symptoms, highlighting the importance of memory, language and mild motor function in early PD. In addition, statistical results from clinical scores confirmed that our model could capture abnormal connectivity that was significantly different between PD and HC. These results are consistent with previous studies, demonstrating the interpretability of our methods.

A multi-band image synchronous fusion method based on saliency

A new saliency-based multi-band image synchronous fusion method is proposed. First, combined window filtering is used to decompose the image into base and detail layers to better protect the edges of the image. Second, gamma correction is used for infrared image salient region extraction. To make it more applicable to the base layer fusion, we propose a method to calculate the parameters adaptively according to the image content. Consequently, the fusion result can better retain the thermal radiation information of the target. We then consider both the image quality and the amount of information contained in the image and construct an optimization model for synchronous fusion and noise reduction of detail layers to reduce the influence of noisy images on the fusion results. Finally, the fused base and detail layers are combined to obtain the final image. Our proposed method is evaluated and compared with representative methods, both qualitatively and quantitatively, applied to the TNO dataset. The results demonstrate that the proposed method aligns with human visual observation in highlighting salient targets and retaining valid detail information. The quality metrics are better than the comparison methods on the whole.

Infrared Sensation-Based Salient Targets Enhancement Methods in Low-Visibility Scenes.

Thermal imaging is an important technology in low-visibility environments, and due to the blurred edges and low contrast of infrared images, enhancement processing is of vital importance. However, to some extent, the existing enhancement algorithms based on pixel-level information ignore the salient feature of targets, the temperature which effectively separates the targets by their color. Therefore, based on the temperature and pixel features of infrared images, first, a threshold denoising model based on wavelet transformation with bilateral filtering (WTBF) was proposed. Second, our group proposed a salient components enhancement method based on a multi-scale retinex algorithm combined with frequency-tuned salient region extraction (MSRFT). Third, the image contrast and noise distribution were improved by using salient features of orientation, color, and illuminance of night or snow targets. Finally, the accuracy of the bounding box of enhanced images was tested by the pre-trained and improved object detector. The results show that the improved method can reach an accuracy of 90% of snow targets, and the average precision of car and people categories improved in four low-visibility scenes, which demonstrates the high accuracy and adaptability of the proposed methods of great significance for target detection, trajectory tracking, and danger warning of automobile driving.

Learning multi-view visual correspondences with self-supervision

Stereo-based 3D reconstruction requires to match features across images captured from slightly different viewing angles to recover 3D coordinates of the image pixels. Despite the workload of collecting data, annotating matched pixels requires also heavy labor. As recent researches for self-supervised representation learning has gained great progress, learning multi-view visual correspondences from large scale raw videos serves as an alternative. However, existing methods which benefit from contrastive learning tend to neglect false negative samples when matching between adjacent frames in a video, leading to sub-optimal optimization for visual features. In this paper, we propose a contrastive learning framework that construct self-supervision by semi-global visual correspondence to alleviate learning degradation when false negatives are involved in training. Our learning framework consists of pixel-level contrastive learning via patch reconstruction and patch-level contrastive learning cross videos. We also introduce saliency guidance to extract salient regions from video frames to further reduce potential false negatives. By optimizing the model with the proposed semi-global contrastive learning method, learned representations are forced to be discriminative and robust. Experiments demonstrate that our proposed method outperforms previous self-supervised methods on video object segmentation tasks. Moreover, when compared to fully-supervised algorithms designed for specific tasks, our proposed method also achieves competitive results.

Video Quality Assessment of Danmaku-Based Video Saliency Regions

At present, objective quality assessment methods have significant advantages in video quality assessment (VQA), among which the objective assessment based on deep learning is superior in processing speed and accuracy. However, the objective assessment method has the limitation of poor interpretability of experimental results and lacks the participation of human thinking and experience. In this paper, we present a quality assessment method based on the danmaku visual saliency region. First, to obtain the sentiment trend of the danmaku, we conduct a sentiment analysis of the content of the danmaku. Then, to speed up the video quality assessment, we select the keyframes of the video according to the sentiment analysis. Finally, we innovatively use human visual characteristics to extract salient regions from video keyframes to make the results more subjective. The experiments show that the Pearson correlation coefficient (PCC) and root mean square error (RMSE) between the results of the proposed method and the subjective evaluation are improved, which conforms to the audience's subjective feelings about video viewing, and the assessment results are more accurate.

A Generative Adversarial Network with Dual Discriminators for Infrared and Visible Image Fusion Based on Saliency Detection

Infrared and visible image fusion needs to preserve both the salient target of the infrared image and the texture details of the visible image. Therefore, an infrared and visible image fusion method based on saliency detection is proposed. Firstly, the saliency map of the infrared image is obtained by saliency detection. Then, the specific loss function and network architecture are designed based on the saliency map to improve the performance of the fusion algorithm. Specifically, the saliency map is normalized to [0, 1], used as a weight map to constrain the loss function. At the same time, the saliency map is binarized to extract salient regions and nonsalient regions. And, a generative adversarial network with dual discriminators is obtained. The two discriminators are used to distinguish the salient regions and the nonsalient regions, respectively, to promote the generator to generate better fusion results. The experimental results show that the fusion results of our method are better than those of the existing methods in both subjective and objective aspects.

Intelligent co-detection of cyclists and motorcyclists based on an improved deep learning method

Vulnerable road user (VRU) collision avoidance is a challenge task in developing an advanced driving assistant system (ADAS). This is because motorcyclists are the most VRUs, while most existing ADAS systems are not capable of distinguishing motorcyclists from cyclists. In order to address this problem, this study proposes a new detection method based on a center and scale prediction-proposal region (CSP-PR) improved deep learning model. This new method attempts to identify the individual difference and similar appearance between cyclists and motorcyclists from different perspectives; by doing so, the cyclists and motorcyclists can be co-detected from each other to enable effective VRU collision avoidance. For this purpose, a shared salient region extraction method is first introduced based on the CSP-PR; then multiple-instance target candidate regions are generated based on the redundancy strategy; last, a multi-task learning method is proposed to train an ResNet50-deep neural network to reduce the risk of overfitting and improve the co-detection rate of the cyclists and motorcyclists. In order to verify the effectiveness of the proposed CSP-PR-ResNet50 method, a local motorcyclist database and the Tsinghua-Daimler cyclist database were combined to form an experimental dataset. The experimental results demonstrate that the proposed method is able to effectively identify the cyclists and the motorcyclists. Compared with popular algorithms, including the faster region-based convolutional neural networks, YOLOv3 (You Only Look Once v3) and single shot multibox detector, the average precision (AP) of the cyclist detection produced by the proposed method is improved by 3.69%, 17.74% and 31.22%, respectively; and the AP of the motorcyclists detection is improved by 4.29%, 15.5% and 32.57%, respectively.

Saliency Cuts: Salient Region Extraction based on Local Adaptive Thresholding for Image Information Recognition of the Visually Impaired

In recent years, there has been an increased scope for assistive software and technologies, which help the visually impaired to perceive and recognize natural scene images. In this article, we propose a novel saliency cuts approach using local adaptive thresholding to obtain four regions from a given saliency map. The saliency cuts approach is an effective tool for salient object detection. First, we produce four regions for image segmentation using a saliency map as an input image and applying an automatic threshold operation. Second, the four regions are used to initialize an iterative version of the Grab Cut algorithm and to produce a robust and high-quality binary mask with a full resolution. Lastly, based on the binary mask and extracted salient object, outer boundaries and internal edges are detected by Canny edge detection method. Extensive experiments demonstrate that the proposed method correctly detects and extracts the main contents of the image sequences for delivering visually salient information to the visually impaired people compared to the results of existing salient object segmentation algorithms

Automatic Salient Object Extraction Based on Locally Adaptive Thresholding to Generate Tactile Graphics

Automatic extraction of salient regions is beneficial for various computer vision applications, such as image segmentation and object recognition. The salient visual information across images is very useful and plays a significant role for the visually impaired in identifying tactile information. In this paper, we introduce a novel saliency cuts method using local adaptive thresholding to obtain four regions from a given saliency map. First, we produced four regions for image segmentation using a saliency map as an input image and local adaptive thresholding. Second, the four regions were used to initialize an iterative version of the GrabCuts algorithm and to produce a robust and high-quality binary mask with a full resolution. Finally, salient objects’ outer boundaries and inner edges were detected using the solution from our previous research. Experimental results showed that local adaptive thresholding using integral images can produce a more robust binary mask compared to the results from previous works that make use of global thresholding techniques for salient object segmentation. The proposed method can extract salient objects with a low-quality saliency map, achieving a promising performance compared to existing methods. The proposed method has advantages in extracting salient objects and generating simple, important edges from natural scene images efficiently for delivering visually salient information to the visually impaired.

Infrared small target detection based on non-subsampled shearlet transform and phase spectrum of quaternion Fourier transform

Infrared small target detection is a crucial part of infrared search and track system, and it has been a significant research topic in the past decades. Inspired by previous studies showing that phase spectrum of quaternion Fourier transform (PQFT) great superiority in salient region extraction and the desirable characteristics of multi-scale, multi-direction and shift-invariant with non-subsampled shearlet transform (NSST), a new target detection method is proposed based on NSST and PQFT in this paper. The original image is first subjected to NSST decomposition to obtain a low frequency component and four high frequency components by NSP. Next, directional localization is achieved by shearing filters which provides multi-directional decomposition. Then, four direction high frequency sub-images decomposed by NSST are introduced as four data channels of PQFT. The reconstruction map that highlights the salient region in the time domain is computed using the inverse PQFT. Lastly, the real target is directly segmented by an adaptive threshold. The proposed method is validated by five test sequences. The experimental results show that our method is superior to other traditional methods in terms of robustness and effectiveness in complex background.

RETRACTED ARTICLE: Kernalized average entropy and density based spatial clustering with noise

Image clustering is one of the key technologies for image processing. Most image clustering methods based on density algorithms encountered with challenges including robust salient region extraction, cluster centre identification and noise. To address these issues, different density-based clustering methods were used. To accommodate this constraint, this paper proposes a method named Kernalized Average Entropy and Density-based Spatial Clustering (KAE-DSC). The KAE-DSC method efficiently extracts the robust salient region for texture and image database using Spatial Candidate Region Detection. Finally, Spatial Density-based Clustering with Noise is used to differentiate between core, non-core and noisy region for grouping similar region. By identifying the three different regions, efficient prediction is said to be made for the given texture and medical images. KAE-DSC method is compared with several other methods based on several evaluation indexes to testify that the proposed KAE-DSC is outperformed. Abundant experimental results proved that KAE-DSC is robust to parameters, which can automatically determine the number of clustering and improve the accuracy of clustering with minimum time complexity.

An extensive evaluation of deep featuresof convolutional neural networks for saliency prediction of human visual attention

Based on transfer learning, feature maps of deep convolutional neural networks (DCNNs) have been used to predict human visual attention. In this paper, we conduct extensive comparisons to investigate effects of feature maps on the predictions of the human visual attention using a deep features based saliency model framework. The feature maps of seven pretrained DCNNs are investigated using classical and class activation maps approaches. The performances of various saliency implementations are evaluated over four datasets using three metrics. The results demonstrate that deep feature maps of the pretrained DCNNs can be used to create saliency maps for the prediction of human visual attention. The incorporation of multiple levels of blurred and multi-scale feature maps improves the extraction of salient regions. Moreover, DCNNs pretrained using the Places dataset provide more localized objects that can be beneficial to the top-down saliency maps.

High Efficiency Video Coding Compliant Perceptual Video Coding Using Entropy Based Visual Saliency Model

In past years, several visual saliency algorithms have been proposed to extract salient regions from multimedia content in view of practical applications. Entropy is one of the important measures to extract salient regions, as these regions have high randomness and attract more visual attention. In the context of perceptual video coding (PVC), computational visual saliency models that utilize the charactertistics of the human visual system to improve the compression ratio are of paramount importance. To date, only a few PVC schemes have been reported that use the visual saliency model. In this paper, we conduct the first attempt to utilize entropy based visual saliency models within the high efficiency video coding (HEVC) framework. The visual saliency map generated for each input video frame is optimally thresholded to generate a binary saliency mask. The proposed HEVC compliant PVC scheme adjusts the quantization parameter according to visual saliency relevance at the coding tree unit (CTU) level. Efficient CTU level rate control is achieved by allocating bits to salient and non-salient CTUs by adjusting the quantization parameter values according to their perceptual weighted map. The attention based on information maximization has shown the best performance on newly created ground truth dataset, which is then incorporated in a HEVC framework. An average bitrate reduction of is achieved by the proposed HEVC compliant PVC scheme with the same perceptual quality and a nominal increase in coding complexity of when compared with HEVC reference software. Moreover, the proposed PVC scheme performs better than other HEVC based PVC schemes when encoded at low data rates.

Saliency-based visualization of hyperspectral satellite images using hierarchical fusion

Owing to a large number of spectral bands, it is always a challenge to devise an optimal visualization method for hyperspectral images. An algorithm must maintain a balance between dimensionality reduction and restoration of maximum spectral information. A methodology for visualization of hyperspectral imagery is proposed based on extraction of salient regions. For that, spectral bands are selected from different combinations of principal component analysis, minimum noise fraction, and saliency maps. A hierarchical fusion method is proposed, which is applied on the selected bands to obtain a final three band RGB image. The qualitative and quantitative results of the proposed method are very encouraging once compared with other state-of-the-art methods.

Salient Region Extraction Algorithm for Multi-target

Salient Region Extraction Algorithm for Multi-target

Vulnerable Plaque Recognition Based on Attention Model with Deep Convolutional Neural Network.

Previous studies have proved that the vulnerable plaque is a major factor leading to the onset of acute coronary syndrome (ACS). Recognizing vulnerable plaques is essential for cardiologists to treat illnesses, early. However, this task often comes with the challenge of insufficient annotated data sets and subtle differences between lesion regions and normal regions. In this paper, we apply the visual attention model with deep neural network to improve the performance of recognizing vulnerable plaques. There are two key ideas about our method: 1) using a top-down attention model to extract salient regions (blood vessels) according to the doctor's prior knowledge, and 2) employing a multi-task neural network to complete the recognition task. The first branch, a typical classification task, is to distinguish whether the image contains vulnerable plaques. The other branch uses a column-wise segmentation to locate vulnerable plaques. We have verified the effectiveness of our proposed method on the data set provided by 2017 CCCV-IVOCT Challenge. The proposed method obtains good performance.

Salient Region Extraction based on Global Contrast Enhancement and Saliency Cut for Image Information Recognition of the Visually Impaired

Salient Region Extraction based on Global Contrast Enhancement and Saliency Cut for Image Information Recognition of the Visually Impaired

Approach toward extraction of sparse texture features and their application in robust two-level bare-hand detection

A practical two-level robust hand detection system is developed using the proposed sparse texture features and color–texture features. Traditional dense features such as a histogram of oriented gradients, Gabor, segmentation-based fractal texture analysis, and histogram features are dense in general, with high time-complexity, limiting their use in practical hand detection systems. However, if only the prominent edge or texture parameters of an image are processed, the time complexity of the system can be significantly reduced, along with an increase in performance. Performance of any practical system is most affected by the presence of spurious objects in the background. Proposed approaches use four efficient filtering techniques to extract salient regions of an image retaining significant object-related information, followed by extraction of texture features, as mentioned above. Therefore, in the first stage, 10 sparse variants of these existing texture features are extracted and assessed using five classification models, namely Naive Bayes, Real AdaBoost, Gentle AdaBoost, Modest AdaBoost, and SVM. In the second phase of the system, a two-level hand detection model is developed using the proposed texture and color–texture features using a sliding window-based SVM classification model. Experimental analysis shows that proposed sparse-based features are not only time efficient but also shows an improvement of 23.4% in the practical two-level detection system over the existing motion and skin filtering-based hand detection system.

Remote Sensing Scene Classification Based on Convolutional Neural Networks Pre-Trained Using Attention-Guided Sparse Filters

Semantic-level land-use scene classification is a challenging problem, in which deep learning methods, e.g., convolutional neural networks (CNNs), have shown remarkable capacity. However, a lack of sufficient labeled images has proved a hindrance to increasing the land-use scene classification accuracy of CNNs. Aiming at this problem, this paper proposes a CNN pre-training method under the guidance of a human visual attention mechanism. Specifically, a computational visual attention model is used to automatically extract salient regions in unlabeled images. Then, sparse filters are adopted to learn features from these salient regions, with the learnt parameters used to initialize the convolutional layers of the CNN. Finally, the CNN is further fine-tuned on labeled images. Experiments are performed on the UCMerced and AID datasets, which show that when combined with a demonstrative CNN, our method can achieve 2.24% higher accuracy than a plain CNN and can obtain an overall accuracy of 92.43% when combined with AlexNet. The results indicate that the proposed method can effectively improve CNN performance using easy-to-access unlabeled images and thus will enhance the performance of land-use scene classification especially when a large-scale labeled dataset is unavailable.