Manifold Ranking Research Articles

Saliency Detection via Manifold Ranking on Multi-Layer Graph

Saliency Detection via Manifold Ranking on Multi-Layer Graph

Effective and Scalable Manifold Ranking-Based Image Retrieval with Output Bound

Image retrieval keeps attracting a lot of attention from both academic and industry over past years due to its variety of useful applications. Due to the rapid growth of deep learning approaches, more better feature vectors of images could be discovered for improving image retrieval. However, most (if not all) existing deep learning approaches consider the similarity between two images locally without considering the similarity among a group of similar images globally , and thus could not return accurate results. In this article, we study the image retrieval with manifold ranking (MR) which considers both the local similarity and the global similarity, which could give more accurate results. However, existing best-known algorithms have one of the following issues: (1) they require to build a bulky index, (2) some of them do not have any theoretical bound on the output, and (3) some of them are time-consuming. Motivated by this, we propose two algorithms, namely Monte Carlo-based MR ( MCMR ) and MCMR+ , for image retrieval, which do not have the above issues. We are the first one to propose an index-free manifold ranking image retrieval with the output theoretical bound. More importantly, our algorithms give the first best-known time complexity result of \(O(n \log n)\) where \(n\) is the total number of images in the database compared with the existing best-known result of \(O(n^2)\) in the literature of computing the exact top- \(k\) results with quality guarantee. Lastly, our experimental result shows that MCMR+ outperforms existing algorithms by up to four orders of magnitude in terms of query time.

Confidence-Aware Active Feedback for Interactive Instance Search

Online relevance feedback (RF) is widely utilized in instance search (INS) tasks to further refine imperfect ranking results, but it often has low interaction efficiency. The active learning (AL) technique addresses this problem by selecting valuable feedback candidates. However, mainstream AL methods require an initial labeled set for a cold start and are often computationally complex to solve. Therefore, they cannot fully satisfy the requirements for online RF in interactive INS tasks. To address this issue, we propose a confidence-aware active feedback method (CAAF) that is specifically designed for online RF in interactive INS tasks. Inspired by the explicit difficulty modeling scheme in self-paced learning, CAAF utilizes a pairwise manifold ranking loss to evaluate the ranking confidence of each unlabeled sample. The ranking confidence improves not only the interaction efficiency by indicating valuable feedback candidates but also the ranking quality by modulating the diffusion weights in manifold ranking. In addition, we design two acceleration strategies, an approximate optimization scheme and a top-K search scheme, to reduce the computational complexity of CAAF. Extensive experiments on both image INS tasks and video INS tasks searching for buildings, landscapes, persons, and human behaviors demonstrate the effectiveness of the proposed method. Notably, in the real-world, large-scale video INS task of NIST TRECVID 2021, CAAF uses 25% fewer feedback samples to achieve a performance that is nearly equivalent to the champion solution. Moreover, with the same number of feedback samples, CAAF's mAP is 51.9%, significantly surpassing the champion solution by 5.9%. Code is available at https://github.com/nercms-mmap/caaf.

A Visual Target Representation using Saliency Detection Approach

The task of saliency detection is to identify the most important and informative part of a scene. Saliency detection is broadly applied to numerous vision problems, including image segmentation, object recognition, image compression, content-based image retrieval, and moving object detection. Existing saliency detection methods suffer a low accuracy rate because of missing components of saliency regions. This study proposes a visual saliency detection method for the target representation to represent targets more accurately. The proposed method consists of five modules. In the first module, the salient region is extracted through manifold ranking on a graph, which incorporates local grouping cues and boundary priors. Secondly, using a region of interest (ROI) algorithm and the subtraction of the salient region from the original image, other parts of the image, either related or nonrelated to the interested target, are segmented. Lastly, those related and non-related regions are classified and distinguished using our proposed algorithm. Experimental result shows that proposed salient region accurately represent the interested target which can be used for object detection and tracking applications.

Self-supervised image co-saliency detection

Self-supervised learning has been shown to be highly effective to avoid the expensive cost of collecting and annotating large-scale datasets. In this paper, we explore its use in image co-saliency detection task which detects and segments the co-concurrent and conspicuous patterns in one image group, and a self-supervised framework is presented that contains three novel module designs. Specifically, we first design an unsupervised graph clustering algorithm to generate an initial detection result from extracted semantical features of convolutional networks. Then, the post-processed output is taken as the pseudo labels of the second module that learns a neural network for more accurate sample affinity estimation, followed by a manifold ranking based label smoothing. Finally, with the smoothed labels as supervision, we design a saliency network that aggregates a hierarchy of multi-scale features to predict final optimized co-saliency maps. In addition, the proposed framework can be readily extended to image saliency object detection task when giving a single image as input. Extensive evaluations on three co-saliency and five saliency object detection benchmark datasets demonstrate that the proposed framework achieves favorable performance against a variety of supervised and unsupervised state-of-the-arts.

Composite Kernel of Mutual Learning on Mid-Level Features for Hyperspectral Image Classification.

By training different models and averaging their predictions, the performance of the machine-learning algorithm can be improved. The performance optimization of multiple models is supposed to generalize further data well. This requires the knowledge transfer of generalization information between models. In this article, a multiple kernel mutual learning method based on transfer learning of combined mid-level features is proposed for hyperspectral classification. Three-layer homogenous superpixels are computed on the image formed by PCA, which is used for computing mid-level features. The three mid-level features include: 1) the sparse reconstructed feature; 2) combined mean feature; and 3) uniqueness. The sparse reconstruction feature is obtained by a joint sparse representation model under the constraint of three-scale superpixels' boundaries and regions. The combined mean features are computed with average values of spectra in multilayer superpixels, and the uniqueness is obtained by the superposed manifold ranking values of multilayer superpixels. Next, three kernels of samples in different feature spaces are computed for mutual learning by minimizing the divergence. Then, a combined kernel is constructed to optimize the sample distance measurement and applied by employing SVM training to build classifiers. Experiments are performed on real hyperspectral datasets, and the corresponding results demonstrated that the proposed method can perform significantly better than several state-of-the-art competitive algorithms based on MKL and deep learning.

Feature augmentation based on manifold ranking and LSTM for image classification

Image classification is a critical topic due to its wide application and several challenges associated. Despite the huge progress made last decades, there is still a demand for context-aware image representation approaches capable of taking into the dataset manifold for improving classification accuracy. In this work, a representation learning approach is proposed, based on a novel feature augmentation strategy. The proposed method aims to exploit available contextual similarity information through rank-based manifold learning used to define and assign weights to samples used in augmentation. The approach is validated using CNN-based features and LSTM models to achieve even higher accuracy results on image classification tasks. Experimental results show that the feature augmentation strategy can indeed improve the accuracy of results on widely used image datasets (CIFAR10, Stanford Dogs, Linnaeus5, Flowers102 and Flowers17) in different CNNs (ResNet152, VGG16, DPN92). The results indicate gains up to 20% and show the potential of the developed approach in achieving higher accuracy results for image classification.

Weakly supervised learning based on hypergraph manifold ranking

Significant challenges still remain despite the impressive recent advances in machine learning techniques, particularly in multimedia data understanding. One of the main challenges in real-world scenarios is the nature and relation between training and test datasets. Very often, only small sets of coarse-grained labeled data are available to train models, which are expected to be applied on large datasets and fine-grained tasks. Weakly supervised learning approaches handle such constraints by maximizing useful training information in labeled and unlabeled data. In this research direction, we propose a weakly supervised approach that analyzes the dataset manifold to expand the available labeled set. A hypergraph manifold ranking algorithm is exploited to represent the contextual similarity information encoded in the unlabeled data and identify strong similarity relations, which are taken as a path to label expansion. The expanded labeled set is subsequently exploited for a more comprehensive and accurate training process. The proposed model was evaluated jointly with supervised and semi-supervised classifiers, including Graph Convolutional Networks. The experimental results on image and video datasets demonstrate significant gains and accurate results for different classifiers in diverse scenarios.

Effects of Different Superpixel Algorithms on Interactive Segmentations

Semi-automated segmentation or more commonly known as interactive image segmentation is an algorithm that extracts a region of interest (ROI) from an image based on the input information from the user. The said algorithm will be repetitively fed with such input information until required region of interest is successfully segmented. To accelerate this segmentation procedure as well as enhancing the result, pre-processing steps can be applied. The application of superpixel is an example of such pre-processing step. Superpixel can be defined as a collection of pixels that share common features such as texture and colours. Though employed as pre-processing step in many interactive segmentation algorithms, to date, no study has been conducted to assess the effects of such incorporations on the segmentation algorithms. Thus, this study aims to address this issue. In this study, five different types of superpixels ranging from watershed, density, graph, clustering and energy optimization categories are evaluated. The superpixels generated by these five algorithms will be used on two interactive image segmentation algorithms: i) Maximal Similarity based Region Merging (MSRM) and ii) Graph-Based Manifold Ranking (GBMR) with single and multiple strokes on various images from the Berkeley image dataset. The result of testing had shown that MSRM achieved better result compared to GBMR in both single and multiple input strokes using SEEDS superpixel algorithm. This study summary concluded that at different superpixel algorithms produced different results and that it is not possible to single out one particular superpixel algorithm that can work well for all the interactive segmentation algorithms. As such, the key to achieving a decent segmentation result lies in choosing the right superpixel algorithms for a given interactive segmentation algorithm.

A K2 graph-based fusion model with manifold ranking for robot image saliency detection

A K2 graph-based fusion model with manifold ranking for robot image saliency detection

Salient object detection based on multi-feature graphs and improved manifold ranking

Salient object detection based on multi-feature graphs and improved manifold ranking

Cosaliency Detection and Region-of-Interest Extraction via Manifold Ranking and MRF in Remote Sensing Images

Saliency-based region-of-interest (ROI) extraction is significant for the interpretation of remote sensing images (RSIs). Recently, cosaliency detection has shown its superiority of better extraction of common ROIs by using both intraimage and interimage cues. However, most existing methods still suffer from the complex backgrounds of RSIs, resulting in incomplete ROI extraction, many false positives, and blurred boundaries. In this article, we propose a cosaliency detection framework via manifold ranking and the Markov random field (MRF) for RSIs to address these problems. First, we design a two-stage manifold ranking schema for converting single-image saliency maps (SISMs) to multi-image saliency maps (MISMs). This step takes full advantage of the correlation between images to improve the integrity of ROIs and reduce false positives. Second, we locally fuse saliency proposals by minimizing the energy function in an MRF. The design of the energy function comprehensively considers the global and local performance of saliency proposals to assign appropriate fusion weights. Finally, we generate the ROI masks by thresholding the cosaliency maps. Our approach is evaluated on four RSI datasets and compared to the state-of-the-art methods. Experimental results demonstrate the effectiveness of our model in both cosaliency detection and ROI extraction.

Improved Manifold Ranking Algorithm for Green Citrus Recognition

Improved Manifold Ranking Algorithm for Green Citrus Recognition

An adaptive defect detection method for LNG storage tank insulation layer based on visual saliency

As the core equipment of the industrial chain, the LNG storage tank is developing in the direction of large-scale and increasing attention has been paid to its economy and safety. The defects of the insulation layer not only affect the insulation performance of the tank but also easily induce safety accidents. To automatically detect these defects, considering the large temperature difference between the defect area and other areas, an adaptive detection method based on infrared thermal imaging technology and a saliency detection algorithm is proposed. Firstly, the median filtering and the Contrast Limited Adaptive Histogram (CLAHE) are introduced to reduce noise and smooth the image. Then, the Adaptive Simple Linear Iterative Clustering algorithm (A-SLIC) based on the entropy of the image is purposed for super-pixel segmentation. After super-pixel segmentation, the saliency map is obtained by manifold ranking, and the salient areas that contain defects of the insulation layer will be segmented automatically. The proposed method is compared with other traditional algorithms on the MSRA1000 dataset and the infrared thermal imaging dataset of LNG storage tanks. The results of the experiments illustrate the effectiveness of the method proposed.

Research on Building Target Detection Based on High-Resolution Optical Remote Sensing Imagery

High-resolution remote sensing image building target detection has wide application value in the fields of land planning, geographic monitoring, smart cities and other fields. However, due to the complex background of remote sensing imagery, some detailed features of building targets are less distinguishable from the background. When carrying out the detection task, it is prone to problems such as distortion and the missing of the building outline. To address this challenge, we developed a novel building target detection method. First, a building detection method based on rectangular approximation and region growth was proposed, and a saliency detection model based on the foreground compactness and local contrast of manifold ranking is used to obtain the saliency map of the building region. Then, the boundary prior saliency detection method based on the improved manifold ranking algorithm was proposed for the target area of buildings with low contrast with the background in remote sensing imagery. Finally, fusing the results of the rectangular approximation-based and saliency-based detection, the proposed fusion method improved the Recall and F1 value of building detection, indicating that this paper provides an effective and efficient high-resolution remote sensing image building target detection method.

A Bregman Divergence Optimization Framework for Ranking on Data Manifold and Its New Extensions

Recently, graph-based ranking algorithms have received considerable interests in machine learning, computer vision and information retrieval communities. Ranking on data manifold (or manifold ranking, MR) is one of the representative approaches. One of the limitations of manifold ranking is its high computational complexity (O(n3), where n is the number of samples in database). In this paper, we cast the manifold ranking into a Bregman divergence optimization framework under which we transform the original MR to an equivalent optimal kernel matrix learning problem.With this new formulation, two effective and efficient extensions are proposed to enhance the ranking performance. Extensive experimental results on two real world image databases show the effectiveness of the proposed approach.

Seeded random walk for multi-view semi-supervised classification

Recently, multi-view learning has captured widespread attention in the machine learning area, yet it is still crucial and challenging to exploit beneficial patterns from multi-view data. Specifically, very limited work has been devoted to multi-view semi-supervised learning, where only a small number of labeled data points are available for model training. Therefore, a simple yet efficient seeded random walk scheme is proposed in this paper to address the multi-view semi-supervised classification problem, where known labeled data points serve as random seeds to be walked with certain probability. In this scheme, the semi-supervised classification indicator is obtained based primarily on an arrival probability and a reward matrix, which are computed by leveraging an initial distribution from some random seeds. Besides, theoretical analyses are then provided to indicate a connection of the proposed model with the existing manifold ranking method. Finally, comprehensive experiments on eight publicly available data sets demonstrate the superiority of the proposed model against compared state-of-the-art semi-supervised methods and fully supervised classifiers. Furthermore, experimental results also suggest that the proposed method comes with positive robustness and promising generalization capability in terms of data classification.

Fast Manifold Ranking With Local Bipartite Graph

During the past decades, manifold ranking has been widely applied to content-based image retrieval and shown excellent performance. However, manifold ranking is computationally expensive in both graph construction and ranking learning. Much effort has been devoted to improve its performance by introducing approximating techniques. In this paper, we propose a fast manifold ranking method, namely Local Bipartite Manifold Ranking (LBMR). Given a set of images, we first extract multiple regions from each image to form a large image descriptor matrix, and then use the anchor-based strategy to construct a local bipartite graph in which a regional k -means (RKM) is proposed to obtain high quality anchors. We propose an iterative method to directly solve the manifold ranking problem from the local bipartite graph, which monotonically decreases the objective function value in each iteration until the algorithm converges. Experimental results on several real-world image datasets demonstrate the effectiveness and efficiency of our proposed method.

Graph Regularized Hierarchical Diffusion Process With Relevance Feedback for Medical Image Retrieval

Befitting from the interpretability and the capacity in capturing the underlying manifold structure, diffusion process (DP) has attracted increasing attention in the field of image retrieval. Within it, hierarchical diffusion process (HDP) has achieved satisfactory results in retrieved performance and complexity. However, the existing hierarchical diffusion process methods only diffuse the affinity values in low-level visual space without considering the high-level semantic information, which cause the problem of semantic gap. To overcome these problems, we propose a Graph Regularized Hierarchical Diffusion Process (GRHDP) method with relevance feedback, and apply it to retrieve medical images. The proposed algorithm firstly establishes a hierarchical structure of the images in medical image database and spreads the affinity values among query images and top-layer images by graph regularization diffusion. Then relevance feedback is introduced to adjust the similarity between query images and retrieved images in top layer, and the affinity values are diffused again according to labeled information of feedback. Finally, the similarity between queries and others in database can be obtained by interpolating the diffused results on the top layer from top to bottom. The experimental results show that our proposed GRHDP with relevance feedback has achieved better retrieval performance than manifold ranking and regularized diffusion process (RDP) when returning top retrieved images.

A scalable sub-graph regularization for efficient content based image retrieval with long-term relevance feedback enhancement

The goal of content-based image retrieval (CBIR) is to search relevant images through the analysis of image content. Manifold Ranking (MR) and Efficient Manifold Ranking (EMR) method has been successfully applied to content-based image retrieval due to its ability to discover underlying geometrical structure of dataset given the query data. But given the image database is scalable, the graph in MR and EMR cannot be extended or updated as their graph size is fixed. In this paper, to solve the above problem, we consider to formulate a sub-graph based on fixed anchors, instead of constructing the graph based on the while dataset, where the anchors are selected by utilizing conventional k-means method and the sub-graph weight matrix is defined by the similarity between any pair-wise anchors. Since the number of anchors is much smaller than the original dataset, updating the sub-graph is much easier than the original graph of whole dataset. Motivated by such sub-graph construction, we then develop an efficient graph regularization framework to predict the ranking scores for the whole data along the sub-graph, where the ranking score is first propagated from query image to the partial anchors, then from partial anchors to all anchors via the sub-graph and finally to the whole dataset. It can also utilize user relevance feedbacks to update the sub-graph so that the discriminative information can be involved to enhance the retrieval performance in a long term. Extensive simulations verify the effectiveness of the proposed method.