K-nearest Neighbor Graph Research Articles

On the law of the iterated logarithm and strong invariance principles in stochastic geometry

We study the law of the iterated logarithm (Khinchin (1924), Kolmogorov (1929)) and related strong invariance principles for functionals in stochastic geometry. As potential applications, we think of well-known functionals defined on the k-nearest neighbors graph and important functionals in topological data analysis such as the Euler characteristic and persistent Betti numbers.

DTFLOW: Inference and Visualization of Single-cell Pseudotime Trajectory Using Diffusion Propagation

One of the major challenges in single-cell data analysis is the determination of cellular developmental trajectories using single-cell data. Although substantial studies have been conducted in recent years, more effective methods are still strongly needed to infer the developmental processes accurately. This work devises a new method, named DTFLOW, for determining the pseudo-temporal trajectories with multiple branches. DTFLOW consists of two major steps: a new method called Bhattacharyya kernel feature decomposition (BKFD) to reduce the data dimensions, and a novel approach named Reverse Searching on k-nearest neighbor graph (RSKG) to identify the multi-branching processes of cellular differentiation. In BKFD, we first establish a stationary distribution for each cell to represent the transition of cellular developmental states based on the random walk with restart algorithm, and then propose a new distance metric for calculating pseudotime of single cells by introducing the Bhattacharyya kernel matrix. The effectiveness of DTFLOW is rigorously examined by using four single-cell datasets. We compare the efficiency of DTFLOW with the published state-of-the-art methods. Simulation results suggest that DTFLOW has superior accuracy and strong robustness properties for constructing pseudotime trajectories. The Python source code of DTFLOW can be freely accessed at https://github.com/statway/DTFLOW.

Refining a [formula omitted]-nearest neighbor graph for a computationally efficient spectral clustering

Spectral clustering became a popular choice for data clustering for its ability of uncovering clusters of different shapes. However, it is not always preferable over other clustering methods due to its computational demands. One of the effective ways to bypass these computational demands is to perform spectral clustering on a subset of points (data representatives) then generalize the clustering outcome, this is known as approximate spectral clustering (ASC). ASC uses sampling or quantization to select data representatives. This makes it vulnerable to 1) performance inconsistency (since these methods have a random step either in initialization or training), 2) local statistics loss (because the pairwise similarities are extracted from data representatives instead of data points). We proposed a refined version of k-nearest neighbor graph, in which we keep data points and aggressively reduce number of edges for computational efficiency. Local statistics were exploited to keep the edges that do not violate the intra-cluster distances and nullify all other edges in the k-nearest neighbor graph. We also introduced an optional step to automatically select the number of clusters C. The proposed method was tested on synthetic and real datasets. Compared to ASC methods, the proposed method delivered a consistent performance despite significant reduction of edges.

Optimal tuning of weighted kNN- and diffusion-based methods for denoising single cell genomics data.

The analysis of single-cell genomics data presents several statistical challenges, and extensive efforts have been made to produce methods for the analysis of this data that impute missing values, address sampling issues and quantify and correct for noise. In spite of such efforts, no consensus on best practices has been established and all current approaches vary substantially based on the available data and empirical tests. The k-Nearest Neighbor Graph (kNN-G) is often used to infer the identities of, and relationships between, cells and is the basis of many widely used dimensionality-reduction and projection methods. The kNN-G has also been the basis for imputation methods using, e.g., neighbor averaging and graph diffusion. However, due to the lack of an agreed-upon optimal objective function for choosing hyperparameters, these methods tend to oversmooth data, thereby resulting in a loss of information with regard to cell identity and the specific gene-to-gene patterns underlying regulatory mechanisms. In this paper, we investigate the tuning of kNN- and diffusion-based denoising methods with a novel non-stochastic method for optimally preserving biologically relevant informative variance in single-cell data. The framework, Denoising Expression data with a Weighted Affinity Kernel and Self-Supervision (DEWÄKSS), uses a self-supervised technique to tune its parameters. We demonstrate that denoising with optimal parameters selected by our objective function (i) is robust to preprocessing methods using data from established benchmarks, (ii) disentangles cellular identity and maintains robust clusters over dimension-reduction methods, (iii) maintains variance along several expression dimensions, unlike previous heuristic-based methods that tend to oversmooth data variance, and (iv) rarely involves diffusion but rather uses a fixed weighted kNN graph for denoising. Together, these findings provide a new understanding of kNN- and diffusion-based denoising methods. Code and example data for DEWÄKSS is available at https://gitlab.com/Xparx/dewakss/-/tree/Tjarnberg2020branch.

On the Merge of k-NN Graph

k-nearest neighbor graph is a fundamental data structure in many disciplines such as information retrieval, data-mining, pattern recognition, and machine learning, etc. In the literature, considerable research has been focusing on how to efficiently build an approximate k-nearest neighbor graph (k-NN graph) for a fixed dataset. Unfortunately, a closely related issue of how to merge two existing k-NN graphs has been overlooked. In this paper, we address the issue of k-NN graph merging in two different scenarios. In the first scenario, a symmetric merge algorithm is proposed to combine two approximate k-NN graphs. The algorithm facilitates large-scale processing by the efficient merging of k-NN graphs that are produced in parallel. In the second scenario, a joint merge algorithm is proposed to expand an existing k-NN graph with a raw dataset. The algorithm enables the incremental construction of a hierarchical approximate k-NN graph. Superior performance is attained when leveraging the hierarchy for NN search of various data types, dimensionality, and distance measures.

Self-Supervised Deep Subspace Clustering for Hyperspectral Images With Adaptive Self-Expressive Coefficient Matrix Initialization

Deep subspace clustering network has shown its effectiveness in hyperspectral image (HSI) clustering. However, there are two major challenges that need to be addressed: 1) lack of effective supervision for feature learning; and 2) negative effect caused by the high redundancy of the global dictionary atoms. In this article, we propose an end-to-end trainable network for HSI clustering. Specifically, to ensure the extracted features are well-suited to subsequent subspace clustering, the cluster assignments with high confidence are employed as pseudo-labels to supervise the feature learning process. Then, an adaptive self-expressive coefficient matrix initialization strategy is designed to reduce the dictionary redundancy, where the spectral similarity between each target sample and its neighbors is modeled via the k-nearest neighbor graph to guide the initialization. Experimental results on three public HSI datasets demonstrate the effectiveness of the proposed method. In particular, our method outperforms several state-of-the-art HSI clustering methods, and achieves overall accuracy of 100% on both SalinasA and Pavia University datasets.

Intelligent fault diagnosis for rolling bearings based on graph shift regularization with directed graphs

Graph shift regularization is a new and effective graph-based semi-supervised classification method, but its performance is closely related to the representation graphs. Since directed graphs can convey more information about the relationship between vertices than undirected graphs, an intelligent method called graph shift regularization with directed graphs (GSR-D) is presented for fault diagnosis of rolling bearings. For greatly improving the diagnosis performance of GSR-D, a directed and weighted k-nearest neighbor graph is first constructed by treating each sample (i.e., each vibration signal segment) as a vertex, in which the similarity between samples is measured by cosine distance instead of the commonly used Euclidean distance, and the edge weights are also defined by cosine distance instead of the commonly used heat kernel. Then, the labels of samples are considered as the graph signals indexed by the vertices of the representation graph. Finally, the states of unlabeled samples are predicted by finding a graph signal that has minimal total variation and satisfies the constraint given by labeled samples as much as possible. Experimental results indicate that GSR-D is better and more stable than the standard convolutional neural network and support vector machine in rolling bearing fault diagnosis, and GSR-D only has two tuning parameters with certain robustness.

Iterative Robust Graph for Unsupervised Change Detection of Heterogeneous Remote Sensing Images.

This work presents a robust graph mapping approach for the unsupervised heterogeneous change detection problem in remote sensing imagery. To address the challenge that heterogeneous images cannot be directly compared due to different imaging mechanisms, we take advantage of the fact that the heterogeneous images share the same structure information for the same ground object, which is imaging modality-invariant. The proposed method first constructs a robust K -nearest neighbor graph to represent the structure of each image, and then compares the graphs within the same image domain by means of graph mapping to calculate the forward and backward difference images, which can avoid the confusion of heterogeneous data. Finally, it detects the changes through a Markovian co-segmentation model that can fuse the forward and backward difference images in the segmentation process, which can be solved by the co-graph cut. Once the changed areas are detected by the Markovian co-segmentation, they will be propagated back into the graph construction process to reduce the influence of changed neighbors. This iterative framework makes the graph more robust and thus improves the final detection performance. Experimental results on different data sets confirm the effectiveness of the proposed method. Source code of the proposed method is made available at https://github.com/yulisun/IRG-McS.

Improving K-Nearest Neighbor Approaches for Density-Based Pixel Clustering in Hyperspectral Remote Sensing Images

We investigated nearest-neighbor density-based clustering for hyperspectral image analysis. Four existing techniques were considered that rely on a K-nearest neighbor (KNN) graph to estimate local density and to propagate labels through algorithm-specific labeling decisions. We first improved two of these techniques, a KNN variant of the density peaks clustering method dpc, and a weighted-mode variant of knnclust, so the four methods use the same input KNN graph and only differ by their labeling rules. We propose two regularization schemes for hyperspectral image analysis: (i) a graph regularization based on mutual nearest neighbors (MNN) prior to clustering to improve cluster discovery in high dimensions; (ii) a spatial regularization to account for correlation between neighboring pixels. We demonstrate the relevance of the proposed methods on synthetic data and hyperspectral images, and show they achieve superior overall performances in most cases, outperforming the state-of-the-art methods by up to 20% in kappa index on real hyperspectral images.

Integrative Analysis of Single-Cell RNA-Seq and ATAC-Seq Data across Treatment Time Points in Pediatric AML

Pediatric AML is a heterogeneous disease in which treatment resistance remains an unsolved problem that is responsible for most deaths (Yeung and Radich 2017). Recently we have come to learn that resistance may be driven by mechanisms that extend beyond somatic mutations and DNA methylation changes (Ghasemi et al. 2020; van Galen et al. 2019; Bell et al. 2019). Transcriptional changes within specific primitive and committed cell types in AML tumours, which may be accompanied by alterations in chromatin structure and topology, can also contribute to disease progression (Ghasemi et al. 2020). To study such changes at the single-cell level, we analyzed single-cell RNA-seq (scRNA-seq) and matched scATAC-seq data from primary, remission and/or relapse samples obtained from three pediatric AML patients enrolled in the AAML1031 clinical trial (Alpenc et al. 2016) (Figure 1). Using the 10X Genomics single-cell platforms, we profiled a total of 39,738 cells using scRNA-seq (~4,826 cells per sample, 1,571 genes per cell), and 46,580 cells and 197,128 peaks using scATAC-seq (~6,718 cells per sample, 5,628 unique reads per cell). We then integrated these data types to determine the extent to which these two modalities corroborated and/or complemented each other in analyses of these longitudinally-obtained samples. Cell subpopulations detected in scRNA-seq through Leiden clustering on a k-nearest neighbor graph were generally consistent with recent observations of malignant and normal cell types detected in the bone marrow and peripheral blood compartments (van Galen et al. 2019; Hay et al. 2018). Malignant-like subpopulations at primary and relapse stages exhibited similar levels of cell type diversity along the myeloid lineage. These included hematopoietic stem-like cells, progenitors, granulocyte-monocyte progenitors, monocytes and dendritic cell-like subpopulations. Remission samples appeared to contain normal blood cell types including natural killers (NK), B and T cells, platelets and erythrocytes, consistent with the clearance of blasts. However, we also observed putative malignant-like conventional dendritic cell subpopulations at remission (50% and 16% in the respective samples), noting that these cells displayed increased expression of genes involved in antigen presentation and lysosomal protein processing. To integrate scATAC-seq with scRNA-seq data we performed clustering of transformed and reduced scATAC-seq data through iterative latent semantic indexing (Granja et al. 2020), and aligned cells in scATAC-seq to cells from scRNA-seq data using canonical correlation analysis (Stuart et al. 2019). We observed similar patterns of T cell expansion, presence of monocyte-like populations and NK cells at remission in the scATAC-seq data. However, scRNA-seq subpopulations dominated by malignant-like cells showed variability in mapping to distinctive chromatin states, with a few notable exceptions (Figures 2 and 3). One such exception is a subpopulation in scRNA-seq, found mostly at relapse, marked by high expression of genes involved in proliferation and growth factor-mediated cellular processes such as YBX3 (binds to GM-CSF promoter), CYTL1, and EGFL7 (regulator of vasculogenesis) (Figures 3 and 4). Cells within this subpopulation mapped to two scATAC-seq clusters whose significantly more highly accessible regions were enriched for functional processes such as blood vessel remodeling and neutrophil/granulocyte activation (Figure 4). These observations are consistent with recent evidence that AML tumour cells can activate the immune system to acquire resistance (Melgar et al. 2020). The scRNA-seq subpopulation, however, did not display high expression of myeloid/granulocyte factors such as CD15, ELANE, and MPO (Figure 4), perhaps consistent with the notion that such transcriptional programs may be primed but not yet activated within these malignant cells. We thus evaluated the potential of scATAC-seq to complement scRNA-seq in understanding transcriptional changes within cell types in AML tumours. We observed that normal cell types and specific malignant cell states could occupy distinctive chromatin states. Through integrative analyses, we conclude that scATAC-seq results can add additional information to complement scRNA-seq data, including identifying nascent transcriptional programs that may be poised for activation within malignant cells. Disclosures No relevant conflicts of interest to declare.

Graph-based Regularization on Embedding Layers for Recommendation

Neural networks have been extensively used in recommender systems. Embedding layers are not only necessary but also crucial for neural models in recommendation as a typical discrete task. In this article, we argue that the widely used l 2 regularization for normal neural layers (e.g., fully connected layers) is not ideal for embedding layers from the perspective of regularization theory in Reproducing Kernel Hilbert Space. More specifically, the l 2 regularization corresponds to the inner product and the distance in the Euclidean space where correlations between discrete objects (e.g., items) are not well captured. Inspired by this observation, we propose a graph-based regularization approach to serve as a counterpart of the l 2 regularization for embedding layers. The proposed regularization incurs almost no extra computational overhead especially when being trained with mini-batches. We also discuss its relationships to other approaches (namely, data augmentation, graph convolution, and joint learning) theoretically. We conducted extensive experiments on five publicly available datasets from various domains with two state-of-the-art recommendation models. Results show that given a kNN (k-nearest neighbor) graph constructed directly from training data without external information, the proposed approach significantly outperforms the l 2 regularization on all the datasets and achieves more notable improvements for long-tail users and items.

Clustering by Detecting Density Peaks and Assigning Points by Similarity-First Search Based on Weighted K-Nearest Neighbors Graph

This paper presents an improved clustering algorithm for categorizing data with arbitrary shapes. Most of the conventional clustering approaches work only with round-shaped clusters. This task can be accomplished by quickly searching and finding clustering methods for density peaks (DPC), but in some cases, it is limited by density peaks and allocation strategy. To overcome these limitations, two improvements are proposed in this paper. To describe the clustering center more comprehensively, the definitions of local density and relative distance are fused with multiple distances, including K-nearest neighbors (KNN) and shared-nearest neighbors (SNN). A similarity-first search algorithm is designed to search the most matching cluster centers for noncenter points in a weighted KNN graph. Extensive comparison with several existing DPC methods, e.g., traditional DPC algorithm, density-based spatial clustering of applications with noise (DBSCAN), affinity propagation (AP), FKNN-DPC, and K-means methods, has been carried out. Experiments based on synthetic data and real data show that the proposed clustering algorithm can outperform DPC, DBSCAN, AP, and K-means in terms of the clustering accuracy (ACC), the adjusted mutual information (AMI), and the adjusted Rand index (ARI).

Diffusion Parameters Analysis in a Content-Based Image Retrieval Task for Mobile Vision.

Most recent computer vision tasks take into account the distribution of image features to obtain more powerful models and better performance. One of the most commonly used techniques to this purpose is the diffusion algorithm, which fuses manifold data and k-Nearest Neighbors (kNN) graphs. In this paper, we describe how we optimized diffusion in an image retrieval task aimed at mobile vision applications, in order to obtain a good trade-off between computation load and performance. From a computational efficiency viewpoint, the high complexity of the exhaustive creation of a full kNN graph for a large database renders such a process unfeasible on mobile devices. From a retrieval performance viewpoint, the diffusion parameters are strongly task-dependent and affect significantly the algorithm performance. In the method we describe herein, we tackle the first issue by using approximate algorithms in building the kNN tree. The main contribution of this work is the optimization of diffusion parameters using a genetic algorithm (GA), which allows us to guarantee high retrieval performance in spite of such a simplification. The results we have obtained confirm that the global search for the optimal diffusion parameters performed by a genetic algorithm is equivalent to a massive analysis of the diffusion parameter space for which an exhaustive search would be totally unfeasible. We show that even a grid search could often be less efficient (and effective) than the GA, i.e., that the genetic algorithm most often produces better diffusion settings when equal computing resources are available to the two approaches. Our method has been tested on several publicly-available datasets: Oxford5k, Oxford5k, Paris6k, Paris6k, and Oxford105k, and compared to other mainstream approaches.

Robust and efficient single-cell Hi-C clustering withapproximate k-nearest neighbor graphs

Robust and efficient single-cell Hi-C clustering withapproximate k-nearest neighbor graphs

Group Re-Identification With Group Context Graph Neural Networks

Group re-identification aims to match groups of people across disjoint cameras. In this task, the contextual information from neighbor individuals can be exploited for re-identifying each individual within the group as well as the entire group. However, compared with single person re-identification, it brings new challenges including group layout and group membership changes. Motivated by the observation that individuals who are close together are more likely to keep in the same group under different cameras than those who are far apart, we propose to model each group as a spatial K-nearest neighbor graph (SKNNG) and design a group context graph neural network (GCGNN) for graph representation learning. Specifically, for each node in the graph, the proposed GCGNN learns an embedding which aggregates the contextual information from neighbor nodes. We design multiple weighting kernels for neighborhood aggregation based on the graph properties including node in-degrees and spatial relationship attributes. We compute the similarity scores between node embeddings of two graphs for group member association and obtain the matching score between the two graphs by summing up the similarity scores of all linked node pairs. Experimental results on three public datasets show that our approach performs favorably against state-of-the-art methods and achieves high efficiency.

Graph Embedding-Based Wireless Link Scheduling With Few Training Samples

Link scheduling in device-to-device (D2D) networks is usually formulated as a non-convex combinatorial problem, which is generally NP-hard and difficult to get the optimal solution. Traditional methods to solve this problem are mainly based on mathematical optimization techniques, where accurate channel state information (CSI), usually obtained through channel estimation and feedback, is needed. To overcome the high computational complexity of the traditional methods and eliminate the costly channel estimation stage, machine leaning (ML) has been introduced recently to address the wireless link scheduling problems. In this article, we propose a novel graph embedding based method for link scheduling in D2D networks. We first construct a fully-connected directed graph for the D2D network, where each D2D pair is a node while interference links among D2D pairs are the edges. Then we compute a low-dimensional feature vector for each node in the graph. The graph embedding process is based on the distances of both communication and interference links, therefore without requiring the accurate CSI. By utilizing a multi-layer classifier, a scheduling strategy can be learned in a supervised manner based on the graph embedding results for each node. We also propose an unsupervised manner to train the graph embedding based method to further reinforce the scalability and develop a K-nearest neighbor graph representation method to reduce the computational complexity. Extensive simulation demonstrates that the proposed method is near-optimal compared with the existing state-of-art methods but is with only hundreds of training network layouts. It is also competitive in terms of scalability and generalizability to more complicated scenarios.

An enhanced KNN-based twin support vector machine with stable learning rules

Among the extensions of twin support vector machine (TSVM), some scholars have utilized K-nearest neighbor (KNN) graph to enhance TSVM's classification accuracy. However, these KNN-based TSVM classifiers have two major issues such as high computational cost and overfitting. In order to address these issues, this paper presents an enhanced regularized K-nearest neighbor based twin support vector machine (RKNN-TSVM). It has three additional advantages: (1) Weight is given to each sample by considering the distance from its nearest neighbors. This further reduces the effect of noise and outliers on the output model. (2) An extra stabilizer term was added to each objective function. As a result, the learning rules of the proposed method are stable. (3) To reduce the computational cost of finding KNNs for all the samples, location difference of multiple distances based k-nearest neighbors algorithm (LDMDBA) was embedded into the learning process of the proposed method. The extensive experimental results on several synthetic and benchmark datasets show the effectiveness of our proposed RKNN-TSVM in both classification accuracy and computational time. Moreover, the largest speedup in the proposed method reaches to 14 times.

Survey on Clustering High-Dimensional data using Hubness

Most data of interest today in data-mining applications is complex and is usually represented by many different features. Such high-dimensional data is by its very nature often quite difficult to handle by conventional machine-learning algorithms. This is considered to be an aspect of the well known curse of dimensionality. Consequently, high-dimensional data needs to be processed with care, which is why the design of machine-learning algorithms needs to take these factors into account. Furthermore, it was observed that some of the arising high-dimensional properties could in fact be exploited in improving overall algorithm design. One such phenomenon, related to nearest-neighbor learning methods, is known as hubness and refers to the emergence of very influential nodes (hubs) in k-nearest neighbor graphs. A crisp weighted voting scheme for the k-nearest neighbor classifier has recently been proposed which exploits this notion.

A Novel and Efficient Automatic Hyperspectral Image Registration Technique

Abstract Multi-module images registration (IR) is a challenging task in the area of hyperspectral images collected from satellite and airborne sensors. This is because of substantial illumination variation and geometric distortions (GD), high accuracies and robust matching of multi-module hyperspectral images is a difficult task. Along with, the ground surface variation challenges feature point set identification in quantity and quality, which is a general problem posed by feature set based IR methods. For addressing the problem existing IR model are designed using SIFT and CNN. However, under extreme appearance difference, identified feature point sets extracted by existing IR method might contain higher ratio of outliers, whereas inlier might be insufficient and unequally dispersed. For overcoming research problem, this paper presented a novel and efficient automatic IR method for hyperspectral image. Feature extraction is done using SURF and SIFT descriptors initially then, using K-Nearest neighbour (KNN) graph the mismatch point is eliminated. Further, the positional accuracy is improved using least square sum method. Multi-quadric transformation method is used for attaining better IR quality at the end. The proposed IR method attain significant performance when compared with existing IR method.

Robust and Efficient Person Re-Identification Model using K-Nearest Neighbor Graph

the state-of-art person re-identification (prid) models for ranking generally depends on labeled pairwise feature sets information to learn a task-dependent distance metric. Further, in retrieval process, re-ranking is an important mechanism for enhancing the accuracy. However, very limited work is carried out for designing a re-ranking method, particularly for automatic and unsupervised strategies. The existing re-ranking based prid model is not efficient when multiple persons appears simultaneously in second camera. This is because the existing model identify person in second camera by matching the feature sets with feature sets in first camera, individually with respect to other person in the second camera. For overcoming research problem, this paper present robust and efficient prid (reprid) model. First, present a robust learning/ranking method using k-nearest neighbor (knn) graph. Then, this work present a re-ranking method to improve accuracy of prid by using information of co-occurrence persons for matching and reorganizing given rank lists. Experiment are conducted on standard dataset shows robustness and effectiveness of proposed prid method.