Spectral Rotation Research Articles

A Novel k-Means Framework via Constrained Relaxation and Spectral Rotation.

Owing to its simplicity, the traditional k - means (Lloyd heuristic) clustering method plays a vital role in a variety of machine-learning applications. Disappointingly, the Lloyd heuristic is prone to local minima. In this article, we propose k - mRSR, which converts the sum-of-squared error (SSE) (Lloyd) into a combinatorial optimization problem and incorporates a relaxed trace maximization term and an improved spectral rotation term. The main advantage of k - mRSR is that it only needs to solve the membership matrix instead of computing the cluster centers in each iteration. Furthermore, we present a nonredundant coordinate descent method that brings the discrete solution infinitely close to the scaled partition matrix. Two novel findings from the experiments are that k - mRSR can further decrease (increase) the objective function values of the k - means obtained by Lloyd (CD), while Lloyd (CD) cannot decrease (increase) the objective function obtained by k - mRSR. In addition, the results of extensive experiments on 15 datasets indicate that k - mRSR outperforms both Lloyd and CD in terms of the objective function value and outperforms other state-of-the-art methods in terms of clustering performance.

Fuzzy Clustering Guided by Spectral Rotation and Scaling.

Representing data in different spaces becomes more powerful and suitable for solving downstream learning tasks. The membership degrees obtained through fuzzy C -means (FCM) clustering cannot capture data structures sufficiently, as they represent samples from a single Euclidean geometrical perspective. To address this issue, we propose a novel fuzzy clustering model guided by spectral rotation and scaling (FCSR). In FCSR, both spectral embeddings and membership degrees are considered as new representations of data. They can complement each other from different perspectives which enables the model to engage more structural properties of the data. The process of solving the problem of membership degrees not only inherits the merits of traditional FCM but also preserves data neighborhood structures revealed by the spectral decomposition based on an affinity matrix. Furthermore, to improve the adaptability and extensibility of FCSR, the projected and kernel versions of FCSR (FCSR-P and FCSR-K) are formed. We demonstrate that FCSR-P is suitable for high-dimensional scenarios and FCSR-K can improve the linear separability among data. Extensive experiments conducted on various well-known data sets illustrate the validity of the proposed ideas.

Tensorized Discrete Multi-View Spectral Clustering

Discrete spectral clustering directly obtains the discrete labels of data, but existing clustering methods assume that the real-valued indicator matrices of different views are identical, which is unreasonable in practical applications. Moreover, they do not effectively exploit the spatial structure and complementary information embedded in views. To overcome this disadvantage, we propose a tensorized discrete multi-view spectral clustering model that integrates spectral embedding and spectral rotation into a unified framework. Specifically, we leverage the weighted tensor nuclear-norm regularizer on the third-order tensor, which consists of the real-valued indicator matrices of views, to exploit the complementary information embedded in the indicator matrices of different views. Furthermore, we present an adaptively weighted scheme that takes into account the relationship between views for clustering. Finally, discrete labels are obtained by spectral rotation. Experiments show the effectiveness of our proposed method.

One-Step Multiview Clustering via Adaptive Graph Learning and Spectral Rotation.

In graph based multiview clustering methods, the ultimate partition result is usually achieved by spectral embedding of the consistent graph using some traditional clustering methods, such as k -means. However, optimal performance will be reduced by this multistep procedure since it cannot unify graph learning with partition generation closely. In this article, we propose a one-step multiview clustering method through adaptive graph learning and spectral rotation (AGLSR). For every view, AGLSR adaptively learns affinity graphs to capture similar relationships of samples. Then, a spectral embedding is designed to take advantage of the potential feature space shared by different views. In addition, AGLSR utilizes a spectral rotation strategy to obtain the discrete clustering labels from the learned spectral embeddings directly. An effective updating algorithm with proven convergence is derived to optimize the optimization problem. Sufficient experiments on benchmark datasets have clearly demonstrated the effectiveness of the proposed method in six metrics. The code of AGLSR is uploaded at https://github.com/tangchuan2000/AGLSR.

Discretize Relaxed Solution of Spectral Clustering via a Nonheuristic Algorithm.

Spectral clustering and its extensions usually consist of two steps: 1) constructing a graph and computing the relaxed solution and 2) discretizing relaxed solutions. Although the former has been extensively investigated, the discretization techniques are mainly heuristic methods, e.g., k -means (KM), spectral rotation (SR). Unfortunately, the goal of the existing methods is not to find a discrete solution that minimizes the original objective. In other words, the primary drawback is the neglect of the original objective when computing the discrete solution. Inspired by the first-order optimization algorithms, we propose to develop a first-order term to bridge the original problem and discretization algorithm, which is the first nonheuristic to the best of our knowledge. Since the nonheuristic method is aware of the original graph cut problem, the final discrete solution is more reliable and achieves the preferable loss value. We also theoretically show that the continuous optimum is beneficial to discretization algorithms though simply finding its closest discrete solution is an existing heuristic algorithm which is also unreliable. Sufficient experiments significantly show the superiority of our method.

A Novel Normalized-Cut Solver With Nearest Neighbor Hierarchical Initialization.

Normalized-Cut (N-Cut) is a famous model of spectral clustering. The traditional N-Cut solvers are two-stage: 1) calculating the continuous spectral embedding of normalized Laplacian matrix; 2) discretization via K-means or spectral rotation. However, this paradigm brings two vital problems: 1) two-stage methods solve a relaxed version of the original problem, so they cannot obtain good solutions for the original N-Cut problem; 2) solving the relaxed problem requires eigenvalue decomposition, which has O(n3) time complexity ( n is the number of nodes). To address the problems, we propose a novel N-Cut solver designed based on the famous coordinate descent method. Since the vanilla coordinate descent method also has O(n3) time complexity, we design various accelerating strategies to reduce the time complexity to O(|E|) ( |E| is the number of edges). To avoid reliance on random initialization which brings uncertainties to clustering, we propose an efficient initialization method that gives deterministic outputs. Extensive experiments on several benchmark datasets demonstrate that the proposed solver can obtain larger objective values of N-Cut, meanwhile achieving better clustering performance compared to traditional solvers.

FINC: An Efficient and Effective Optimization Method for Normalized Cut.

The optimization methods for solving the normalized cut model usually involve three steps, i.e., problem relaxation, problem solving and post-processing. However, these methods are problematic in both performance since they do not directly solve the original problem, and efficiency since they usually depend on the time-consuming eigendecomposition and k-means (or spectral rotation) for post-processing. In this paper, we propose a fast optimization method to speedup the classical normalized cut clustering process, in which an auxiliary variable is introduced and alternatively updated along with the cluster indicator matrix. The new method is faster than the conventional three-step optimization methods since it solves the normalized cut problem in one step. Theoretical analysis reveals that the new method is able to monotonically decrease the normalized cut objective function and converge in finite iterations. Moreover, we have proposed efficient methods for adjust two regularization parameters. Extensive experimental results show the superior performance of the new method. Moreover, it is faster than the existing methods for solving the normalized cut.

JGSED: An End-to-End Spectral Clustering Model for Joint Graph Construction, Spectral Embedding and Discretization

Most of the existing graph-based clustering models performed clustering by adopting a two-stage strategy which first completes the spectral embedding from a given fixed graph and then resorts to other clustering methods such as <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> means to achieve discrete cluster results. On one hand, such a discretization operation easily causes that the obtained results deviate far from the true solution. On the other hand, clustering performance heavily relies on the quality of graph; therefore, the fixed graph is usually not optimal enough. In addition, clustering by separated steps inevitably breaks the underlying connections among the graph construction, spectral embedding and discretization. To address these drawbacks, in this paper, we propose a new spectral clustering model termed JGSED which integrates the graph construction, spectral embedding and spectral rotation together into a unified objective. JGSED is an end-to-end framework to directly take data as input and output the final binary cluster indicator matrix. An efficient algorithm is proposed to optimize the model variables in JGSED, which can be co-evolved towards the optimum. Extensive experiments are conducted on both synthetic and real data sets and the results demonstrate that JGSED outperforms the other state-of-the-art spectral clustering models, indicating the effectiveness of joint optimization.

Binary multi-view clustering with spectral embedding

Techniques for large-scale multi-view clustering have attracted increasing attention in recent years. To explore complementary and consensus information across multiple views, mapping multi-view data into a hamming space for binary multi-view learning is highly effective. However, for binary multi-view clustering, it is difficult to adopt the local manifold structure of original data into the consensus representation. Therefore, a novel algorithm named Binary Multi-View Clustering With Spectral Embedding (BMVC_SE) is proposed in this study. This algorithm unifies binary consensus representation learning and multi-view spectral learning into a joint framework. Specifically, this algorithm exploits the shareable and individual information across multi-view data to learn a binary consensus representation. Furthermore, spectral rotation technique is employed to embed manifold information from original multi-view data into binary consensus representation. Note that a multi-view GPI algorithm is proposed in this study to obtain the better local manifold information. We also provide an effective optimization method to solve the existing method. Extensive experimental results show that our method outperforms some state-of-the-art methods.

Multiview Clustering by Consensus Spectral Rotation Fusion.

Multiview clustering (MVC) aims to partition data into different groups by taking full advantage of the complementary information from multiple views. Most existing MVC methods fuse information of multiple views at the raw data level. They may suffer from performance degradation due to the redundant information contained in the raw data. Graph learning-based methods often heavily depend on one specific graph construction, which limits their practical applications. Moreover, they often require a computational complexity of O(n3) because of matrix inversion or eigenvalue decomposition for each iterative computation. In this paper, we propose a consensus spectral rotation fusion (CSRF) method to learn a fused affinity matrix for MVC at the spectral embedding feature level. Specifically, we first introduce a CSRF model to learn a consensus low-dimensional embedding, which explores the complementary and consistent information across multiple views. We develop an alternating iterative optimization algorithm to solve the CSRF optimization problem, where a computational complexity of O(n2) is required during each iterative computation. Then, the sparsity policy is introduced to design two different graph construction schemes, which are effectively integrated with the CSRF model. Finally, a multiview fused affinity matrix is constructed from the consensus low-dimensional embedding in spectral embedding space. We analyze the convergence of the alternating iterative optimization algorithm and provide an extension of CSRF for incomplete MVC. Extensive experiments on multiview datasets demonstrate the effectiveness and efficiency of the proposed CSRF method.

One step multi-view spectral clustering via joint adaptive graph learning and matrix factorization

Multi-view clustering based on graph learning has attracted extensive attention due to its simplicity and efficiency in recent years. However, there are still some issues in most of the existing graph-based multi-view clustering methods. First, most of those methods require post-processing such as K-means or spectral rotation to get the final discrete clustering result. Second, graph-based clustering methods perform clustering on a fixed input similarity graph, which could induce bad clustering results if the initial graph is with low quality. Third, these methods have high computation cost, which hinders them for dealing with large-scale data. In order to solve these problems, we propose a multi-view spectral clustering method via joint Adaptive Graph Learning and Matrix Factorization (AGLMF). In this method, to reduce computational cost, we adopt the anchor-based strategy to construct the input similarity graphs. Then, we use the l1-norm to learn a high quality similarity graph adaptively from original similarity graphs which can make the final graph more robust than original ones. In addition, AGLMF uses symmetric non-negative matrix factorization to learn the final clustering indicators which can show the final consistent clustering result directly. Finally, experimental results on multiple multi-view datasets validate the effectiveness of the proposed algorithm when compared with previous multi-view spectral clustering algorithms. The demo code of this work is publicly available at https://github.com/theywq/AGLMF.git.

Sample-level weights learning for multi-view clustering on spectral rotation

Multi-view clustering usually yields better results than single-view clustering since it utilizes more information from multi-view data. However, in the original multi-view data, some samples may be corrupted in partial views. Under this situation, the locations of the corrupted data are often unknown. But there is limited literature regarding this problem. Moreover, most existing multi-view spectral clustering methods need a post-processed algorithm k-means after obtaining the partition matrix, which leads to deviations in the clustering results. To resolve these problems, we propose a multi-view spectral clustering method named sample-level weights learning for Multi-view Clustering on Spectral Rotation (SR-MC) in this paper. By learning the weights in the sample level, SR-MC can make full use of the helpful complementary information among different views while reducing the effects of low-quality data for each sample. Therefore, it can deal with various multi-view clustering scenarios such as data are complete or corrupted in partial views. To reduce the deviations of the clustering results, a joint framework is designed for combining the learning of the consensus Laplacian matrix, the real-valued partition matrix and the binary indicator matrix together. The objective function of SR-MC can be efficiently optimized by an alternative optimization algorithm. Compared with other ten baselines, experiments on seven datasets show the superiority of our method.

An improved multi-view spectral clustering based on tissue-like P systems

Multi-view spectral clustering is one of the multi-view clustering methods widely studied by numerous scholars. The first step of multi-view spectral clustering is to construct the similarity matrix of each view. Consequently, the clustering performance will be greatly affected by the quality of the similarity matrix of each view. To solve this problem well, an improved multi-view spectral clustering based on tissue-like P systems is proposed in this paper. The optimal per-view similarity matrix is generated in an iterative manner. In addition, spectral clustering is combined with the symmetric nonnegative matrix factorization method to directly output the clustering results to avoid the secondary operation, such as k-means or spectral rotation. Furthermore, improved multi-view spectral clustering is integrated with the tissue-like P system to enhance the computational efficiency of the multi-view clustering algorithm. Extensive experiments verify the effectiveness of this algorithm over other state-of-the-art algorithms.

Research on construction method and validity mechanism of robust analysis model in snow peach quality detection based on visible-near infrared spectroscopy.

Visible-near infrared (Vis-NIR) spectra analysis method is widely used in the quality grading of bulk fruits with its rapid, non-destructive, diverse detection modes and flexible modular integration scheme. However, during the online grading of fruits, the random mechanized way of dropping fruit onto the conveyor belt method and the open detection environment led to more spectral abnormal samples, which affect the accuracy of the detection. In this paper, the soluble solids content (SSC) of snow peach is quantitatively analyzed by static and online detection methods. Several spectral preprocessing methods including Norris-Williams Smoothing (NWS), Savitzky-Golay Smoothing (SGS), Continuous Wavelet Derivative (CWD), Multivariate Scattering Correction (MSC), and Variable Sorting for Normalization (VSN) are adopted to eliminate spectral rotation and translation errors and improve the signal-to-noise ratio. Monte Carlo Uninformative Variable Elimination (MCUVE) method is used for the selection of optimal characteristic modeling variables. Partial Least Squares Regression (PLSR) is used to model and analyze the preprocessed spectra and the spectral variables optimized by MCUVE, and the effectiveness of the method is evaluated. Sparse Partial Least Squares Regression (SPLSR) and Sparse Partial Robust M Regression (SPRMR) are used for the construction of robust models. The results showed that the SGS preprocessing method can effectively improve the analysis accuracy of static and online models. The MCUVE method can realize the extraction of stable characteristic variables. The SPRMR model based on SGS preprocessing method and the effective variables has the optimal analysis results. The analysis accuracy of snow peach static model is slightly better than that of online analytical model. Through the test results of the PLSR, SPLSR and SPRMR models by the artificially adding noise test method, it can be seen that the SPRMR method eliminates the influence of abnormal samples on the model during the modeling process, which can effectively improve the anti-noise ability and detection reliability.

One-step incomplete multiview clustering with low-rank tensor graph learning

Incomplete multiview clustering (IMVC) has attracted extensive attention in the field of machine learning due to its excellent performance in handling incomplete multiview data (IMVD). However, existing IMVC methods often use spectral clustering or k-means on the obtained affinity matrices to obtain the final clustering results, that is, the affinity matrices learning process and clustering process are separated. In this paper, we propose a new ingenious IMVC method, one-step incomplete multiview clustering with low-rank tensor graph learning (OIMVC/LTGL), to solve the problem. This method combines graph learning, low-rank tensor constraint, common representation learning and clustering into a unified optimization framework. First, we utilize the adjacency relationship between views to complete the similarity graph matrices. Second, we stack all similarity graph matrices into a third-order tensor. To mine higher-order correlations among different views, we impose a low-rank tensor constraint, the tensor nuclear norm (TNN), into the constructed tensor. Third, to explore consistent information between views, we introduce a common representation learning term to learn the optimal consensus representation. Last, we introduce spectral rotation to the consensus representation matrices to directly obtain the clustering labels. These steps promote each other for better clustering performance. We solve the optimization problem by using the augmented Lagrange multiplier (ALM) method. Experiments based on five well-known datasets show the superiority of our method.

A novel method for optimizing spectral rotation embedding K-means with coordinate descent

Lloyd’s heuristic K-Means, one of the most widely used clustering methods, plays a vital role in a variety of downstream tasks in machine learning owing to its simplicity. However, Lloyd’s heuristic sometimes performs poorly in finding the local minimum and is heavily influenced by the initial points. To address these issues, we propose a novel optimization method for the K-Means model. First, we establish that the K-Means minimization problem can be reformulated as a trace maximization problem, which can be seen as a unified view of spectral clustering. Then, we relax the constraint of the scaled cluster matrix and implement an improved spectral rotation to bring the cluster matrix infinitely close to the binary indicator matrix. To this end, an efficient and redundancy-free coordinate descent (CD) method is used to optimize the spectral rotation. Extensive experiments including a hybrid test on several different datasets showed that the proposed algorithm achieved better local objective values compared to Lloyd’s heuristic under different initialization strategies (random or K-Means++). In the hybrid test, the proposed algorithm could further decrease the convergence value of the objective function obtained by Lloyd’s heuristic; conversely, Lloyd’s heuristic did not work. Moreover, statistical hypothesis and comparison tests further validated the superiority of the proposed algorithm.

FGC_SS: Fast Graph Clustering Method by Joint Spectral Embedding and Improved Spectral Rotation

Spectral clustering, one of the most popular clustering methods, has attracted considerable attention in many fields owing to its excellent empirical properties. However, previously developed solutions to spectral clustering problems consist of two successive steps: (1) performing spectral embedding and eigendecomposition to obtain relaxed partition matrix; (2) performing k-means or spectral rotation to obtain the final binary indicator matrix. However, these two steps may miss a co-optimal solution, leading to unreasonable results. Furthermore, the implementation of eigendecomposition is significantly time-consuming and requires a computational complexity of 0(n3). To address these issues, in this study, we propose a new framework for jointly performing spectral embedding and improving spectral rotation using an anchor-based acceleration strategy. In addition, we replace the spectral rotation term with a more mathematically rigorous form. To effectively solve the proposed model, we iteratively use the generalized power iteration method and the improved version of the coordinate descent method to solve our unified framework directly. Extensive experimental results on real benchmark datasets validate the effectiveness of the proposed algorithm compared with other state-of-the-art spectral-based methods. The proposed algorithm achieves the best performance on six-eighths of small-scale datasets, five-sixths of large-scale datasets, and three-fourths of high-dimensional datasets, while maintaining superior implementation efficiency.

Directly solving normalized cut for multi-view data

Graph-based multi-view clustering, which aims to uncover clusters from multi-view data with graph clustering technique, is one of the most important multi-view clustering methods. Such methods usually perform eigen-decomposition first to solve the relaxed problem and then obtain the final cluster indicator matrix from eigenvectors by k-means or spectral rotation. However, such a two-step process may result in undesired clustering result since the two steps aim to solve different problems. In this paper, we propose a k-way normalized cut method for multi-view data, named as the Multi-view Discrete Normalized Cut (MDNC). The new method learns a set of implicit weights for each view to identify its quality, and a novel iterative algorithm is proposed to directly solve the new model without relaxation and post-processing. Moreover, we propose a new method to adjust the distribution of the implicit view weights to obtain better clustering result. Extensive experimental results show that the performance of our approach is superior to the state-of-the-art methods.

Consensus graph and spectral representation for one-step multi-view kernel based clustering

Recently, multi-view clustering has received much attention in the fields of machine learning and pattern recognition. Spectral clustering for single and multiple views has been the common solution. Despite its good clustering performance, it has a major limitation: it requires an extra step of clustering. This extra step, which could be the famous k-means clustering, depends heavily on initialization, which may affect the quality of the clustering result. To overcome this problem, a new method called Multi-view Clustering via Consensus Graph Learning and Nonnegative Embedding (MVCGE) is presented in this paper. In the proposed approach, the consensus affinity matrix (graph matrix), consensus representation and cluster index matrix (nonnegative embedding) are learned simultaneously in a unified framework. Our proposed method takes as input the different kernel matrices corresponding to the different views. The proposed learning model integrates two interesting constraints: (i) the cluster indices should be as smooth as possible over the consensus graph and (ii) the cluster indices are set to be as close as possible to the graph convolution of the consensus representation. In this approach, no post-processing such as k-means or spectral rotation is required. Our approach is tested with real and synthetic datasets. The experiments performed show that the proposed method performs well compared to many state-of-the-art approaches.

Multiple partitions alignment via spectral rotation

Multi-view spectral clustering has drawn much attention due to the effectiveness of exploiting the similarity relationships among data points. These methods typically reveal the intrinsic structure using a predefined graph for each view. The predefined graphs are fused to a consensus one, on which the final clustering results are obtained. However, such common strategies may lead to information loss because of the inconsistency or noise among multiple views. In this paper, we propose to merge multi-view information in partition level instead of the raw feature space where the data points lie. The partition of each view is treated as a perturbation of the consensus clustering, and the multiple partitions are integrated by estimating a distinct rotation for each partition. The proposed model is formulated as a joint learning framework, i.e., with the input data matrix, our model directly outputs the final discrete clustering result. Hence it is an end-to-end single-stage learning model. An iterative updating algorithm is proposed to solve the learning problem, in which the involved variables can be optimized in a mutual reinforcement manner. Experimental results on real-world data sets illustrate the effectiveness of our model.