Spectral Embedding Research Articles

Valid two‐sample graph testing via optimal transport Procrustes and multiscale graph correlation with applications in connectomics

Testing whether two graphs come from the same distribution is of interest in many real‐world scenarios, including brain network analysis. Under the random dot product graph model, the nonparametric hypothesis testing framework consists of embedding the graphs using the adjacency spectral embedding (ASE), followed by aligning the embeddings using the median flip heuristic and finally applying the nonparametric maximum mean discrepancy (MMD) test to obtain a p value. Using synthetic data generated from Drosophila brain networks, we show that the median flip heuristic results in an invalid test and demonstrate that optimal transport Procrustes (OTP) for alignment resolves the invalidity. We further demonstrate that substituting the MMD test with the multiscale graph correlation (MGC) test leads to a more powerful test both in synthetic and in simulated data. Lastly, we apply this powerful test to the right and left hemispheres of the larval Drosophila mushroom body brain networks and conclude that there is not sufficient evidence to reject the null hypothesis that the two hemispheres are equally distributed.

Chaoticity versus stochasticity in financial markets: Are daily S&P 500 return dynamics chaotic?

In this paper, we empirically show the dynamics of daily wavelet-filtered (denoised) S&P 500 returns (2000–2020) to consist of an almost equally divided combination of stochastic and deterministic chaos, rendering the series unpredictable after expiration of the Lyapunov time, resulting in futile forecasting attempts. We observe a clear distinction of the nature of the underlying time series dynamics by applying a novel and combinatory chaos analysis framework by comparing the wavelet-filtered S&P 500 returns with surrogate datasets, Brownian motion returns, and a Lorenz system realisation. Furthermore, we are the first to observe the strange attractor of the daily S&P 500 return system graphically via Takens’ embedding and a spectral embedding in combination with Laplacian eigenmaps. Finally, we critically discuss the implications and future prospects of financial forecasting.

Rare event estimation using stochastic spectral embedding

Estimating the probability of rare failure events is an essential step in the reliability assessment of engineering systems. Computing this failure probability for complex non-linear systems is challenging, and has recently spurred the development of active-learning reliability methods. These methods approximate the limit-state function (LSF) using surrogate models trained with a sequentially enriched set of model evaluations. A recently proposed method called stochastic spectral embedding (SSE) aims to improve the local approximation accuracy of global, spectral surrogate modelling techniques by sequentially embedding local residual expansions in subdomains of the input space. In this work we apply SSE to the LSF, giving rise to a stochastic spectral embedding-based reliability (SSER) method. The resulting partition of the input space decomposes the failure probability into a set of easy-to-compute conditional failure probabilities. We propose a set of modifications that tailor the algorithm to efficiently solve rare event estimation problems. These modifications include specialized refinement domain selection, partitioning and enrichment strategies. We showcase the algorithm performance on four benchmark problems of various dimensionality and complexity in the LSF.

Spectral Clustering on Spherical Coordinates Under the Degree-Corrected Stochastic Blockmodel

ABSTRACT Spectral clustering is a popular method for community detection in network graphs: starting from a matrix representation of the graph, the nodes are clustered on a low-dimensional projection obtained from a truncated spectral decomposition of the matrix. Estimating correctly the number of communities and the dimension of the reduced latent space is critical for good performance of spectral clustering algorithms. Furthermore, many real-world graphs, such as enterprise computer networks studied in cyber-security applications, often display heterogeneous within-community degree distributions. Such heterogeneous degree distributions are usually not well captured by standard spectral clustering algorithms. In this article, a novel spectral clustering algorithm is proposed for community detection under the degree-corrected stochastic blockmodel. The proposed method is based on a transformation of the spectral embedding to spherical coordinates, and a novel modeling assumption in the transformed space. The method allows for simultaneous and automated selection of the number of communities and the latent dimension for spectral embeddings of graphs with uneven node degrees. Results show improved performance over competing methods in representing computer networks.

One-Step Adaptive Spectral Clustering Networks

Deep spectral clustering is a popular and efficient algorithm in unsupervised learning. However, deep spectral clustering methods are organized into three separate steps: affinity matrix learning, spectral embedding learning, and K-means clustering on spectral embedding. In this case, although each step can achieve its own performance, it is still difficult to obtain robust clustering results. In this letter, we propose a one-step adaptive spectral clustering network to overcome the aforementioned shortcomings. The network embeds the three parts of affinity matrix learning, spectral embedding learning, and indicator learning into a unified framework. The affinity matrix is adaptively adjusted by spectral embedding in a deep subspace. We introduce spectral rotation to discretize spectral embedding, which makes the spectral embedding and indicator be learned simultaneously to improve clustering quality. Each part of the model can be iteratively updated based on other parts to optimize the clustering results. Experimental results on four real datasets show the effectiveness of our method on the ACC and NMI clustering evaluation metrics. In particular, our method achieves an NMI of 0.932 and an ACC of 0.973 on the MNIST dataset, a decent performance boost compared to the best baseline.

Fast Multi-view Clustering via Prototype Graph

Multi-view clustering attracts considerable attention due to its effectiveness in unsupervised learning. However, previous multi-view spectral clustering methods include two separated steps: 1) Obtaining a spectral embedding; 2) Performing classical clustering methods. Although these methods have achieved promising performance, there is still some limitations. First, in computing spectral embedding, multi-view spectral clustering approaches exist high computational complexity since they usually need eigenvalue decomposition on laplacian matrix L. Its computational complexity is O(n³) where n is the number of samples; Second, in constructing similarity matrices, previous methods need to compute similarity between any two samples; Third, the two-stage approach only can obtain sub-optimal solution; Fourth, treating equally all views is unreasonable. To address these issues, we propose a Fast Multi-view Clustering via Prototype Graph (FMVPG) method. Specifically, the prototype graph is firstly constructed, and then simultaneously perform spectral embedding to obtain the real matrix and spectral rotation to get the indicator matrix. In addition, the Alternating Direction Method of Multipliers (ADMM) is used to solve the joint optimization problem. Further, we conduct extensive experiments to evaluate the proposed FMVPG approach. These experimental results show the comparable or even better clustering performance than the state-of-the-art approaches.

Fast Multiview Clustering With Spectral Embedding.

Spectral clustering has been a hot topic in unsupervised learning owing to its remarkable clustering effectiveness and well-defined framework. Despite this, due to its high computation complexity, it is unable of handling large-scale or high-dimensional data, particularly multi-view large-scale data. To address this issue, in this paper, we propose a fast multi-view clustering algorithm with spectral embedding (FMCSE), which speeds up both the spectral embedding and spectral analysis stages of multi-view spectral clustering. Furthermore, unlike conventional spectral clustering, FMCSE can acquire all sample categories directly after optimization without extra k-means, which can significantly enhance efficiency. Moreover, we also provide a fast optimization strategy for solving the FMCSE model, which divides the optimization problem into three decoupled small-scale sub-problems that can be solved in a few iteration steps. Finally, extensive experiments on a variety of real-world datasets (including large-scale and high-dimensional datasets) show that, when compared to other state-of-the-art fast multi-view clustering baselines, FMCSE can maintain comparable or even better clustering effectiveness while significantly improving clustering efficiency.

Spectral graph clustering via the expectation-solution algorithm

The stochastic blockmodel (SBM) models the connectivity within and between disjoint subsets of nodes in networks. Prior work demonstrated that the rows of an SBM’s adjacency spectral embedding (ASE) and Laplacian spectral embedding (LSE) both converge in law to Gaussian mixtures where the components are curved exponential families. Maximum likelihood estimation via the Expectation-Maximization (EM) algorithm for a full Gaussian mixture model (GMM) can then perform the task of clustering graph nodes, albeit without appealing to the components’ curvature. Noting that EM is a special case of the Expectation-Solution (ES) algorithm, we propose two ES algorithms that allow us to take full advantage of these curved structures. After presenting the ES algorithm for the general curved-Gaussian mixture, we develop those corresponding to the ASE and LSE limiting distributions. Simulating from artificial SBMs and a brain connectome SBM reveals that clustering graph nodes via our ES algorithms can improve upon that of EM for a full GMM for a wide range of settings.

Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition

Despite the rapid progress of automatic speech recognition (ASR) technologies targeting normal speech in recent decades, accurate recognition of dysarthric and elderly speech remains highly challenging tasks to date. Sources of heterogeneity commonly found in normal speech including accent or gender, when further compounded with the variability over age and speech pathology severity level, create large diversity among speakers. To this end, speaker adaptation techniques play a key role in personalization of ASR systems for such users. Motivated by the spectro-temporal level differences between dysarthric, elderly and normal speech that systematically manifest in articulatory imprecision, decreased volume and clarity, slower speaking rates and increased dysfluencies, novel spectro-temporal subspace basis deep embedding features derived using SVD speech spectrum decomposition are proposed in this paper to facilitate auxiliary feature based speaker adaptation of state-of-the-art hybrid DNN/TDNN and end-to-end Conformer speech recognition systems. Experiments were conducted on four tasks: the English UASpeech and TORGO dysarthric speech corpora; the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets. The proposed spectro-temporal deep feature adapted systems outperformed baseline i-Vector and x-Vector adaptation by up to 2.63% absolute (8.63% relative) reduction in word error rate (WER). Consistent performance improvements were retained after model based speaker adaptation using learning hidden unit contributions (LHUC) was further applied. The best speaker adapted system using the proposed spectral basis embedding features produced the lowest published WER of 25.05% on the UASpeech test set of 16 dysarthric speakers.

Online Change Point Detection for Weighted and Directed Random Dot Product Graphs

Given a sequence of random (directed and weighted) graphs, we address the problem of online monitoring and detection of changes in the underlying data distribution. Our idea is to endow sequential change-point detection (CPD) techniques with a graph representation learning substrate based on the versatile Random Dot Product Graph (RDPG) model. We consider efficient, online updates of a judicious monitoring function, which quantifies the discrepancy between the streaming graph observations and the nominal RDPG. This reference distribution is inferred via spectral embeddings of the first few graphs in the sequence. We characterize the distribution of this running statistic to select thresholds that guarantee error-rate control, and under simplifying approximations we offer insights on the algorithm’s detection resolution and delay. The end result is a lightweight online CPD algorithm, that is also explainable by virtue of the well-appreciated interpretability of RDPG embeddings. This is in stark contrast with most existing graph CPD approaches, which either rely on extensive computation, or they store and process the entire observed time series. An apparent limitation of the RDPG model is its suitability for undirected and unweighted graphs only, a gap we aim to close here to broaden the scope of the CPD framework. Unlike previous proposals, our non-parametric RDPG model for weighted graphs does not require a priori specification of the weights’ distribution to perform inference and estimation. This network modeling contribution is of independent interest beyond CPD. We offer an open-source implementation of the novel online CPD algorithm for weighted and direct graphs, whose effectiveness and efficiency are demonstrated via (reproducible) synthetic and real network data experiments.

CSiT: A Multiscale Vision Transformer for Hyperspectral Image Classification

The hyperspectral image (HSI) has nearly continuous spectral information, thus, the target of interest can be accurately identified by the subtle details of spectral properties. Spectral resolution at different scales can capture different levels of spectral features: small-scale spectral bands are beneficial for extracting global details in vision transformers, while large-scale spectral bands are more effective for local features. Transformer shows advantages in global information extraction with self-attention module and even surpasses CNNs in various tasks. Some works based on the vision transformer have performed surprisingly in HSI classification. However, single-scale vision transformers are insufficient to balance the extraction of local details and redundancy on different scales. The recent work, a multi-scale vision transformer, has provided a solution with spatial patch-wise features in image classification. Inspired by this, we propose the Cross-spectral vision transformer (CSiT) with two branches to extract pixel-wise multi-scale features and further design a multi-scale spectral embedding module to enhance local details between neighboring spectral bands. Moreover, based on the cross-attention operation, a single token for each branch is recognized as a query and used to exchange information with other branches. We evaluate the classification performance of the proposed CSiT in three classic HSI datasets with extensive experiments, showing the multi-scale vision transformer architecture has a promising result for HSI classification with one-dimensional spectral bands.

Effective Clustering via Structured Graph Learning

Given an affinity graph of data samples, graph-based clustering aims to partition these samples into disjoint groups based on the affinities, and most previous works are based on spectral clustering. However, two problems among spectral-based methods heavily affect the clustering performance. Firstly, the randomness of post-processing procedures, such as <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -means, affects the stability of clustering. Secondly, the separated stages of spectral-based methods, including graph construction, spectral embedding learning, and clustering decision, lead to mismatched problems. In this paper, we explore a structured graph learning (SGL) framework that aims to fuse these stages to improve clustering stability. Specifically, SGL adaptively learns a structured affinity graph that contains exact <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> connected components. Each connected component corresponds to a cluster so clustering assignments can be directly obtained according to the connectivity of the learned graph. In this way, SGL avoids the randomness brought by reliance on traditional post-processing procedures. Meanwhile, the graph construction and structured graph learning procedures happen simultaneously, which alleviates the mismatched problem effectively. Moreover, we propose an efficient algorithm to solve the involved optimization problems and discuss the connections between this work and previous works. Numerical experiments on several synthetic and real datasets demonstrate the effectiveness of our methods.

SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers

Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Due to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers and propose a novel backbone network called <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SpectralFormer</u> . Beyond bandwise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding groupwise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layerwise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse “soft” residuals across layers. It is worth noting that the proposed SpectralFormer is a highly flexible backbone network, which can be applicable to both pixelwise and patchwise inputs. We evaluate the classification performance of the proposed SpectralFormer on three HS datasets by conducting extensive experiments, showing the superiority over classic transformers and achieving a significant improvement in comparison with state-of-the-art backbone networks. The codes of this work will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/danfenghong/IEEE_TGRS_SpectralFormer</uri> for the sake of reproducibility.

Robust Matrix Factorization With Spectral Embedding.

Nonnegative matrix factorization (NMF) and spectral clustering are two of the most widely used clustering techniques. However, NMF cannot deal with the nonlinear data, and spectral clustering relies on the postprocessing. In this article, we propose a Robust Matrix factorization with Spectral embedding (RMS) approach for data clustering, which inherits the advantages of NMF and spectral clustering, while avoiding their shortcomings. In addition, to cluster the data represented by multiple views, we present the multiview version of RMS (M-RMS), and the weights of different views are self-tuned. The main contributions of this research are threefold: 1) by integrating spectral clustering and matrix factorization, the proposed methods are able to capture the nonlinear data structure and obtain the cluster indicator directly; 2) instead of using the squared Frobenius-norm, the objectives are developed with the l2,1 -norm, such that the effects of the outliers are alleviated; and 3) the proposed methods are totally parameter-free, which increases the applicability for various real-world problems. Extensive experiments on several single-view/multiview data sets demonstrate the effectiveness of our methods and verify their superior clustering performance over the state of the arts.

Distance Preserving Model Order Reduction of Graph-Laplacians and Cluster Analysis

Graph-Laplacians and their spectral embeddings play an important role in multiple areas of machine learning. This paper is focused on graph-Laplacian dimension reduction for the spectral clustering of data as a primary application, however, it can also be applied in data mining, data manifold learning, etc. Spectral embedding provides a low-dimensional parametrization of the data manifold which makes the subsequent task (e.g., clustering with k-means or any of its approximations) much easier. However, despite reducing the dimensionality of data, the overall computational cost may still be prohibitive for large data sets due to two factors. First, computing the partial eigendecomposition of the graph-Laplacian typically requires a large Krylov subspace. Second, after the spectral embedding is complete, one still has to operate with the same number of data points, which may ruin the efficiency of the approach. For example, clustering of the embedded data is typically performed with various relaxations of k-means which computational cost scales poorly with respect to the size of data set. Also, they become prone to getting stuck in local minima, so their robustness depends on the choice of initial guess. In this work, we switch the focus from the entire data set to a subset of graph vertices (target subset). We develop two novel algorithms for such low-dimensional representation of the original graph that preserves important global distances between the nodes of the target subset. In particular, it allows to ensure that target subset clustering is consistent with the spectral clustering of the full data set if one would perform such. That is achieved by a properly parametrized reduced-order model (ROM) of the graph-Laplacian that approximates accurately the diffusion transfer function of the original graph for inputs and outputs restricted to the target subset. Working with a small target subset reduces greatly the required dimension of Krylov subspace and allows to exploit the conventional algorithms (like approximations of k-means) in the regimes when they are most robust and efficient. This was verified in the numerical clustering experiments with both synthetic and real data. We also note that our ROM approach can be applied in a purely transfer-function-data-driven way, so it becomes the only feasible option for extremely large graphs that are not directly accessible. There are several uses for our algorithms. First, they can be employed on their own for representative subset clustering in cases when handling the full graph is either infeasible or simply not required. Second, they may be used for quality control. Third, as they drastically reduce the problem size, they enable the application of more sophisticated algorithms for the task under consideration (like more powerful approximations of k-means based on semi-definite programming (SDP) instead of the conventional Lloyd’s algorithm). Finally, they can be used as building blocks of a multi-level divide-and-conquer type algorithm to handle the full graph. The latter will be reported in a separate article.

MS2DeepScore: a novel deep learning similarity measure to compare tandem mass spectra

Mass spectrometry data is one of the key sources of information in many workflows in medicine and across the life sciences. Mass fragmentation spectra are generally considered to be characteristic signatures of the chemical compound they originate from, yet the chemical structure itself usually cannot be easily deduced from the spectrum. Often, spectral similarity measures are used as a proxy for structural similarity but this approach is strongly limited by a generally poor correlation between both metrics. Here, we propose MS2DeepScore: a novel Siamese neural network to predict the structural similarity between two chemical structures solely based on their MS/MS fragmentation spectra. Using a cleaned dataset of > 100,000 mass spectra of about 15,000 unique known compounds, we trained MS2DeepScore to predict structural similarity scores for spectrum pairs with high accuracy. In addition, sampling different model varieties through Monte-Carlo Dropout is used to further improve the predictions and assess the model’s prediction uncertainty. On 3600 spectra of 500 unseen compounds, MS2DeepScore is able to identify highly-reliable structural matches and to predict Tanimoto scores for pairs of molecules based on their fragment spectra with a root mean squared error of about 0.15. Furthermore, the prediction uncertainty estimate can be used to select a subset of predictions with a root mean squared error of about 0.1. Furthermore, we demonstrate that MS2DeepScore outperforms classical spectral similarity measures in retrieving chemically related compound pairs from large mass spectral datasets, thereby illustrating its potential for spectral library matching. Finally, MS2DeepScore can also be used to create chemically meaningful mass spectral embeddings that could be used to cluster large numbers of spectra. Added to the recently introduced unsupervised Spec2Vec metric, we believe that machine learning-supported mass spectral similarity measures have great potential for a range of metabolomics data processing pipelines.

Connectivity gradients on tractography data: Pipeline and example applications.

Gray matter connectivity can be described in terms of its topographical organization, but the differential role of white matter connections underlying that organization is often unknown. In this study, we propose a method for unveiling principles of organization of both gray and white matter based on white matter connectivity as assessed using diffusion magnetic ressonance imaging (MRI) tractography with spectral embedding gradient mapping. A key feature of the proposed approach is its capacity to project the individual connectivity gradients it reveals back onto its input data in the form of projection images, allowing one to assess the contributions of specific white matter tracts to the observed gradients. We demonstrate the ability of our proposed pipeline to identify connectivity gradients in prefrontal and occipital gray matter. Finally, leveraging the use of tractography, we demonstrate that it is possible to observe gradients within the white matter bundles themselves. Together, the proposed framework presents a generalized way to assess both the topographical organization of structural brain connectivity and the anatomical features driving it.

Automated facial-vestibulocochlear nerve complex identification based on data-driven tractography clustering.

Small size and intricate anatomical environment are the main difficulties facing tractography of the facial-vestibulocochlear nerve complex (FVN), and lead to challenges in fiber orientation distribution (FOD) modeling, fiber tracking, region-of-interest selection, and fiber filtering. Experts need rich experience in anatomy and tractography, as well as substantial labor costs, to identify the FVN. Thus, we present a pipeline to identify the FVN automatically, in what we believe is the first study of the automated identification of the FVN. First, we created an FVN template. Forty high-resolution multishell data were used to perform data-driven fiber clustering based on the multishell multitissue constraint spherical deconvolution FOD model and deterministic tractography. We selected the brainstem and cerebellum (BS-CB) region as the seed region and removed the fibers that reach other brain regions. We then performed spectral fiber clustering twice. The first clustering was to create a BS-CB atlas and separate the fibers that pass through the cerebellopontine angle, and the other one was to extract the FVN. Second, we registered the subject-specific fibers in the space of the FVN template and assigned each fiber to the closest cluster to identify the FVN automatically by spectral embedding. We applied the proposed method to different acquirement sites, including two different healthy datasets and two tumor patient datasets. Experimental results showed that our automatic identification results have ideal colocalization with expert manual identification in terms of spatial overlap and visualization. Importantly, we successfully applied our method to tumor patient data. The FVNs identified by the proposed method were in agreement with intraoperative findings.

Robust Graph Factorization for Multivariate Electricity Consumption Series Clustering

Multivariate electricity consumption series clustering can reflect trends of power consumption changes in the past time period, which can provide reliable guidance for electricity production. However, there are some abnormal series in the past multivariate electricity consumption series data, while outliers will affect the discovery of electricity consumption trends in different time periods. To address this problem, we propose a robust graph factorization model for multivariate electricity consumption clustering (RGF-MEC), which performs graph factorization and outlier discovery simultaneously. RGF-MEC first obtains a similarity graph by calculating distance among multivariate electricity consumption series data and then performs robust matrix factorization on the similarity graph. Meanwhile, the similarity graph is decomposed into a class-related embedding and a spectral embedding, where the class-related embedding directly reveals the final clustering results. Experimental results on realistic multivariate time-series datasets and multivariate electricity consumption series datasets demonstrate effectiveness of the proposed RGF-MEC model.

Efficient Estimation for Random Dot Product Graphs via a One-Step Procedure

We propose a one-step procedure to estimate the latent positions in random dot product graphs efficiently. Unlike the classical spectral-based methods, the proposed one-step procedure takes advantage of both the low-rank structure of the expected adjacency matrix and the Bernoulli likelihood information of the sampling model simultaneously. We show that for each vertex, the corresponding row of the one-step estimator (OSE) converges to a multivariate normal distribution after proper scaling and centering up to an orthogonal transformation, with an efficient covariance matrix. The initial estimator for the one-step procedure needs to satisfy the so-called approximate linearization property. The OSE improves the commonly adopted spectral embedding methods in the following sense: Globally for all vertices, it yields an asymptotic sum of squares error no greater than those of the spectral methods, and locally for each vertex, the asymptotic covariance matrix of the corresponding row of the OSE dominates those of the spectral embeddings in spectra. The usefulness of the proposed one-step procedure is demonstrated via numerical examples and the analysis of a real-world Wikipedia graph dataset.