Sparsification Methods Research Articles

Active sparse Bayesian committee machine potential for isothermal-isobaric molecular dynamics simulations.

Recent advancements in machine learning potentials (MLPs) have significantly impacted the fields of chemistry, physics, and biology by enabling large-scale first-principles simulations. Among different machine learning approaches, kernel-based MLPs distinguish themselves through their ability to handle small datasets, quantify uncertainties, and minimize over-fitting. Nevertheless, their extensive computational requirements present considerable challenges. To alleviate these, sparsification methods have been developed, aiming to reduce computational scaling without compromising accuracy. In the context of isothermal and isobaric ML molecular dynamics (MD) simulations, achieving precise pressure estimation is crucial for reproducing reliable system behavior under constant pressure. Despite progress, sparse kernel MLPs struggle with precise pressure prediction. Here, we introduce a virial kernel function that significantly enhances the pressure estimation accuracy of MLPs. Additionally, we propose the active sparse Bayesian committee machine (BCM) potential, an on-the-fly MLP architecture that aggregates local sparse Gaussian process regression (SGPR) MLPs. The sparse BCM potential overcomes the steep computational scaling with the kernel size, and a predefined restriction on the size of kernel allows for fast and efficient on-the-fly training. Our advancements facilitate accurate and computationally efficient machine learning-enhanced MD (MLMD) simulations across diverse systems, including ice-liquid coexisting phases, Li10Ge(PS6)2 lithium solid electrolyte, and high-pressure liquid boron nitride.

A tabu search with geometry‐based sparsification methods for angular traveling salesman problems

AbstractThe angular‐metric traveling salesman problem (AngleTSP) aims to find a tour visiting a given set of vertices in the Euclidean plane exactly once while minimizing the cost given by the sum of all turning angles. If the cost is obtained by combining the sum of all turning angles and the traveled distance, the problem is called angular‐distance‐metric traveling salesman problem (AngleDistanceTSP). In this work, we study the symmetric variants of these problems. Because both the AngleTSP and AngleDistanceTSP are NP‐hard, multiple heuristic approaches have been proposed in the literature. Nevertheless, a good tradeoff between solution quality and runtime is hard to find. We propose a granular tabu search (GTS) that considers the geometric features of the two problems in the design of starting solutions and sparsification methods. We further enrich the GTS with components that guarantee both intensification and diversification during the search. The computational results on benchmark instances from the literature show that (i) for the AngleTSP, our GTS lies on the Pareto frontier of the best performing‐heuristics, and (ii) for the AngleDistanceTSP, our GTS provides the best solution quality across all existing heuristics in competitive runtimes. In addition, new best‐known solutions are found for most benchmark instances for which an optimal solution is not available.

Efficient and Secure Federated Learning for Financial Applications

Conventional machine learning (ML) and deep learning approaches require sharing customers’ sensitive information with an external credit bureau to generate a prediction model, thereby increasing the risk of privacy leakage. This poses a significant challenge for financial companies. To address this challenge, federated learning has emerged as a promising approach to protect data privacy. However, the high communication costs associated with federated systems, particularly for large neural networks, can be a bottleneck. To mitigate this issue, it is necessary to limit the number and size of communications for practical training of large neural structures. Gradient sparsification is a technique that has gained increasing attention as a method to reduce communication costs, as it updates only significant gradients and accumulates insignificant gradients locally. However, the secure aggregation framework cannot directly employ gradient sparsification. To overcome this limitation, this article proposes two sparsification methods for reducing the communication costs of federated learning. The first method is a time-varying hierarchical sparsification method for model parameter updates, which addresses the challenge of maintaining model accuracy after a high sparsity ratio. This method can significantly reduce the cost of a single communication. The second method is to apply sparsification to the secure aggregation framework. Specifically, the encryption mask matrix is sparsified to reduce communication costs while protecting privacy. Experiments demonstrate that our method can reduce the upload communication costs to approximately 2.9% to 18.9% of the conventional federated learning algorithm under different non-IID experiment settings when the sparsity rate is 0.01.

SparSFA: Towards robust and communication-efficient peer-to-peer federated learning

Federated Learning (FL) has emerged as a powerful paradigm to train collaborative machine learning (ML) models, preserving the privacy of the participants’ datasets. However, standard FL approaches present some limitations that can hinder their applicability in some applications. Thus, the need of a server or aggregator to orchestrate the learning process may not be possible in scenarios with limited connectivity, as in some IoT applications, and offer less flexibility to personalize the ML models for the different participants. To sidestep these limitations, peer-to-peer FL (P2PFL) provides more flexibility, allowing participants to train their own models in collaboration with their neighbors. However, given the huge number of parameters of typical Deep Neural Network architectures, the communication burden can also be very high. On the other side, it has been shown that standard aggregation schemes for FL are very brittle against data and model poisoning attacks. In this paper, we propose SparSFA, an algorithm for P2PFL capable of reducing the communication costs. We show that our method outperforms competing sparsification methods in P2P scenarios, speeding the convergence and enhancing the stability during training. SparSFA also includes a mechanism to mitigate poisoning attacks for each participant in any random network topology. Our empirical evaluation on real datasets for intrusion detection in IoT, considering both balanced and imbalanced-dataset scenarios, shows that SparSFA is robust to different indiscriminate poisoning attacks launched by one or multiple adversaries, outperforming other robust aggregation methods whilst reducing the communication costs through sparsification.

Approximating Matrix Eigenvalues by Subspace Iteration with Repeated Random Sparsification

Traditional numerical methods for calculating matrix eigenvalues are prohibitively expensive for high-dimensional problems. Iterative random sparsification methods allow for the estimation of a single dominant eigenvalue at reduced cost by leveraging repeated random sampling and averaging. We present a general approach to extending such methods for the estimation of multiple eigenvalues and demonstrate its performance for several benchmark problems in quantum chemistry.

Communication-Efficient Federated Learning Based on Compressed Sensing

In this article, we investigate the problem of federated learning (FL) in a communication-constrained environment of the Internet of Things (IoT), where multiple IoT clients train a global model collectively by communicating model updates with a central server instead of sending raw data sets. To ease the communication burden in IoT systems, several approaches have been proposed for the FL tasks, including sparsification methods and data quantization strategies. To overcome the shortcomings of the existing methods, we propose two new FL algorithms based on compressed sensing (CS) referred to as the CS-FL algorithm and the 1-bit CS-FL algorithm, both of which compress the upstream and downstream data while communicating between the clients and the central server. The proposed algorithms improve upon the existing algorithms by letting the clients send analog and 1-bit data, respectively, to the server after compression with a random measurement matrix. Based on that, in CS-FL and 1-bit CS-FL, the clients update the model locally utilizing the result of sparse reconstruction obtained by iterative hard thresholding (IHT) and binary IHT (BIHT), respectively. Experiments conducted on the MNIST and the Fashion-MNIST data sets reveal the superiority of the proposed algorithm over the baseline algorithms, SignSGD with a majority vote, FL based on sparse ternary compression, and FedAvg.

Exploiting Activation Sparsity for Fast CNN Inference on Mobile GPUs

Over the past several years, the need for on-device deep learning has been rapidly increasing, and efficient CNN inference on mobile platforms has been actively researched. Sparsity exploitation has been one of the most active research themes, but the studies mostly focus on weight sparsity by weight pruning. Activation sparsity, on the contrary, requires compression at runtime for every input tensor. Hence, the research on activation sparsity mainly targets NPUs that can efficiently process this with their own hardware logic. In this paper, we observe that it is difficult to accelerate CNN inference on mobile GPUs with natural activation sparsity and that the widely used CSR-based sparse convolution is not sufficiently effective due to the compression overhead. We propose several novel sparsification methods that can boost activation sparsity without harming accuracy. In particular, we selectively sparsify some layers with an extremely high sparsity and adopt sparse convolution or dense convolution depending on the layers. Further, we present an efficient sparse convolution method without compression and demonstrate that it can be faster than the CSR implementation. With ResNet-50, we achieved 1.88 speedup compared to TFLite on a Mali-G76 GPU.

RTop-k: A Statistical Estimation Approach to Distributed SGD

The large communication cost for exchanging gradients between different nodes significantly limits the scalability of distributed training for large-scale learning models. Motivated by this observation, there has been significant recent interest in techniques that reduce the communication cost of distributed Stochastic Gradient Descent (SGD), with gradient sparsification techniques such as top-k and random-k shown to be particularly effective. The same observation has also motivated a separate line of work in distributed statistical estimation theory focusing on the impact of communication constraints on the estimation efficiency of different statistical models. The primary goal of this paper is to connect these two research lines and demonstrate how statistical estimation models and their analysis can lead to new insights in the design of communication-efficient training techniques. We propose a simple statistical estimation model for the stochastic gradients which captures the sparsity and skewness of their distribution. The statistically optimal communication scheme arising from the analysis of this model leads to a new sparsification technique for SGD, which concatenates random-k and top-k, considered separately in the prior literature. We show through extensive experiments on both image and language domains with CIFAR-10, ImageNet, and Penn Treebank datasets that the concatenated application of these two sparsification methods consistently and significantly outperforms either method applied alone.

Multiclass classification of patients during different stages of Alzheimer’s disease using fMRI time-series

Alzheimer’s Disease (AD) begins several years before the symptoms develop. It starts with Mild Cognitive Impairment (MCI) which can be separated into Early MCI and Late MCI (EMCI and LMCI). Functional connectivity analysis and classification are done among the different stages of illness with Functional Magnetic Resonance Imaging (fMRI). In this study, in addition to the four stages including healthy, EMCI, LMCI, and AD, the patients have been tracked for a year. Indeed, the classification has been done among 7 groups to analyze the functional connectivity changes in one year in different stages. After generating the functional connectivity graphs for eliminating the weak links, three different sparsification methods were used. In addition to simple thresholding, spectral sparsification based on effective resistance and sparse autoencoder were performed in order to analyze the effect of sparsification routine on classification results. Also, instead of extracting common features, the correlation matrices were reshaped to a correlation vector and used as a feature vector to enter the classifier. Since the correlation matrix is symmetric, in another analysis half of the feature vector was used, moreover, the Genetic Algorithm (GA) also utilized for feature vector dimension reduction. The non-linear SVM classifier with a polynomial kernel applied. The results showed that the autoencoder sparsification method had the greatest discrimination power with the accuracy of 98.35% for classification when the feature vector was the full correlation matrix.

GRASS: Graph Spectral Sparsification Leveraging Scalable Spectral Perturbation Analysis

Spectral graph sparsification aims to find ultra-sparse subgraphs whose Laplacian matrix can well approximate the original Laplacian eigenvalues and eigenvectors. In recent years, spectral sparsification techniques have been extensively studied for accelerating various numerical and graph-related applications. Prior nearly-linear-time spectral sparsification methods first extract low-stretch spanning tree from the original graph to form the backbone of the sparsifier, and then recover small portions of spectrally-critical off-tree edges to the spanning tree to significantly improve the approximation quality. However, it is not clear how many off-tree edges should be recovered for achieving a desired spectral similarity level within the sparsifier. Motivated by recent graph signal processing techniques, this work proposes a similarity-aware spectral graph sparsification framework that leverages efficient spectral off-tree edge embedding and filtering schemes to construct spectral sparsifiers with guaranteed spectral similarity (relative condition number) level. An iterative graph densification scheme is also introduced to facilitate efficient and effective filtering of off-tree edges for highly ill-conditioned problems. The proposed method has been validated using various kinds of graphs obtained from public domain sparse matrix collections relevant to very large-scale integration computer-aided design, finite element analysis, as well as social and data networks frequently studied in many machine learning and data mining applications. For instance, a sparse SDD matrix with 40 million unknowns and 180 million nonzeros can be solved (1E-3 accuracy level) within 2 min using a single CPU core and about 6-GB memory.

Least-squares policy iteration algorithms for robotics: Online, continuous, and automatic

Reinforcement learning (RL) is a general framework to acquire intelligent behavior by trial-and-error and many successful applications and impressive results have been reported in the field of robotics. In robot control problem settings, it is oftentimes characteristic that the algorithms have to learn online through interaction with the system while it is operating, and that both state and action spaces are continuous. Least-squares policy iteration (LSPI) based approaches are therefore particularly hard to employ in practice, and parameter tuning is a tedious and costly enterprise. In order to mitigate this problem, we derive an automatic online LSPI algorithm that operates over continuous action spaces and does not require an a-priori, hand-tuned value function approximation architecture. To this end, we first show how the kernel least-squares policy iteration algorithm can be modified to handle data online by recursive dictionary and learning update rules. Next, borrowing sparsification methods from kernel adaptive filtering, the continuous action-space approximation in the online least-squares policy iteration algorithm can be efficiently automated as well. We then propose a similarity-based information extrapolation for the recursive temporal difference update in order to perform the dictionary expansion step efficiently in both algorithms. The performance of the proposed algorithms is compared with respect to their batch or hand-tuned counterparts in a simulation study. The novel algorithms require less prior tuning and data is processed completely on the fly, yet the results indicate that similar performance can be obtained as by careful hand-tuning. Therefore, engineers from both robotics and AI can benefit from the proposed algorithms when an LSPI algorithm is faced with online data collection and tuning by experiment is costly.

Uncertain Graph Sparsification

Uncertain graphs are prevalent in several applications including communications systems, biological databases, and social networks. The ever increasing size of the underlying data renders both graph storage and query processing extremely expensive. Sparsification has often been used to reduce the size of deterministic graphs by maintaining only the important edges. However, adaptation of deterministic sparsification methods fails in the uncertain setting. To overcome this problem, we introduce the first sparsification techniques aimed explicitly at uncertain graphs. The proposed methods reduce the number of edges and redistribute their probabilities in order to decrease the graph size, while preserving its underlying structure. The resulting graph can be used to efficiently and accurately approximate any query and mining tasks on the original graph. An extensive experimental evaluation with real and synthetic datasets illustrates the effectiveness of our techniques on several common graph tasks, including clustering coefficient, page rank, reliability, and shortest path distance.

Kernel recursive maximum correntropy with Nyström approximation

Kernel adaptive filters (KAFs) with growing network structures incur high computational burden. Generally, sparsification methods are introduced to curb the growth of the filter structure under some threshold rules, resulting in a variable structure. Unlike the sparsification methods, the Nyström method uses a subset of data samples to form the filter structure of a fixed size. In this paper, to combat the large outliers efficiently, the kernel recursive maximum correntropy with Nyström approximation (KRMC-NA) is proposed to achieve desirable filtering performance under a fixed and efficient filter structure. In addition, the theoretical analysis on the convergence characteristics of KRMC-NA is provided. Simulation results illustrate that the proposed KRMC-NA has better filtering accuracy than KAFs with sparsification, and approaches the filtering accuracy of the kernel recursive maximum correntropy using lower computational complexity.

A real-time conflict solution algorithm for the train rescheduling problem

We consider the real-time resolution of conflicts arising in real-world train management applications. In particular, given a nominal timetable for a set of trains and a set of modifications due to delays or other resources unavailability, we are aiming at defining a set of actions which must be implemented to grant safety, e.g., to avoid potential conflicts such as train collisions or headway violations, and restore quality by reducing the delays. To be compatible with real-time management, the required actions must be determined in a few seconds, hence specialized fast heuristics must be used.We propose a fast and effective parallel algorithm that is based on an iterated greedy scheduling of trains on a time-space network. The algorithm uses several sortings to define the initial train dispatching rule and different shaking methods between iterations. The performance is further enhanced by using various sparsification methods for the time-space network. The best algorithm configuration is determined through extensive experiments, conducted on a set of instances derived from real-world networks and instances from the literature. The resulting heuristic proved able to consistently resolve the existing conflicts and obtaining excellent solution quality within just two seconds of computing time on a standard personal computer, for instances involving up to 151 trains and two hours of planning time horizon.

Designing granular solution methods for routing problems with time windows

The use of granular neighborhoods is one way to improve the run-time of local-search-based metaheuristics for combinatorial optimization problems without compromising solution quality. So-called sparsification methods are applied to restrict the neighborhoods to include only elements which are likely to be part of high-quality solutions. The goal of this work is to provide insights about the design of effective and efficient granular solution methods for routing problems with time windows. In extensive numerical experiments with a granular tabu search (GTS) for the vehicle-routing problem with time windows (VRPTW), we find that sparsification methods using reduced-cost values based on the solution of a linear relaxation of the original problem outperform standard sparsification methods. Furthermore, including additional depot arcs into the restricted arc set (beyond those selected by the sparsification method) improves solution quality. The same is true for dynamically inserting into the restricted arc set (i) the arcs involved in the best move selected in each iteration, and (ii) the arcs of the current best-known solution. Moreover, for small restricted arc sets, guaranteeing a minimum number of incoming and outgoing arcs per vertex is beneficial. Finally, dynamically altering the size of the restricted arc set can be used to successfully diversify and intensify the search, which has a significant positive effect on solution quality. The usefulness of the gained insights about the design of granular solution methods is demonstrated by the performance of the final configuration of our GTS for VRPTW, which obtains state-of-the-art results and reaches an outstanding computational efficiency.

Single- and multi-level network sparsification by algebraic distance

Network sparsification methods play an important role in modern network analysis when fast estimation of computationally expensive properties (such as the diameter, centrality indices, and paths) is required. We propose a method of network sparsification that preserves a wide range of structural properties. Depending on the analysis goals, the method allows to distinguish between local and global range edges that can be filtered out during the sparsification. First we rank edges by their algebraic distances and then we sample them. We also introduce a multilevel framework for sparsification that can be used to control the sparsification process at various coarse-grained resolutions. Based primarily on the matrix-vector multiplications, our method is easily parallelized for different architectures.

Revisiting compressed sensing: exploiting the efficiency of simplex and sparsification methods

We propose two approaches to solve large-scale compressed sensing problems. The first approach uses the parametric simplex method to recover very sparse signals by taking a small number of simplex pivots, while the second approach reformulates the problem using Kronecker products to achieve faster computation via a sparser problem formulation. In particular, we focus on the computational aspects of these methods in compressed sensing. For the first approach, if the true signal is very sparse and we initialize our solution to be the zero vector, then a customized parametric simplex method usually takes a small number of iterations to converge. Our numerical studies show that this approach is 10 times faster than state-of-the-art methods for recovering very sparse signals. The second approach can be used when the sensing matrix is the Kronecker product of two smaller matrices. We show that the best-known sufficient condition for the Kronecker compressed sensing (KCS) strategy to obtain a perfect recovery is more restrictive than the corresponding condition if using the first approach. However, KCS can be formulated as a linear program with a very sparse constraint matrix, whereas the first approach involves a completely dense constraint matrix. Hence, algorithms that benefit from sparse problem representation, such as interior point methods (IPMs), are expected to have computational advantages for the KCS problem. We numerically demonstrate that KCS combined with IPMs is up to 10 times faster than vanilla IPMs and state-of-the-art methods such as $$\ell _1\_\ell _s$$ and Mirror Prox regardless of the sparsity level or problem size.

Structure-preserving sparsification methods for social networks

Sparsification reduces the size of networks while preserving structural and statistical properties of interest. Various sparsifying algorithms have been proposed in different contexts. We contribute the first systematic conceptual and experimental comparison of edge sparsification methods on a diverse set of network properties. It is shown that they can be understood as methods for rating edges by importance and then filtering globally or locally by these scores. We show that applying a local filtering technique improves the preservation of all kinds of properties. In addition, we propose a new sparsification method (Local Degree) which preserves edges leading to local hub nodes. All methods are evaluated on a set of social networks from Facebook, Google+, Twitter and LiveJournal with respect to network properties including diameter, connected components, community structure, multiple node centrality measures and the behavior of epidemic simulations. To assess the preservation of the community structure, we also include experiments on synthetically generated networks with ground truth communities. Experiments with our implementations of the sparsification methods (included in the open-source network analysis tool suite NetworKit) show that many network properties can be preserved down to about 20 % of the original set of edges for sparse graphs with a reasonable density. The experimental results allow us to differentiate the behavior of different methods and show which method is suitable with respect to which property. While our Local Degree method is best for preserving connectivity and short distances, other newly introduced local variants are best for preserving the community structure.

Quantifying loss of information in network-based dimensionality reduction techniques

To cope with the complexity of large networks, a number of dimensionality reduction techniques for graphs have been developed. However, the extent to which information is lost or preserved when these techniques are employed has not yet been clear. Here we develop a framework, based on algorithmic information theory, to quantify the extent to which information is preserved when network motif analysis, graph spectra and spectral sparsification methods are applied to over twenty different biological and artificial networks. We find that the spectral sparsification is highly sensitive to high number of edge deletion, leading to significant inconsistencies, and that graph spectral methods are the most irregular, capturing algebraic information in a condensed fashion but largely losing most of the information content of the original networks. However, the approach shows that network motif analysis excels at preserving the relative algorithmic information content of a network, hence validating and generalizing the remarkable fact that despite their inherent combinatorial possibilities, local regularities preserve information to such an extent that essential properties are fully recoverable across different networks to determine their family group to which they belong to (eg genetic vs social network). Our algorithmic information methodology thus provides a rigorous framework enabling a fundamental assessment and comparison between different data dimensionality reduction methods thereby facilitating the identification and evaluation of the capabilities of old and new methods.

Clustering and visualizing similarity networks of membrane proteins.

We proposed a fast and unsupervised clustering method, minimum span clustering (MSC), for analyzing the sequence-structure-function relationship of biological networks, and demonstrated its validity in clustering the sequence/structure similarity networks (SSN) of 682 membrane protein (MP) chains. The MSC clustering of MPs based on their sequence information was found to be consistent with their tertiary structures and functions. For the largest seven clusters predicted by MSC, the consistency in chain function within the same cluster is found to be 100%. From analyzing the edge distribution of SSN for MPs, we found a characteristic threshold distance for the boundary between clusters, over which SSN of MPs could be properly clustered by an unsupervised sparsification of the network distance matrix. The clustering results of MPs from both MSC and the unsupervised sparsification methods are consistent with each other, and have high intracluster similarity and low intercluster similarity in sequence, structure, and function. Our study showed a strong sequence-structure-function relationship of MPs. We discussed evidence of convergent evolution of MPs and suggested applications in finding structural similarities and predicting biological functions of MP chains based on their sequence information.