Low-rank Format Research Articles

Edge Private Graph Neural Networks with Singular Value Perturbation

Graph neural networks (GNNs) play a key role in learning representations from graph-structured data and are demonstrated to be useful in many applications. However, the GNN training pipeline has been shown to be vulnerable to node feature leakage and edge extraction attacks. This paper investigates a scenario where an attacker aims to recover private edge information from a trained GNN model. Previous studies have employed differential privacy (DP) to add noise directly to the adjacency matrix or a compact graph representation. The added perturbations cause the graph structure to be substantially morphed, reducing the model utility. We propose a new privacy-preserving GNN training algorithm, Eclipse, that maintains good model utility while providing strong privacy protection on edges. Eclipse is based on two key observations. First, adjacency matrices in graph structures exhibit low-rank behavior. Thus, Eclipse trains GNNs with a low-rank format of the graph via singular values decomposition (SVD), rather than the original graph. Using the low-rank format, Eclipse preserves the primary graph topology and removes the remaining residual edges. Eclipse adds noise to the low-rank singular values instead of the entire graph, thereby preserving the graph privacy while still maintaining enough of the graph structure to maintain model utility. We theoretically show Eclipse provide formal DP guarantee on edges. Experiments on benchmark graph datasets show that Eclipse achieves significantly better privacy-utility tradeoff compared to existing privacy-preserving GNN training methods. In particular, under strong privacy constraints (𝜖 < 4), Eclipse shows significant gains in the model utility by up to 46%. We further demonstrate that Eclipse also has better resilience against common edge attacks (e.g., LPA), lowering the attack AUC by up to 5% compared to other state-of-the-art baselines.

Low-rank solutions to the stochastic Helmholtz equation

In this paper, we consider low-rank approximations for the solutions to the stochastic Helmholtz equation with random coefficients. A Stochastic Galerkin finite element method is used for the discretization of the Helmholtz problem. Existence theory for the low-rank approximation is established when the system matrix is indefinite. The low-rank algorithm does not require the construction of a large system matrix which results in an advantage in terms of CPU time and storage. Numerical results show that, when the operations in a low-rank method are performed efficiently, it is possible to obtain an advantage in terms of storage and CPU time compared to computations in full rank. We also propose a general approach to implement a preconditioner using the low-rank format efficiently.

Sparse Approximate Multifrontal Factorization with Composite Compression Methods

This article presents a fast and approximate multifrontal solver for large sparse linear systems. In a recent work by Liu et al., we showed the efficiency of a multifrontal solver leveraging the butterfly algorithm and its hierarchical matrix extension, HODBF (hierarchical off-diagonal butterfly) compression to compress large frontal matrices. The resulting multifrontal solver can attain quasi-linear computation and memory complexity when applied to sparse linear systems arising from spatial discretization of high-frequency wave equations. To further reduce the overall number of operations and especially the factorization memory usage to scale to larger problem sizes, in this article we develop a composite multifrontal solver that employs the HODBF format for large-sized fronts, a reduced-memory version of the nonhierarchical block low-rank format for medium-sized fronts, and a lossy compression format for small-sized fronts. This allows us to solve sparse linear systems of dimension up to 2.7 × larger than before and leads to a memory consumption that is reduced by 70% while ensuring the same execution time. The code is made publicly available in GitHub.

Fast hierarchical low-rank view factor matrices for thermal irradiance on planetary surfaces

We present an algorithm for compressing the radiosity view factor model commonly used in radiation heat transfer and computer graphics. We use a format inspired by the hierarchical off-diagonal low rank format, where elements are recursively partitioned using a quadtree or octree and blocks are compressed using a sparse singular value decomposition—the hierarchical matrix is assembled using dynamic programming. The motivating application is time-dependent thermal modeling on vast planetary surfaces, with a focus on permanently shadowed craters which receive energy through indirect irradiance. In this setting, shape models are comprised of a large number of triangular facets which conform to a rough surface. At each time step, a quadratic number of triangle-to-triangle scattered fluxes must be summed; that is, as the sun moves through the sky, we must solve the same view factor system of equations for a potentially unlimited number of time-varying righthand sides. We first conduct numerical experiments with a synthetic spherical cap-shaped crater, where the equilibrium temperature is analytically available. We also test our implementation with triangle meshes of planetary surfaces derived from digital elevation models recovered by orbiting spacecraft. Our results indicate that the compressed view factor matrix can be assembled in quadratic time, which is comparable to the time it takes to assemble the full view matrix itself. Memory requirements during assembly are reduced by a large factor. Finally, for a range of compression tolerances, the size of the compressed view factor matrix and the speed of the resulting matrix vector product both scale linearly (as opposed to quadratically for the full matrix), resulting in orders of magnitude savings in processing time and memory space.

Low-rank tensor structure preservation in fractional operators by means of exponential sums

The use of fractional differential equations is a key tool in modeling non-local phenomena. Often, an efficient scheme for solving a linear system involving the discretization of a fractional operator is computing inverse fractional powers of the standard discretized Laplace operator. In this work, an exponential sum approximation for such fractional powers is derived. It is accurate over all positive real numbers larger than one, and allows to efficiently approximate the action of such operators on tensors stored in a variety of low-rank formats (CP, TT, Tucker). The results are relevant from a practical and theoretical perspective, as they predict the low-rank approximability of the solutions of these linear systems in low-rank tensor formats.

Low-rank tensor methods for partial differential equations

Low-rank tensor representations can provide highly compressed approximations of functions. These concepts, which essentially amount to generalizations of classical techniques of separation of variables, have proved to be particularly fruitful for functions of many variables. We focus here on problems where the target function is given only implicitly as the solution of a partial differential equation. A first natural question is under which conditions we should expect such solutions to be efficiently approximated in low-rank form. Due to the highly nonlinear nature of the resulting low-rank approximations, a crucial second question is at what expense such approximations can be computed in practice. This article surveys basic construction principles of numerical methods based on low-rank representations as well as the analysis of their convergence and computational complexity.

Low-rank tensor methods for Markov chains with applications to tumor progression models

Cancer progression can be described by continuous-time Markov chains whose state space grows exponentially in the number of somatic mutations. The age of a tumor at diagnosis is typically unknown. Therefore, the quantity of interest is the time-marginal distribution over all possible genotypes of tumors, defined as the transient distribution integrated over an exponentially distributed observation time. It can be obtained as the solution of a large linear system. However, the sheer size of this system renders classical solvers infeasible. We consider Markov chains whose transition rates are separable functions, allowing for an efficient low-rank tensor representation of the linear system’s operator. Thus we can reduce the computational complexity from exponential to linear. We derive a convergent iterative method using low-rank formats whose result satisfies the normalization constraint of a distribution. We also perform numerical experiments illustrating that the marginal distribution is well approximated with low rank.

Structured Matrix Approximations via Tensor Decompositions

We provide a computational framework for approximating a class of structured matrices; here, the term structure is very general, and may refer to a regular sparsity pattern (e.g., block-banded), or be more highly structured (e.g., symmetric block Toeplitz). The goal is to uncover {\it additional latent structure} that will in turn lead to computationally efficient algorithms when the new structured matrix approximations are employed in the place of the original operator. Our approach has three steps: map the structured matrix to tensors, use tensor compression algorithms, and map the compressed tensors back to obtain two different matrix representations -- sum of Kronecker products and block low-rank format. The use of tensor decompositions enables us to uncover latent structure in the problem and leads to compressed representations of the original matrix that can be used efficiently in applications. The resulting matrix approximations are memory efficient, easy to compute with, and preserve the error that is due to the tensor compression in the Frobenius norm. Our framework is quite general. We illustrate the ability of our method to uncover block-low-rank format on structured matrices from two applications: system identification, space-time covariance matrices. In addition, we demonstrate that our approach can uncover sum of structured Kronecker products structure on several matrices from the SuiteSparse collection. Finally, we show that our framework is broad enough to encompass and improve on other related results from the literature, as we illustrate with the approximation of a three-dimensional blurring operator.

{mathcal{H}}-matrix approximability of inverses of FEM matrices for the time-harmonic Maxwell equations

The inverse of the stiffness matrix of the time-harmonic Maxwell equation with perfectly conducting boundary conditions is approximated in the blockwise low-rank format of {mathcal{H}}-matrices. Under a technical assumption on the mesh, we prove that root exponential convergence in the block rank can be achieved, if the block structure conforms to a standard admissibility criterion.

RFN: A Random-Feature Based Newton Method for Empirical Risk Minimization in Reproducing Kernel Hilbert Spaces

In supervised learning using kernel methods, we often encounter a large-scale finite-sum minimization over a reproducing kernel Hilbert space (RKHS). Large-scale finite-sum problems can be solved using efficient variants of Newton method, where the Hessian is approximated via sub-samples of data. In RKHS, however, the dependence of the penalty function to kernel makes standard sub-sampling approaches inapplicable, since the gram matrix is not readily available in a low-rank form. In this paper, we observe that for this class of problems, one can naturally use kernel approximation to speed up the Newton method. Focusing on randomized features for kernel approximation, we provide a novel second-order algorithm that enjoys local superlinear convergence and global linear convergence (with high probability). We derive the theoretical lower bound for the number of random features required for the approximated Hessian to be close to the true Hessian in the norm sense. Our numerical experiments on real-world data verify the efficiency of our method compared to several benchmarks.

A Relaxed Interior Point Method for Low-Rank Semidefinite Programming Problems with Applications to Matrix Completion

A new relaxed variant of interior point method for low-rank semidefinite programming problems is proposed in this paper. The method is a step outside of the usual interior point framework. In anticipation to converging to a low-rank primal solution, a special nearly low-rank form of all primal iterates is imposed. To accommodate such a (restrictive) structure, the first order optimality conditions have to be relaxed and are therefore approximated by solving an auxiliary least-squares problem. The relaxed interior point framework opens numerous possibilities how primal and dual approximated Newton directions can be computed. In particular, it admits the application of both the first- and the second-order methods in this context. The convergence of the method is established. A prototype implementation is discussed and encouraging preliminary computational results are reported for solving the SDP-reformulation of matrix-completion problems.

A generalized Fourier transform by means of change of variables within multilinear approximation

The paper deals with approximations of periodic functions that play a significant role in harmonic analysis. The approach revisits the trigonometric polynomials, seen as combinations of functions, and proposes to extend the class of models of the combined functions to a wider class of functions. The key here is to use structured functions, that have low complexity, with suitable functional representation and adapted parametrizations for the approximation. Such representation enables to approximate multivariate functions with few eventually random samples. The new parametrization is determined automatically with a greedy procedure, and a low rank format is used for the approximation associated with each new parametrization. A supervised learning algorithm is used for the approximation of a function of multiple random variables in tree-based tensor format, here the particular Tensor Train format. Adaptive strategies using statistical error estimates are proposed for the selection of the underlying tensor bases and the ranks for the Tensor-Train format. The method is applied for the estimation of the wall pressure for a flow over a cylinder for a range of low to medium Reynolds numbers for which we observe two flow regimes: a laminar flow with periodic vortex shedding and a laminar boundary layer with a turbulent wake (sub-critic regime). The automatic re-parametrization enables here to take into account the specific periodic feature of the pressure.

Reconstruction of low-rank aggregation kernels in univariate population balance equations

The dynamics of particle processes can be described by population balance equations which are governed by phenomena including growth, nucleation, breakage and aggregation. Estimating the kinetics of the aggregation phenomena from measured density data constitutes an ill-conditioned inverse problem. In this work, we focus on the aggregation problem and present an approach to estimate the aggregation kernel in discrete, low rank form from given (measured or simulated) data. The low-rank assumption for the kernel allows the application of fast techniques for the evaluation of the aggregation integral (mathcal {O}(nlog n) instead of mathcal {O}(n^{2}) where n denotes the number of unknowns in the discretization) and reduces the dimension of the optimization problem, allowing for efficient and accurate kernel reconstructions. We provide and compare two approaches which we will illustrate in numerical tests.

Prospects of Tensor-Based Numerical Modeling of the Collective Electrostatics in Many-Particle Systems

Recently the rank-structured tensor approach suggested a progress in the numerical treatment of the long-range electrostatic potentials in many-particle systems and the respective interaction energy and forces [39,40,2]. In this paper, we outline the prospects for tensor-based numerical modeling of the collective electrostatic potential on lattices and in many-particle systems of general type. We generalize the approach initially introduced for the rank-structured grid-based calculation of the collective potentials on 3D lattices [39] to the case of many-particle systems with variable charges placed on $L^{\otimes d}$ lattices and discretized on fine $n^{\otimes d}$ Cartesian grids for arbitrary dimension $d$. As result, the interaction potential is represented in a parametric low-rank canonical format in $O(d L n)$ complexity. The energy is then calculated in $O(d L)$ operations. Electrostatics in large biomolecules is modeled by using the novel range-separated (RS) tensor format [2], which maintains the long-range part of the 3D collective potential of the many-body system represented on $n\times n \times n$ grid in a parametric low-rank form in $O(n)$-complexity. We show that the force field can be easily recovered by using the already precomputed electric field in the low-rank RS format. The RS tensor representation of the discretized Dirac delta [45] enables the efficient energy preserving regularization scheme for solving the 3D elliptic PDEs with strongly singular right-hand side arising in bio-sciences. We conclude that the rank-structured tensor-based approximation techniques provide the promising numerical tools for applications to many-body dynamics, protein docking and classification problems and for low-parametric interpolation of scattered data in data science.

Underwater reverberation suppression based on non-negative matrix factorisation

Reverberation is the main background interference in underwater active sonar target detection that seriously interferes with the extraction of the target highlight echo. In consideration of the difference in the time-frequency distributions of the target highlight echo and reverberation, an underwater reverberation suppression method based on non-negative matrix factorisation is proposed in this study. The Wigner-Ville time-frequency matrix of the underwater target echo is used as the input, and non-negative matrix factorisation is applied to express the time-frequency matrix as a low-rank matrix. The influences of rank selection on the reverberation suppression effect and signal expression of the target highlight echo are analysed. Given that the time-frequency matrix of the target highlight echo cannot be expressed in the low-rank form when transmitting linear frequency modulation signal, this matrix is processed with low-rank preprocessing through matrix rotation. The components of the target highlight echo, which can be expressed in the form of a low-rank matrix, are preserved. By contrast, the components of the reverberation interference, which cannot be represented in such a form, are removed in the underwater target echoes. The simulation and experimental results show that the algorithm can effectively improve the signal-to-reverberation ratio of underwater target echoes.

Singular Value Decomposition in Sobolev Spaces: Part I

A well known result from functional analysis states that any compact operator between Hilbert spaces admits a singular value decomposition (SVD). This decomposition is a powerful tool that is the workhorse of many methods both in mathematics and applied fields. A prominent application in recent years is the approximation of high-dimensional functions in a low-rank format. This is based on the fact that, under certain conditions, a tensor can be identified with a compact operator and SVD applies to the latter. One key assumption for this application is that the tensor product norm is not weaker than the injective norm. This assumption is not fulfilled in Sobolev spaces, which are widely used in the theory and numerics of partial differential equations. Our goal is the analysis of the SVD in Sobolev spaces. This work consists of two parts. In this manuscript (part I), we address low-rank approximations and minimal subspaces in H1. We analyze the H1-error of the SVD performed in the ambient L2-space. In part II, we will address variants of the SVD in norms stronger than the L2-norm. We will provide a few numerical examples that support our theoretical findings.

Computer Aided Diagnosis For Retinal Lesion Detection Based on Adaptive Boosting Classifier

Timely analysis and constant intensive care of patient’s pain since eye ailments must remained main worries in the computer-aided detection (CAD) ways and means. Intuiting sole or few precise styles of retinal wrongs has set an essential origination in computed-aid screen in the first few dates. Thus far, in line for to range of retinal wrongs and multilayered common useful productions, unplanned innovation of wounds with unrevealed and wide-ranging styles as retina leavings an exciting work. In this pattern, a weakly supervised practice, demanding just an order of common and not level retinal pictures underprivileged of vital to just luster their places and multiplicities, is helpful for this charge. Specifically, a eye copy is considered as a superposition of related, blood vessels and related noise (lesions embraced for unequal descriptions). Background is given as low-rank form followed by preprocessing steps, plus three-D plan, color regularization and blood vessels rejection. Related noise is eyed as stochastic changeable and reputable over Gaussian for classic imageries and mixture of Gaussian (MoG) for asymmetrical pictures, harmoniously. The coming run through codes together the related facts of fund us pictures and the related noise into on its own one and only ideal, and corporately enriches the ideal by typical and unequal pictures, which absolutely expose the low-rank subspace of the related and discriminate the wounds from related noise in unequal fundus pictures. Lastly, Ada boost classification process is castoff to increase sensing retinal injuries and to categorize the kinds of injury using a group of usual ones for exercise topographies. Investigational consequences prove that the future method is of well skills correctness and outdoes the preceding connected approaches.

Nonlinear model order reduction with low rank tensor approximation

AbstractIn this article, two methods of model order reduction based on the low rank approximation of tensor are introduced for the large scale nonlinear problem. We first introduce some definitions and results on tensor extended from matrix theory. Then we show how the general nonlinear system can be converted into the low rank form we treated in this research. We put the model order reduction of it in two frameworks, that is, polynomial framework and moment‐matching framework. In these two frameworks we construct the algorithms correspondingly, and analyze properties of these algorithms, including the preservation of stability, and moment‐matching properties. Next the priorities of these algorithms are presented. Finally we setup several numerical experiments to validate the effectiveness of the algorithms.

An Accelerated Accuracy-enhanced Randomized Singular Value Decomposition for Factorizing Matrices with Low-rank Structure

Big data has in recent years gained ground in many scientific and engineering problems. It seems to some extent prohibitive for traditional matrix decomposition methods (i.e. QR, SVD, EVD, etc.) to handle such large-scale problems involving data matrix. Many researchers have developed several algorithms to decompose such big data matrices. An accuracy-enhanced randomized singular value decomposition method (referred to as AE-RSVDM) with orthonormalization recently becomes the state-of-the-art to factorize large data matrices with satisfactory speed and accuracy. In our paper, low-rank matrix approximations based on randomization are studied, with emphasis on accelerating the computational efficiency on large data matrices. By this, we accelerate the AE-RSVDM with modified normalized power iteration to result in an accelerated version. The accelerated version is grounded on a two-stage scheme. The first stage seeks to find the range of a sketch matrix which involves a Gaussian random matrix. A low-dimensional space is then created from the high-dimensional data matrix via power iteration. Numerical experiments on matrices of different sizes demonstrate that our accelerated variant achieves speedups while attaining the same reconstruction error as the AE-RSVDM with orthonormalization. And with data from Google art project, we have made known the computational speed-up of the accelerated variant over the AE-RSVDM algorithm for decomposing large data matrices with low-rank form.

Performance and Scalability of the Block Low-Rank Multifrontal Factorization on Multicore Architectures

Matrices coming from elliptic Partial Differential Equations have been shown to have a low-rank property that can be efficiently exploited in multifrontal solvers to provide a substantial reduction of their complexity. Among the possible low-rank formats, the Block Low-Rank format (BLR) is easy to use in a general purpose multifrontal solver and its potential compared to standard (full-rank) solvers has been demonstrated. Recently, new variants have been introduced and it was proved that they can further reduce the complexity but their performance has never been analyzed. In this article, we present a multithreaded BLR factorization and analyze its efficiency and scalability in shared-memory multicore environments. We identify the challenges posed by the use of BLR approximations in multifrontal solvers and put forward several algorithmic variants of the BLR factorization that overcome these challenges by improving its efficiency and scalability. We illustrate the performance analysis of the BLR multifrontal factorization with numerical experiments on a large set of problems coming from a variety of real-life applications.