Low-rank Manifold Research Articles

Compact Model Training by Low-Rank Projection With Energy Transfer.

Low-rankness plays an important role in traditional machine learning but is not so popular in deep learning. Most previous low-rank network compression methods compress networks by approximating pretrained models and retraining. However, the optimal solution in the Euclidean space may be quite different from the one with low-rank constraint. A well-pretrained model is not a good initialization for the model with low-rank constraints. Thus, the performance of a low-rank compressed network degrades significantly. Compared with other network compression methods such as pruning, low-rank methods attract less attention in recent years. In this article, we devise a new training method, low-rank projection with energy transfer (LRPET), that trains low-rank compressed networks from scratch and achieves competitive performance. We propose to alternately perform stochastic gradient descent training and projection of each weight matrix onto the corresponding low-rank manifold. Compared to retraining on the compact model, this enables full utilization of model capacity since solution space is relaxed back to Euclidean space after projection. The matrix energy (the sum of squares of singular values) reduction caused by projection is compensated by energy transfer. We uniformly transfer the energy of the pruned singular values to the remaining ones. We theoretically show that energy transfer eases the trend of gradient vanishing caused by projection. In modern networks, a batch normalization (BN) layer can be merged into the previous convolution layer for inference, thereby influencing the optimal low-rank approximation (LRA) of the previous layer. We propose BN rectification to cut off its effect on the optimal LRA, which further improves the performance. Comprehensive experiments on CIFAR-10 and ImageNet have justified that our method is superior to other low-rank compression methods and also outperforms recent state-of-the-art pruning methods. For object detection and semantic segmentation, our method still achieves good compression results. In addition, we combine LRPET with quantization and hashing methods and achieve even better compression than the original single method. We further apply it in Transformer-based models to demonstrate its transferability. Our code is available at https://github.com/BZQLin/LRPET.

Dynamical low-rank approximation of the Vlasov–Poisson equation with piecewise linear spatial boundary

Dynamical low-rank approximation (DLRA) for the numerical simulation of Vlasov–Poisson equations is based on separation of space and velocity variables, as proposed in several recent works. The standard approach for the time integration in the DLRA model uses a splitting of the tangent space projector for the low-rank manifold according to the separated variables. It can also be modified to allow for rank-adaptivity. A less studied aspect is the incorporation of boundary conditions in the DLRA model. In this work, a variational formulation of the projector splitting is proposed which allows to handle inflow boundary conditions on spatial domains with piecewise linear boundary. Numerical experiments demonstrate the principle feasibility of this approach.

A robust collision source method for rank adaptive dynamical low-rank approximation in radiation therapy

Deterministic models for radiation transport describe the density of radiation particles moving through a background material. In radiation therapy applications, the phase space of this density is composed of energy, spatial position and direction of flight. The resulting six-dimensional phase space prohibits fine numerical discretizations, which are essential for the construction of accurate and reliable treatment plans. In this work, we tackle the high dimensional phase space through a dynamical low-rank approximation of the particle density. Dynamical low-rank approximation (DLRA) evolves the solution on a low-rank manifold in time. Interpreting the energy variable as a pseudo-time lets us employ the DLRA framework to represent the solution of the radiation transport equation on a low-rank manifold for every energy. Stiff scattering terms are treated through an efficient implicit energy discretization and a rank adaptive integrator is chosen to dynamically adapt the rank in energy. To facilitate the use of boundary conditions and reduce the overall rank, the radiation transport equation is split into collided and uncollided particles through a collision source method. Uncollided particles are described by a directed quadrature set guaranteeing low computational costs, whereas collided particles are represented by a low-rank solution. It can be shown that the presented method is L2-stable under a time step restriction which does not depend on stiff scattering terms. Moreover, the implicit treatment of scattering does not require numerical inversions of matrices. Numerical results for radiation therapy configurations as well as the line source benchmark underline the efficiency of the proposed method.

Matrix Completion via Successive Low-rank Matrix Approximation

In this paper, a successive low-rank matrix approximation algorithm is presented for the matrix completion (MC) based on hard thresholding method, which approximate the optimal low-rank matrix from rank-one matrix step by step. The algorithm enables the distance between the matrix with the observed elements and the projection on low-rank manifold to be minimum. The optimal low-rank matrix with observed elements is obtained when the distance is zero. In theory, convergence and convergent error of the new algorithm are analyzed in detail. Furthermore, some numerical experiments show that the algorithm is more effective in CPU time and precision than the orthogonal rank-one matrix pursuit(OR1MP) algorithm and the augmented Lagrange multiplier (ALM) method when the sampling rate is low.

Riemannian Low-Rank Model Compression for Federated Learning With Over-the-Air Aggregation

Low-rank model compression is a widely used technique for reducing the computational load when training machine learning models. However, existing methods often rely on relaxing the low-rank constraint of the model weights using a regularized nuclear norm penalty, which requires an appropriate hyperparameter that can be difficult to determine in practice. Furthermore, existing compression techniques are not directly applicable to efficient over-the-air (OTA) aggregation in federated learning (FL) systems for distributed Internet-of-Things (IoT) scenarios. In this paper, we propose a novel manifold optimization formulation for low-rank model compression in FL that does not relax the low-rank constraint. Our optimization is conducted directly over the low-rank manifold, guaranteeing that the model is exactly low-rank. We also introduce a consensus penalty in the optimization formulation to support OTA aggregation. Based on our optimization formulation, we propose an alternating Riemannian optimization algorithm with a precoder that enables efficient OTA aggregation of low-rank local models without sacrificing training performance. Additionally, we provide convergence analysis in terms of key system parameters and conduct extensive experiments with real-world datasets to demonstrate the effectiveness of our proposed Riemannian low-rank model compression scheme compared to various state-of-the-art baselines.

A sweep-based low-rank method for the discrete ordinate transport equation

The dynamical low-rank (DLR) approximation is an efficient technique to approximate the solution to matrix differential equations. Recently, the DLR method was applied to radiation transport calculations to reduce memory requirements and computational costs. This work extends the low-rank scheme for the time-dependent radiation transport equation in 2-D and 3-D Cartesian geometries with discrete ordinates discretization in angle (SN method). The reduced system that evolves on a low-rank manifold is constructed via the “unconventional” basis update & Galerkin integrator to avoid a substep that is backward in time, which could be unstable for dissipative problems. The resulting system preserves the information on angular direction by applying separate low-rank decompositions in each octant where angular intensity has the same sign as the direction cosines. Then, transport sweeps and source iteration can efficiently solve this low-rank-SN system. The numerical results in 2-D and 3-D Cartesian geometries demonstrate that the low-rank solution requires less memory and computational time than solving the full rank equations using transport sweeps without losing accuracy.

Numerical Assessment of a Nonintrusive Surrogate Model Based on Recurrent Neural Networks and Proper Orthogonal Decomposition: Rayleigh–Bénard Convection

Recent developments in diagnostic and computing technologies offer to leverage numerous forms of nonintrusive modelling approaches from data where machine learning can be used to build computationally cheap and accurate surrogate models. To this end, we present a nonlinear proper orthogonal decomposition (POD) framework, denoted as NLPOD, to forge a nonintrusive reduced-order model for the Boussinesq equations. In our NLPOD approach, we first employ the POD procedure to obtain a set of global modes to build a linear-fit latent space and utilise an autoencoder network to compress the projection of this latent space through a nonlinear unsupervised mapping of POD coefficients. Then, long short-term memory (LSTM) neural network architecture is utilised to discover temporal patterns in this low-rank manifold. While performing a detailed sensitivity analysis for hyperparameters of the LSTM model, the trade-off between accuracy and efficiency is systematically analysed for solving a canonical Rayleigh–Bénard convection system.

A Sweep-Based Low-Rank Method for the Discrete Ordinate Transport Equation

The dynamical low-rank (DLR) approximation is an efficient technique to approximate the solution to matrix differential equations. Recently, the DLR method was applied to radiation transport calculations to reduce memory requirements and computational costs. This work extends the low-rank scheme for the time-dependent radiation transport equation in 2-D and 3-D Cartesian geometries with discrete ordinates discretization in angle (SN method). The reduced system that evolves on a low-rank manifold is constructed via an unconventional basis update and Galerkin integrator to avoid a substep that is backward in time, which could be unstable for dissipative problems. The resulting system preserves the information on angular direction by applying separate low-rank decompositions in each octant where angular intensity has the same sign as the direction cosines. Then, transport sweeps and source iteration can efficiently solve this low-rank-SN system. The numerical results in 2-D and 3-D Cartesian geometries demonstrate that the low-rank solution requires less memory and computational time than solving the full rank equations using transport sweeps without losing accuracy.

Computing low-rank rightmost eigenpairs of a class of matrix-valued linear operators

In this article, a new method is proposed to approximate the rightmost eigenpair of certain matrix-valued linear operators, in a low-rank setting. First, we introduce a suitable ordinary differential equation, whose solution allows us to approximate the rightmost eigenpair of the linear operator. After analyzing the behaviour of its solution on the whole space, we project the ODE on a low-rank manifold of prescribed rank and correspondingly analyze the behaviour of its solutions. For a general linear operator we prove that—under generic assumptions—the solution of the ODE converges globally to its leading eigenmatrix. The analysis of the projected operator is more subtle due to its nonlinearity; when ca is self-adjoint, we are able to prove that the associated low-rank ODE converges (at least locally) to its rightmost eigenmatrix in the low-rank manifold, a property which appears to hold also in the more general case. Two explicit numerical methods are proposed, the second being an adaptation of the projector splitting integrator proposed recently by Lubich and Oseledets. The numerical experiments show that the method is effective and competitive.

Existence of dynamical low-rank approximations to parabolic problems

The existence and uniqueness of weak solutions to dynamical low-rank evolution problems for parabolic partial differential equations in two spatial dimensions is shown, covering also non-diagonal diffusion in the elliptic part. The proof is based on a variational time-stepping scheme on the low-rank manifold. Moreover, this scheme is shown to be closely related to practical methods for computing such low-rank evolutions.

Deep Manifold Learning for Dynamic MR Imaging

Purpose: To develop a deep learning method on a nonlinear manifold to explore the temporal redundancy of dynamic signals to reconstruct cardiac MRI data from highly undersampled measurements. Methods: Cardiac MR image reconstruction is modeled as general compressed sensing (CS) based optimization on a low-rank tensor manifold. The nonlinear manifold is designed to characterize the temporal correlation of dynamic signals. Iterative procedures can be obtained by solving the optimization model on the manifold, including gradient calculation, projection of the gradient to tangent space, and retraction of the tangent space to the manifold. The iterative procedures on the manifold are unrolled to a neural network, dubbed as Manifold-Net. The Manifold-Net is trained using in vivo data with a retrospective electrocardiogram (ECG)-gated segmented bSSFP sequence. Results: Experimental results at high accelerations demonstrate that the proposed method can obtain improved reconstruction compared with a compressed sensing (CS) method k-t SLR and two state-of-the-art deep learning-based methods, DC-CNN and CRNN. Conclusion: This work represents the first study unrolling the optimization on manifolds into neural networks. Specifically, the designed low-rank manifold provides a new technical route for applying low-rank priors in dynamic MR imaging.

A low-rank method for two-dimensional time-dependent radiation transport calculations

The low-rank approximation is a complexity reduction technique to approximate a tensor or a matrix with a reduced rank, which has been applied to the simulation of high dimensional problems to reduce the memory required and computational cost. In this work, a dynamical low-rank approximation method is developed for the time-dependent radiation transport equation in 1-D and 2-D Cartesian geometries. Using a finite volume discretization in space and a spherical harmonics basis in angle, we construct a system that evolves on a low-rank manifold via an operator splitting approach. Numerical results on five test problems demonstrate that the low-rank solution requires less memory and computational time than solving the full rank equations with the same accuracy. It is furthermore shown that the low-rank algorithm can obtain high-fidelity results by increasing the number of basis functions while keeping the rank fixed.

Information Constrained Control Analysis of Eye Gaze Distribution Under Workload

We describe a novel model of human eye gaze behavior under workload, derived from the basic principle of information constrained control. The model assumes two distributions over the visual field: A saliency distribution, which is nongoal oriented, and a reward task-related distribution. The eye gaze behavior is determined by the tradeoff between these two distributions, where the goal is to preserve the task-related constraints, while remaining as close as possible to the saliency distribution representing a comfort zone. Based on minimum Kullback–Liebler divergence principles, the model gives rise to a family of gaze distributions controlled by a single tradeoff parameter. The model was evaluated experimentally in a driving simulator that consisted of an immersive environment with clear tasks and accurate monitoring capabilities. The findings confirm the theoretical predictions with respect to the low rank manifold and order relations in the data. We show that the model can be used to visualize the unknown reward function associated with a task, and predict human workload based on gaze pattern.

A low-rank projector-splitting integrator for the Vlasov–Maxwell equations with divergence correction

The Vlasov–Maxwell equations are used for the kinetic description of magnetized plasmas. As they are posed in an up to 3+3 dimensional phase space, solving this problem is extremely expensive from a computational point of view. In this paper, we exploit the low-rank structure in the solution of the Vlasov equation. More specifically, we consider the Vlasov–Maxwell system and propose a dynamic low-rank integrator. The key idea is to approximate the dynamics of the system by constraining it to a low-rank manifold. This is accomplished by a projection onto the tangent space. There, the dynamics is represented by the low-rank factors, which are determined by solving lower-dimensional partial differential equations. The proposed scheme performs well in numerical experiments and succeeds in capturing the main features of the plasma dynamics. We demonstrate this good behavior for a range of test problems. The coupling of the Vlasov equation with the Maxwell system, however, introduces additional challenges. In particular, the divergence of the electric field resulting from Maxwell's equations is not consistent with the charge density computed from the Vlasov equation. We propose a correction based on Lagrange multipliers which enforces Gauss' law up to machine precision.

A Geometric Approach to Dynamical Model Order Reduction

Any model order reduced dynamical system that evolves a modal decomposition to approximate the discretized solution of a stochastic PDE can be related to a vector field tangent to the manifold of fixed rank matrices. The dynamically orthogonal (DO) approximation is the canonical reduced-order model for which the corresponding vector field is the orthogonal projection of the original system dynamics onto the tangent spaces of this manifold. The embedded geometry of the fixed rank matrix manifold is thoroughly analyzed. The curvature of the manifold is characterized and related to the smallest singular value through the study of the Weingarten map. Differentiability results for the orthogonal projection onto embedded manifolds are reviewed and used to derive an explicit dynamical system for tracking the truncated singular value decomposition (SVD) of a time-dependent matrix. It is demonstrated that the error made by the DO approximation remains controlled under the minimal condition that the original solution stays close to the low rank manifold, which translates into an explicit dependence of this error on the gap between singular values. The DO approximation is also justified as the dynamical system that applies instantaneously the SVD truncation to optimally constrain the rank of the reduced solution. Riemannian matrix optimization is investigated in this extrinsic framework to provide algorithms that adaptively update the best low rank approximation of a smoothly varying matrix. The related gradient flow provides a dynamical system that converges to the truncated SVD of an input matrix for almost every initial datum.

Low-rank manifold optimization for overlay variations in lithography process

Overlay variations occur frequently in lithography process, which should be controlled within the tolerance to guarantee the better pattern resolutions. The operational optimization of overlay aims to predict the unknown overlay variations and compensate them into the wafer production. Due to the uncertain yield, the overlay data for learning are usually incomplete, which makes the overlay optimization very challenging. This paper proposes a novel overlay optimization framework called low-rank manifold optimization (LRMO), which provides new insight to address incomplete overlay data via exploiting low-rank property. First, LRMO can use effectively the correlations from incomplete overlay data, which builds a low-rank model for overlay optimization. In addition, LRMO resorts to Riemannian optimization and designs an efficient algorithm for this low-rank model. The proposed LRMO algorithm analyzes the manifold structure of the overlay data and computes accurate overlay variations with a low computational complexity. The experiments validate that LRMO obtains satisfying performance on the operational optimization of overlay variations.

Riemannian Submanifold Tracking on Low-Rank Algebraic Variety

Matrix recovery aims to learn a low-rank structure from high dimensional data, which arises in numerous learning applications. As a popular heuristic to matrix recovery, convex relaxation involves iterative calling of singular value decomposition (SVD). Riemannian optimization based method can alleviate such expensive cost in SVD for improved scalability, which however is usually degraded by the unknown rank. This paper proposes a novel algorithm RIST that exploits the algebraic variety of low-rank manifold for matrix recovery. Particularly, RIST utilizes an efficient scheme that automatically estimate the potential rank on the real algebraic variety and tracks the favorable Riemannian submanifold. Moreover, RIST utilizes the second-order geometric characterization and achieves provable superlinear convergence, which is superior to the linear convergence of most existing methods. Extensive comparison experiments demonstrate the accuracy and ef- ficiency of RIST algorithm.

Discretized Dynamical Low-Rank Approximation in the Presence of Small Singular Values

Low-rank approximations to large time-dependent matrices and tensors are the subject of this paper. These matrices and tensors either are given explicitly or are the unknown solutions of matrix and tensor differential equations. Based on splitting the orthogonal projection onto the tangent space of the low-rank manifold, novel time integrators for obtaining approximations by low-rank matrices and low-rank tensor trains were recently proposed. By standard theory, the Lie--Trotter and Strang projector-splitting methods are first and second order accurate, respectively, but the usual error bounds break down when the low-rank approximation has small singular values. This happens when the singular values of the solution decay without a distinct gap or when the effective rank of the solution is overestimated. On the other hand, the integrators are exact when given time-dependent matrices or tensors are already of the prescribed rank. We provide an error analysis which unifies these properties. We show that in c...

A projector-splitting integrator for dynamical low-rank approximation

The dynamical low-rank approximation of time-dependent matrices is a low-rank factorization updating technique. It leads to differential equations for factors of the matrices, which need to be solved numerically. We propose and analyze a fully explicit, computationally inexpensive integrator that is based on splitting the orthogonal projector onto the tangent space of the low-rank manifold. As is shown by theory and illustrated by numerical experiments, the integrator enjoys robustness properties that are not shared by any standard numerical integrator. This robustness can be exploited to change the rank adaptively. Another application is in optimization algorithms for low-rank matrices where truncation back to the given low rank can be done efficiently by applying a step of the integrator proposed here.

A Variational Approach to Video Registration with Subspace Constraints

This paper addresses the problem of non-rigid video registration, or the computation of optical flow from a reference frame to each of the subsequent images in a sequence, when the camera views deformable objects. We exploit the high correlation between 2D trajectories of different points on the same non-rigid surface by assuming that the displacement of any point throughout the sequence can be expressed in a compact way as a linear combination of a low-rank motion basis. This subspace constraint effectively acts as a trajectory regularization term leading to temporally consistent optical flow. We formulate it as a robust soft constraint within a variational framework by penalizing flow fields that lie outside the low-rank manifold. The resulting energy functional can be decoupled into the optimization of the brightness constancy and spatial regularization terms, leading to an efficient optimization scheme. Additionally, we propose a novel optimization scheme for the case of vector valued images, based on the dualization of the data term. This allows us to extend our approach to deal with colour images which results in significant improvements on the registration results. Finally, we provide a new benchmark dataset, based on motion capture data of a flag waving in the wind, with dense ground truth optical flow for evaluation of multi-frame optical flow algorithms for non-rigid surfaces. Our experiments show that our proposed approach outperforms state of the art optical flow and dense non-rigid registration algorithms.