Low-rank Assumption Research Articles

Sample efficient nonparametric regression via low-rank regularization

Nonparametric regression suffers from curse of dimensionality, requiring a relatively large sample size for accurate estimation beyond the univariate case. In this paper, we consider a simple method of dimension reduction in nonparametric regression via series estimation, based on the concept of low-rankness which was previously studied in parametric multivariate reduced-rank regression and matrix regression. For d > 2 , the low-rank assumption is realized via tensor regression. We establish a faster convergence rate of the estimator in the (approximate) low-rank case. Limitations of the model are also discussed. Through simulation studies and real data analysis, we compare the estimation accuracy of the proposed method with that of existing approaches. The results demonstrate that the proposed method yields estimates with lower RMSE compared to existing methods.

Compact lossy compression of tensors via neural tensor-train decomposition

AbstractMany real-world datasets are represented as tensors, i.e., multi-dimensional arrays of numerical values. Storing them without compression often requires substantial space, which grows exponentially with the order. While many tensor compression algorithms are available, many of them rely on strong data assumptions regarding its order, sparsity, rank, and smoothness. In this work, we propose TensorCodec, a lossy compression algorithm for general tensors that do not necessarily adhere to strong input data assumptions.TensorCodec incorporates three key ideas. The first idea is neural tensor-train decomposition (NTTD) where we integrate a recurrent neural network into Tensor-Train Decomposition to enhance its expressive power and alleviate the limitations imposed by the low-rank assumption. Another idea is to fold the input tensor into a higher-order tensor to reduce the space required by NTTD. Finally, the mode indices of the input tensor are reordered to reveal patterns that can be exploited by NTTD for improved approximation. In addition, we extend TensorCodec to enable the lossy compression of tensors with missing entries, often found in real-world datasets. Our analysis and experiments on 8 real-world datasets demonstrate that TensorCodec is (a) Concise: it gives up to $$7.38 \times $$ 7.38 × more compact compression than the best competitor with similar reconstruction error, (b) Accurate: given the same budget for compressed size, it yields up to $$3.33\times $$ 3.33 × more accurate reconstruction than the best competitor, (c) Scalable: Its empirical compression time is linear in the number of tensor entries, and it reconstructs each entry in logarithmic time. Our code and datasets are available at https://github.com/kbrother/TensorCodec.

CP decomposition-based algorithms for completion problem of motion capture data

Motion capture (MoCap) technology is an essential tool for recording and analyzing movements of objects or humans. However, MoCap systems frequently encounter the challenge of missing data, stemming from mismatched markers, occlusion, or equipment limitations. Recovery of these missing data is imperative to maintain the reliability and integrity of MoCap recordings. This paper introduces a novel application of the tensor framework for MoCap data completion. We propose three completion algorithms based on the canonical polyadic (CP) decomposition of tensors. The first algorithm utilizes CP decomposition to capture the low-rank structure of the tensor. However, relying only on low-rank assumptions may be insufficient to deal with complex motion data. Thus, we propose two modified CP decompositions that incorporate additional information, SmoothCP and SparseCP decompositions. SmoothCP integrates piecewise smoothness prior, while SparseCP incorporates sparsity prior, each aiming to improve the accuracy and robustness of MoCap data recovery. To compare and evaluate the merit of the proposed algorithms over other tensor completion methods in terms of several evaluation metrics, we conduct numerical experiments with different MoCap sequences from the CMU motion capture dataset.

Exploiting indirect linear correlation for label distribution learning

Label distribution learning represents the relevance of labels to samples using description degree, which can provide richer semantic information, thus finding wider applications. Exploiting label correlations is an effective approach to narrow down the hypothesis space of label distribution learning models. In existing works that utilize low-rank assumptions or label linear dependence to mine correlations, it is assumed that a label can be linearly expressed by other labels. However, this assumption can only be satisfied when there are linear dependency relationships between labels, thus the label correlation obtained by such methods is subject to certain distortion. To address this issue, this paper assumes that labels can be linearly represented by the same set of bases. The correlation between labels is represented by sharing common bases. Specifically, the paper employs matrix factorization to extract bases that can be used to represent all labels. And then designs a label distribution learning algorithm based on the property of sharing the same set of bases of the ground truth label distribution and predict label distribution. The effectiveness of the algorithm is verified through experimental validation. Generally speaking, the algorithm presented in this paper achieves optimal performance at 73.15% of the cases, with the best average ranking. In the two-tailed t-test, the algorithm in this paper exhibits statistical superiority compared to all comparison algorithms.

Stability estimates for the holonomy inverse problem

On a Riemannian manifold (M, g) with Anosov geodesic flow, the problem of recovering a connection from the knowledge of traces of its holonomies along primitive closed geodesics is known as the holonomy inverse problem. In this paper, we prove Hölder type stability estimates for this inverse problem: locally, near generic connections; globally, for line bundles, and for vector bundles satisfying a certain low-rank assumption over negatively curved base (M, g). The proofs are based on a combination of microlocal analysis along with a new non-Abelian approximate Livšic Theorem in hyperbolic dynamics.

On Low-Rank Directed Acyclic Graphs and Causal Structure Learning.

Despite several advances in recent years, learning causal structures represented by directed acyclic graphs (DAGs) remains a challenging task in high-dimensional settings when the graphs to be learned are not sparse. In this article, we propose to exploit a low-rank assumption regarding the (weighted) adjacency matrix of a DAG causal model to help address this problem. We utilize existing low-rank techniques to adapt causal structure learning methods to take advantage of this assumption and establish several useful results relating interpretable graphical conditions to the low-rank assumption. Specifically, we show that the maximum rank is highly related to hubs, suggesting that scale-free (SF) networks, which are frequently encountered in practice, tend to be low rank. Our experiments demonstrate the utility of the low-rank adaptations for a variety of data models, especially with relatively large and dense graphs. Moreover, with a validation procedure, the adaptations maintain a superior or comparable performance even when graphs are not restricted to be low rank.

A higher Gross–Zagier formula and the structure of Selmer groups

We describe a Kolyvagin system-theoretic refinement of Gross–Zagier formula by comparing Heegner point Kolyvagin systems with Kurihara numbers when the root number of a rational elliptic curve E E over an imaginary quadratic field K K is − 1 -1 . When the root number of E E over K K is 1, we first establish the structure theorem of the p ∞ p^\infty -Selmer group of E E over K K . The description is given by the values of certain families of quaternionic automorphic forms, which is a part of bipartite Euler systems. By comparing bipartite Euler systems with Kurihara numbers, we also obtain an analogous refinement of Waldspurger formula. No low analytic rank assumption is imposed in both refinements. We also prove the equivalence between the non-triviality of various “Kolyvagin systems” and the corresponding main conjecture localized at the augmentation ideal. As consequences, we obtain new applications of (weaker versions of) the Heegner point main conjecture and the anticyclotomic main conjecture to the structure of p ∞ p^\infty -Selmer groups of elliptic curves of arbitrary rank. In particular, the Heegner point main conjecture localized at the augmentation ideal implies the strong rank one p p -converse to the theorem of Gross–Zagier and Kolyvagin.

Unsupervised graph denoising via feature-driven matrix factorization

Graph denoising is becoming a promising solution for robust graph embedding, which aims to construct an ideal network by removing noisy edges. Currently, the mainstream denoising approaches often build upon the low-rank assumption by removing high-rank harmful edges with singular value decomposition. Although this approach often allows yielding good graph embedding, the high time complexity limits its application to large-scale networks. In this paper, we propose an effective and efficient algorithm for robust graph embedding: Latent Feature-driven Graph Denoising (LFGD). The basic idea is to leverage node latent features to construct an ideal low-rank network by exploiting the relationship between topology and feature information. To this end, the original features are first mapped into a latent space, and then a low-rank network is reconstructed by imposing the semantic preservation loss and structure loss (including regression loss and sparsity constraint). In addition, we propose a sampling-based strategy to further speed up the proposed method, which finally results in linear time complexity. LFGD works independently of downstream task, which makes the denoised structure more general and reliable. Extensive experiments have demonstrated the superiority of the LFGD to many state-of-the-art algorithms under various attacks in terms of node classification, robustness and running time. Our code is available at: https://github.com/hanwangme/LFGD.

A Tensor-Based Online RPCA Model for Compressive Background Subtraction.

Background subtraction of videos has been a fundamental research topic in computer vision in the past decades. To alleviate the computation burden and enhance the efficiency, background subtraction from online compressive measurements has recently attracted much attention. However, current methods still have limitations. First, they are all based on matrix modeling, which breaks the spatial structure within video frames. Second, they generally ignore the complex disturbance within the background, which reduces the efficiency of the low-rank assumption. To alleviate this issue, we propose a tensor-based online compressive video reconstruction and background subtraction method, abbreviated as NIOTenRPCA, by explicitly modeling the background disturbance in different frames as nonidentical but correlated noise. By virtue of such sophisticated modeling, the proposed method can well adapt to complex video scenes and, thus, perform more robustly. Extensive experiments on a series of real-world video datasets have demonstrated the effectiveness of the proposed method compared with the existing state of the arts. The code of our method is released on the website: https://github.com/crystalzina/NIOTenRPCA.

Superpixel-based robust tensor low-rank approximation for multimedia data recovery

Recently, the low-rank matrix/tensor-based methods have attracted increasing attention for multi-dimensional multimedia data (e.g., image and video) recovery, which assumes the holistic data is low-rank. For most multimedia data, the low-rank assumption is usually violated due to their spatially diverse local similarity. The cube-based method, which breaks the holistic image into small local cubes with a fixed size, can alleviate this issue to some extent. However, cubes with a fixed size are not flexible to cover the regions of local similarity at different scales. Inspired by the superpixel, we suggest the size-adaptive super-tensor as the generic unit instead of the cube with the fixed spatial size, which allows us to flexibly exploit the local similarity at different scales. The super-tensor enjoys significant low-rankness and avoids excessive overlapped pixels as compared with cubes of fixed sizes. Empowering with the super-tensor, we propose a low-rank super-tensor approximation (LRSTA) model for multi-dimensional data recovery, which can fully exploit the local similarity and low-rankness at different scales. Moreover, we develop an efficient alternating direction method of multipliers (ADMM) algorithm to solve the proposed model. Extensive experiments on multimedia data, including color images, multispectral images (MSIs), and videos, verify that the proposed method outperforms the other competing methods.

Image inpainting via Smooth Tucker decomposition and Low-rank Hankel constraint

Image inpainting, aiming at exactly recovering missing pixels from partially observed entries, is typically an ill-posed problem. As a powerful constraint, low-rank priors have been widely applied in image inpainting to transform such problems into well-posed ones. However, the low-rank assumption of original visual data is only in an approximate mode, which in turn results in suboptimal recovery of fine-grained details, particularly when the missing rate is extremely high. Moreover, a single prior cannot faithfully capture the complex texture structure of an image. In this paper, we propose a joint usage of Smooth Tucker decomposition and Low-rank Hankel constraint (STLH) for image inpainting, which enables simultaneous capturing of the global low-rankness and local piecewise smoothness. Specifically, based on the Hankelization operation, the original image is mapped to a high-order structure for capturing more spatial and spectral information. By employing Tucker decomposition for optimizing the Hankel tensor and simultaneously applying Discrete Total Variation (DTV) to the Tucker factors, sharper edges are generated and better isotropic properties are enhanced. Moreover, to approximate the essential rank of the Tucker decomposition and avoid facing the uncertainty problem of the upper-rank limit, a reverse strategy is adopted to approximate the true rank of the Tucker decomposition. Finally, the overall image inpainting model is optimized by the well-known alternate least squares (ALS) algorithm. Extensive experiments show that the proposed method achieves state-of-the-art performance both quantitatively and qualitatively. Particularly, in the extreme case with 99% pixels missed, the results from STLH are averagely ahead of others at least 0.9dB in terms of PSNR values.

Low-Rank Hankel Tensor Completion for Traffic Speed Estimation

This paper studies the traffic state estimation (TSE) problem using sparse observations from mobile sensors. Most existing TSE methods either rely on well-defined physical traffic flow models or require large amounts of simulation data as input to train learning algorithms. Different from previous studies, in this paper we propose a purely data-driven and model-free solution. We consider TSE as a spatiotemporal matrix completion/interpolation problem and apply spatiotemporal delay embedding to transform the original incomplete matrix into a fourth-order Hankel structured tensor. By imposing a low-rank assumption on this tensor structure, we can approximate and characterize both global patterns and local patterns in a data-driven manner. We use a truncated nuclear norm of a balanced spatiotemporal unfolding to approximate the tensor rank and develop an efficient algorithm based on the Alternating Direction Method of Multipliers (ADMM) to solve the problem. The proposed framework only involves two hyperparameters, spatial and temporal window lengths, which are easy to set given the degree of data sparsity. To validate the effectiveness of our proposed method, we conducted numerical experiments on real-world high-resolution trajectory data, which demonstrated its superiority in some challenging scenarios. The proposed method shows great potential for solving the TSE problem using sparse observations from mobile sensors and can be applied in various traffic applications.

Matrix Completion via Non-Convex Relaxation and Adaptive Correlation Learning.

The existing matrix completion methods focus on optimizing the relaxation of rank function such as nuclear norm, Schatten- p norm, etc. They usually need many iterations to converge. Moreover, only the low-rank property of matrices is utilized in most existing models and several methods that incorporate other knowledge are quite time-consuming in practice. To address these issues, we propose a novel non-convex surrogate that can be optimized by closed-form solutions, such that it empirically converges within dozens of iterations. Besides, the optimization is parameter-free and the convergence is proved. Compared with the relaxation of rank, the surrogate is motivated by optimizing an upper-bound of rank. We theoretically validate that it is equivalent to the existing matrix completion models. Besides the low-rank assumption, we intend to exploit the column-wise correlation for matrix completion, and thus an adaptive correlation learning, which is scaling-invariant, is developed. More importantly, after incorporating the correlation learning, the model can be still solved by closed-form solutions such that it still converges fast. Experiments show the effectiveness of the non-convex surrogate and adaptive correlation learning.

Label Distribution Learning by Exploiting Label Distribution Manifold.

Label correlation is helpful to alleviate the overwhelming output space of label distribution learning (LDL). However, existing studies either only consider one of global and local label correlations or exploit label correlation by some prior knowledge (e.g., low-rank assumption, which may not hold sometimes). To efficiently exploit both global and local label correlations in a data-driven way, we propose in this article a new LDL method called label distribution learning by exploiting label distribution manifold (LDL-LDM). Our basic idea is that the underlying manifold structure of label distribution may encode the correlations among labels. LDL-LDM works as follows. First, to exploit global label correlation, we learn the label distribution manifold and encourage the outputs of our model to lie in the same manifold. Second, we learn the label distribution manifold of different clusters of samples to consider local label correlations. Third, to handle incomplete label distribution learning (incomplete LDL), we jointly learn label distribution and label distribution manifold. Theoretical analysis demonstrates the generalization of our method. Finally, experimental results validate the effectiveness of LDL-LDM in both full and incomplete LDL cases.

Hyperspectral Image Mixed Noise Removal via Nonlinear Transform-Based Block-Term Tensor Decomposition

Recently, block-term decomposition with rank-( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L_r</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L_r</i> ,1) (termed as LL1 decomposition), which is physically inspired by linear spectral unmixing, has received increasing attention in hyperspectral images (HSIs) denoising. However, due to the intrinsic nonlinear structure of real-world HSIs, the low-rankness of HSIs is usually implicit. Moreover, the essential uniqueness guarantee is usually violated with the low-rank assumption of the abundance maps unsupported in real scenarios, which hampers the successful deployment of LL1 decomposition. Inspired by the nonlinear spectral unmixing, we propose a nonlinear learnable transform-based LL1 decomposition (NT-LL1) for characterizing the implicit low-rank structure of real-world HSIs. More concretely, the nonlinear learnable transform in NT-LL1 decomposition is a composed transform consisting of a linear semi-orthogonal transform and a component-wise nonlinear transform, which collaboratively enhances the low-rankness of the abundance maps. Empowering with the NT-LL1 decomposition, we propose an NT-LL1 decomposition-based model for HSIs denoising. To tackle the resulting model, we develop an efficient proximal alternating minimization-based algorithm with a convergence guarantee. Extensive experimental results including simulated and real data collectively verify the superiority of the proposed method as compared with the competing methods.

Characteristic-constrained accelerating MR T1rho mapping with blockwise infimal convolution of matrix elastic-net regularization.

Magnetic resonance parameter mapping (MRPM) plays an important role in clinical applications and biomedical researches. However, the acceleration of MRPM remains a major challenge for achieving furtherimprovements. In this work, a new undersampled k-space based joint multi-contrast image reconstruction approach named CC-IC-LMEN is proposed for accelerating MR T1rhomapping. The reconstruction formulation of the proposed CC-IC-LMEN method imposes a blockwise low-rank assumption on the characteristic-image series (c-p space) and utilizes infimal convolution (IC) to exploit and balance the generalized low-rank properties in low-and high-order c-p spaces, thereby improving the accuracy. In addition, matrix elastic-net (MEN) regularization based on the nuclear and Frobenius norms is incorporated to obtain stable and exact solutions in cases with large accelerations and noisy observations. This formulation results in a minimization problem, that can be effectively solved using a numerical algorithm based on the alternating direction method of multipliers (ADMM). Finally, T1rho maps are then generated according to the reconstructed images using nonlinear least-squares (NLSQ) curve fitting with an established relaxometrymodel. The relative l2 -norm error (RLNE) and structural similarity (SSIM) in the regions of interest (ROI) show that the CC-IC-LMEN approach is more accurate than other competing methods even in situations with heavy undersampling or noisyobservation. Our proposed CC-IC-LMEN method provides accurate and robust solutions for accelerated MR T1rhomapping.

Matrix Completion With Data-Dependent Missingness Probabilities

The problem of completing a large matrix with lots of missing entries has received widespread attention in the last couple of decades. Two popular approaches to the matrix completion problem are based on singular value thresholding and nuclear norm minimization. Most of the past works on this subject assume that there is a single number <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> such that each entry of the matrix is available independently with probability <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> and missing otherwise. This assumption may not be realistic for many applications. In this work, we replace it with the assumption that the probability that an entry is available is an unknown function <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$f$ </tex-math></inline-formula> of the entry itself. For example, if the entry is the rating given to a movie by a viewer, then it seems plausible that high value entries have greater probability of being available than low value entries. We propose two new estimators, based on singular value thresholding and nuclear norm minimization, to recover the matrix under this assumption. The estimators involve no tuning parameters, and are shown to be consistent under a low rank assumption. We also provide a consistent estimator of the unknown function <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$f$ </tex-math></inline-formula> .

Discrete subspace structure constrained human motion capture data recovery

Recently, low-rank matrix completion (LRMC) has shown a promising performance in human motion capture (mocap) data recovery, which assumes that the input data is low-rank. However, for most mocap data comprising multiple different activities, only using LRMC to restore the data may result in erroneous results, since the low-rank assumption is not necessarily satisfied. The current method solves this problem in a two-stage manner by combining the LRMC and subspace clustering. But it ignores the fact that the processes of recovering and clustering depend on each other, and so is sub-optimal. In this paper, we propose a novel discrete subspace structure (DSS) constrained approach to recover human mocap data, which jointly optimizing the tasks of subspace clustering and the LRMC. The proposed DSS algorithm learns different subspaces’ indicators such that the mocap recovery problem is divided into several LRMC subproblems, where each matrix of the points drawn from a single subspace is low-rank. As a byproduct, we can obtain the temporal clustering results of mocap data. To better approximate the low-rank property, we utilize the tth power of Schatten p-norm to approximate the rank instead of the nuclear norm. Moreover, we add two regularization terms to take care of the noise effect and temporal stability of mocap data. The obtained model comes down to a non-convex optimization problem, which we solve tactically and efficiently by employing the alternating direction method of multipliers (ADMM). Experiments on the CMU dataset and the HDM05 dataset validate the effectiveness of the proposed algorithm in both mocap data recovery and temporal subspace clustering.

Smoothed tensor quantile regression estimation for longitudinal data

As extensions of vector and matrix data with ultrahigh dimensionality and complex structures, tensor data are fast emerging in a large variety of scientific applications. In this paper, a two-stage estimation procedure for linear tensor quantile regression (QR) with longitudinal data is proposed. In the first stage, we account for within-subject correlations by using the generalized estimating equations and then impose a low-rank assumption on tensor coefficients to reduce the number of parameters by a canonical polyadic decomposition. To avoid the asymptotic analysis and computation problems caused by the non-smooth QR score function, kernel smoothing method is applied in the second stage to construct the smoothed tensor QR estimator. When the number of rank is given, a block-relaxation algorithm is proposed to estimate the regression coefficients. A modified BIC is applied to estimate the number of rank in practice and show the rank selection consistency. Further, a regularized estimator and its algorithm are investigated for better interpretation and efficiency. The asymptotic properties of the proposed estimators are established. Simulation studies and a real example on Beijing Air Quality data set are provided to show the performance of the proposed estimators.

Low-Rank Autoregressive Tensor Completion for Spatiotemporal Traffic Data Imputation

Spatiotemporal traffic time series (e.g., traffic volume/speed) collected from sensing systems are often incomplete with considerable corruption and large amounts of missing values, preventing users from harnessing the full power of the data. Missing data imputation has been a long-standing research topic and critical application for real-world intelligent transportation systems. A widely applied imputation method is low-rank matrix/tensor completion; however, the low-rank assumption only preserves the global structure while ignores the strong local consistency in spatiotemporal data. In this paper, we propose a low-rank autoregressive tensor completion (LATC) framework by introducing \textit{temporal variation} as a new regularization term into the completion of a third-order (sensor $\times$ time of day $\times$ day) tensor. The third-order tensor structure allows us to better capture the global consistency of traffic data, such as the inherent seasonality and day-to-day similarity. To achieve local consistency, we design the temporal variation by imposing an AR($p$) model for each time series with coefficients as learnable parameters. Different from previous spatial and temporal regularization schemes, the minimization of temporal variation can better characterize temporal generative mechanisms beyond local smoothness, allowing us to deal with more challenging scenarios such "blackout" missing. To solve the optimization problem in LATC, we introduce an alternating minimization scheme that estimates the low-rank tensor and autoregressive coefficients iteratively. We conduct extensive numerical experiments on several real-world traffic data sets, and our results demonstrate the effectiveness of LATC in diverse missing scenarios.