Low-rank Matrices Research Articles

Low-rank rotation matrices-based DOA estimation with a moored circular vector sensor array

Abstract Water currents affect circular vector sensor arrays (CVSAs) suspended from a moored platform, causing them to rotate underwater. This rotation alters the direction of sources within the array coordinate system over time. Traditional methods that rely on numerous snapshots often yield inaccurate results, particularly for faint sources. To improve the accuracy, we introduce a direction-finding technique for CVSAs that employs low-rank rotation matrices. The low-rank rotation matrices are constructed using the heading information of the CVSAs and the subregion array manifold vector matrices to achieve spatial focusing. When these matrices are applied to the measurement data, the resulting covariance matrix displays subspace characteristics similar to that of a stationary CVSA. Our performance analysis revealed that low-rank rotation matrices offer higher focusing gains than conventional rotation matrices. The method proposed in this study effectively improves the direction estimation performance for weak targets and extends the practical applicability of measurement techniques for rotating platforms. Both simulations and experiments confirm that our approach outperforms the modified traditional beamforming and other spatial focusing techniques in terms of resolution and precision. Notably, when the signal-to-noise ratio (SNR) is below -4 dB, the resolution for distinguishing between the two sources increases by more than 50%.

Graph total variation and low-rank regularization for heterogeneous change detection

Heterogeneous change detection (HCD) is challenging because different imaging mechanisms for various sensors make images difficult to compare directly. To address this problem, a graph-based regression algorithm is proposed for HCD, by leveraging the Graph Total Variation regularization and Low-Rank matrices decomposition (GTVLR). Utilizing graph signal processing (GSP) theory, a directed graph (digraph) model is employed to effectively represent the orientation and correlation information of images, thereby enabling direct comparison of heterogeneous data within the same domain after graph filtering. The GTVLR framework facilitates the decomposition of post-event images into regression and changed images. This decomposition ensures that the regression image mirrors the structure similarity of the pre-event image, while the changed image highlights areas of alteration, aiding in change detection. The model characterizes the piecewise smoothness and Low-Rank properties of data through GTV regularization and Low-Rank penalty, respectively. Moreover, by integrating the higher-order neighboring information within the digraph to refine the model. Experiments conducted on three real-world datasets and comparison with several state-of-the-art methods demonstrate the effectiveness of the proposed algorithm.

Advancing Persistent Character Generation: Comparative Analysis of Fine-Tuning Techniques for Diffusion Models

In the evolving field of artificial intelligence, fine-tuning diffusion models is crucial for generating contextually coherent digital characters across various media. This paper examines four advanced fine-tuning techniques: Low-Rank Adaptation (LoRA), DreamBooth, Hypernetworks, and Textual Inversion. Each technique enhances the specificity and consistency of character generation, expanding the applications of diffusion models in digital content creation. LoRA efficiently adapts models to new tasks with minimal adjustments, making it ideal for environments with limited computational resources. It excels in low VRAM contexts due to its targeted fine-tuning of low-rank matrices within cross-attention layers, enabling faster training and efficient parameter tweaking. DreamBooth generates highly detailed, subject-specific images but is computationally intensive and suited for robust hardware environments. Hypernetworks introduce auxiliary networks that dynamically adjust the model’s behavior, allowing for flexibility during inference and on-the-fly model switching. This adaptability, however, can result in slightly lower image quality. Textual Inversion embeds new concepts directly into the model’s embedding space, allowing for rapid adaptation to novel styles or concepts, but is less effective for precise character generation. This analysis shows that LoRA is the most efficient for producing high-quality outputs with minimal computational overhead. In contrast, DreamBooth excels in high-fidelity images at the cost of longer training. Hypernetworks provide adaptability with some tradeoffs in quality, while Textual Inversion serves as a lightweight option for style integration. These techniques collectively enhance the creative capabilities of diffusion models, delivering high-quality, contextually relevant outputs.

Robust adaptive beamforming for cylindrical uniform conformal arrays based on low-rank covariance matrix reconstruction

Recently, conformal arrays have attracted considerable interest because such arrays can provide reduced radar cross-section and increased angle coverage. In this article, we devise a robust adaptive beamforming (RAB) approach using cylindrical uniform conformal array (CUCA). Firstly, we derive the minimum variance distortionless response (MVDR) beamformer for the CUCA by utilizing the noise subspace of interference covariance matrix (ICM) and steering vector (SV) of the signal-of-interest (SOI). Subsequently, the ICM is reconstructed by estimating the noise-free covariance matrix of the CUCA outputs and the interference projection matrix. Specifically, the noise-free covariance matrix can be regarded as multiple low-rank covariance matrices, and each low-rank matrix is reconstructed by formulating a nuclear norm minimization (NNM) problem. With the reconstructed covariance matrix, the 2-D DOAs of sources are determined by employing 2-D MUSIC spectrum to form the interference projection matrix. In addition, the SOI SV is estimated by solving a quadratically constrained quadratic programming (QCQP) problem. Numerical results demonstrate that the proposed approach is obviously superior to the existing RAB techniques.

Quadratic Matrix Factorization With Applications to Manifold Learning.

Matrix factorization is a popular framework for modeling low-rank data matrices. Motivated by manifold learning problems, this paper proposes a quadratic matrix factorization (QMF) framework to learn the curved manifold on which the dataset lies. Unlike local linear methods such as the local principal component analysis, QMF can better exploit the curved structure of the underlying manifold. Algorithmically, we propose an alternating minimization algorithm to optimize QMF and establish its theoretical convergence properties. To avoid possible over-fitting, we then propose a regularized QMF algorithm and discuss how to tune its regularization parameter. Finally, we elaborate how to apply the regularized QMF to manifold learning problems. Experiments on a synthetic manifold learning dataset and three real-world datasets, including the MNIST handwritten dataset, a cryogenic electron microscopy dataset, and the Frey Face dataset, demonstrate the superiority of the proposed method over its competitors.

Leveraged Matrix Completion With Noise.

Completing low-rank matrices from subsampled measurements has received much attention in the past decade. Existing works indicate that O(nrlog2(n)) datums are required to theoretically secure the completion of an n ×n noisy matrix of rank r with high probability, under some quite restrictive assumptions: 1) the underlying matrix must be incoherent and 2) observations follow the uniform distribution. The restrictiveness is partially due to ignoring the roles of the leverage score and the oracle information of each element. In this article, we employ the leverage scores to characterize the importance of each element and significantly relax assumptions to: 1) not any other structure assumptions are imposed on the underlying low-rank matrix and 2) elements being observed are appropriately dependent on their importance via the leverage score. Under these assumptions, instead of uniform sampling, we devise an ununiform/biased sampling procedure that can reveal the "importance" of each observed element. Our proofs are supported by a novel approach that phrases sufficient optimality conditions based on the Golfing scheme, which would be of independent interest to the wider areas. Theoretical findings show that we can provably recover an unknown n×n matrix of rank r from just about O(nrlog2 (n)) entries, even when the observed entries are corrupted with a small amount of noisy information. The empirical results align precisely with our theories.

Hidden Convexity, Optimization, and Algorithms on Rotation Matrices

This paper studies hidden convexity properties associated with constrained optimization problems over the set of rotation matrices [Formula: see text]. Such problems are nonconvex because of the constraint [Formula: see text]. Nonetheless, we show that certain linear images of [Formula: see text] are convex, opening up the possibility for convex optimization algorithms with provable guarantees for these problems. Our main technical contributions show that any two-dimensional image of [Formula: see text] is convex and that the projection of [Formula: see text] onto its strict upper triangular entries is convex. These results allow us to construct exact convex reformulations for constrained optimization problems over [Formula: see text] with a single constraint or with constraints defined by low-rank matrices. Both of these results are maximal in a formal sense. Funding: A. Ramachandran was supported by the H2020 program of the European Research Council [Grant 805241-QIP]. A. L. Wang was supported by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek [Grant OCENW.GROOT.2019.015 (OPTIMAL)]. K. Shu was supported by the Georgia Institute of Technology (ACO-ARC fellowship).

Optimizing a transformer-based network for a deep-learning seismic processing workflow

StorSeismic is a recently introduced model based on the transformer network to adapt to various seismic processing tasks through its pretraining and fine-tuning strategy. In the original implementation, StorSeismic uses a sinusoidal positional encoding (PE) and a conventional self-attention mechanism, borrowed from natural language processing applications. For seismic processing, they provide good results but also indicate limitations in efficiency and expressiveness. We develop modifications to these two key components, by using relative PE and low-rank attention matrices as replacements for the standard ones. Our changes are tested on processing tasks applied to realistic Marmousi and offshore field data as a sequential strategy, starting from denoising, direct-arrival removal, multiple attenuation, and finally root-mean-squared velocity ([Formula: see text]) prediction for normal moveout correction. We observe faster pretraining and competitive results on the fine-tuning tasks and, in addition, fewer parameters to train compared with the standard model.

The Hadamard decomposition problem

AbstractWe introduce the Hadamard decomposition problem in the context of data analysis. The problem is to represent exactly or approximately a given matrix as the Hadamard (or element-wise) product of two or more low-rank matrices. The motivation for this problem comes from situations where the input matrix has a multiplicative structure. The Hadamard decomposition has potential for giving more succint but equally accurate representations of matrices when compared with the gold-standard of singular value decomposition (svd). Namely, the Hadamard product of two rank-$$h$$ h matrices can have rank as high as $${h}^2$$ h 2 . We study the computational properties of the Hadamard decomposition problem and give gradient-based algorithms for solving it approximately. We also introduce a mixed model that combines svd and Hadamard decomposition. We present extensive empirical results comparing the approximation accuracy of the Hadamard decomposition with that of the svd using the same number of basis vectors. The results demonstrate that the Hadamard decomposition is competitive with the svd and, for some datasets, it yields a clearly higher approximation accuracy, indicating the presence of multiplicative structure in the data.

Approximation Conjugate Gradient Method for Low-Rank Matrix Recovery

Large-scale symmetric and asymmetric matrices have emerged in predicting the relationship between genes and diseases. The emergence of large-scale matrices increases the computational complexity of the problem. Therefore, using low-rank matrices instead of original symmetric and asymmetric matrices can greatly reduce computational complexity. In this paper, we propose an approximation conjugate gradient method for solving the low-rank matrix recovery problem, i.e., the low-rank matrix is obtained to replace the original symmetric and asymmetric matrices such that the approximation error is the smallest. The conjugate gradient search direction is given through matrix addition and matrix multiplication. The new conjugate gradient update parameter is given by the F-norm of matrix and the trace inner product of matrices. The conjugate gradient generated by the algorithm avoids SVD decomposition. The backtracking linear search is used so that the approximation conjugate gradient direction is computed only once, which ensures that the objective function decreases monotonically. The global convergence and local superlinear convergence of the algorithm are given. The numerical results are reported and show the effectiveness of the algorithm.

Low-rank matrices, tournaments, and symmetric designs

Let a=(ai)i≥1 be a sequence in a field F, and f:F×F→F be a function such that f(ai,ai)≠0 for all i≥1. For any tournament T over [n], consider the n×n symmetric matrix MT with zero diagonal whose (i,j)th entry (for i<j) is f(ai,aj) if i→j in T, and f(aj,ai) if j→i in T. It is known [3] that if T is a uniformly random tournament over [n], then rank(MT)≥(12−o(1))n with high probability when char(F)≠2 and f is a linear function.In this paper, we investigate the other extremal question: how low can the ranks of such matrices be? We work with sequences a that take only two distinct values, so the rank of any such n×n matrix is at least n/2. First, we show that the rank of any such matrix depends on whether an associated bipartite graph has certain eigenvalues of high multiplicity. Using this, we show that if f is linear, then there are real matrices MT(f;a) of rank at most n2+O(1). For rational matrices, we show that for each ε>0 we can find a sequence a(ε) for which there are matrices MT(f;a) of rank at most (12+ε)n+O(1). These matrices are constructed from symmetric designs, and we also use them to produce bisection-closed families of size greater than ⌊3n/2⌋−2 for n≤15, which improves the previously best known bound given in [5].

Singular value thresholding two-stage matrix completion for drug sensitivity discovery

Incomplete data presents significant challenges in drug sensitivity analysis, especially in critical areas like oncology, where precision is paramount. Our study introduces an innovative imputation method designed specifically for low-rank matrices, addressing the crucial challenge of data completion in anticancer drug sensitivity testing. Our method unfolds in two main stages: Initially, the singular value thresholding algorithm is employed for preliminary matrix completion, establishing a solid foundation for subsequent steps. Then, the matrix rows are segmented into distinct blocks based on hierarchical clustering of correlation coefficients, applying singular value thresholding to the largest block, which has been proved to possess the largest entropy. This is followed by a refined data restoration process, where the reconstructed largest block is integrated into the initial matrix completion to achieve the final matrix completion. Compared to other methods, our approach not only improves the accuracy of data restoration but also ensures the integrity and reliability of the imputed values, establishing it as a robust tool for future drug sensitivity analysis.

Community Detection in Multiplex Networks Using Orthogonal Non-Negative Matrix Tri-Factorization Based on Graph Regularization and Diversity

In recent years, community detection has received increasing interest. In network analysis, community detection refers to the identification of tightly connected subsets of nodes, which are called “communities” or “groups”, in the network. Non-negative matrix factorization models are often used to solve the problem. Orthogonal non-negative matrix tri-factorization (ONMTF) exhibits significant potential as an approach for community detection within multiplex networks. This paper explores the application of ONMTF in multiplex networks, aiming to detect both shared and exclusive communities simultaneously. The model decomposes each layer within the multiplex network into two low-rank matrices. One matrix corresponds to shared communities across all layers, and the other to unique communities within each layer. Additionally, graph regularization and the diversity of private communities are taken into account in the algorithm. The Hilbert Schmidt Independence Criterion (HSIC) is used to constrain the independence of private communities. The results prove that ONMTF effectively addresses community detection in multiplex networks. It also offers strong interpretability and feature extraction capabilities. Therefore, it is an advanced method for community detection in multiplex networks.

A Designed Thresholding Operator for Low-Rank Matrix Completion

In this paper, a new thresholding operator, namely, designed thresholding operator, is designed to recover the low-rank matrices. With the change of parameter in designed thresholding operator, the designed thresholding operator can apply less bias to the larger singular values of a matrix compared with the classical soft thresholding operator. Based on the designed thresholding operator, an iterative thresholding algorithm for recovering the low-rank matrices is proposed. Numerical experiments on some image inpainting problems show that the proposed thresholding algorithm performs effectively in recovering the low-rank matrices.

HYPERSPECTRAL ANOMALY DETECTION WITH AN IMPROVED APPROACH: INTEGRATION OF GO DECOMPOSITION ALGORITHM AND LAPLACIAN MATRIX MODIFIER

In this study, a hyperspectral anomaly detection method based on Laplacian matrix (HADLAP) is proposed. This paper addresses the problem of determining covariance matrix inversion in high-dimensional data and proposes a new approach for identifying anomalies in hyperspectral images (HSIs). The study’s goals are to find anomalous locations in HSIs and to deal with the problem of calculating the inversion of the covariance matrix of high dimensional data. The method is centered on two main concepts. One of them is decomposition process. The other one is detection process. First, HSI data is decomposed as a low rank and sparse matrices. Second, the sparse component of the data is used to build Mahalanobis Distance (MD). In this study, go decomposition (GoDec) algorithm is employed to decompose the data. Then, the distance is calculated by obtained matrix with aim of detection of anomalous pixels in the HSIs. The method differs from previous studies that covariance matrix in the distance is computed with Laplacian matrix and MD. Experiments conducted on three hyperspectral datasets present the superiority and effectiveness of the proposed framework in terms of detection performance with respect to state-of-the-art methods.

Numerical solution of the boundary value problems for the biharmonic equations via quasiseparable representations

AbstractThe paper incorporates new methods of numerical linear algebra for the approximation of the biharmonic equation with potential, namely, numerical solution of the Dirichlet problem for $$ \left( \frac{d}{dx}\right) ^4u(x)+c(x)u(x)=\phi (x),\quad 0&lt;x&lt;1. $$ d dx 4 u ( x ) + c ( x ) u ( x ) = ϕ ( x ) , 0 &lt; x &lt; 1 . High-order discrete finite difference operators are presented, constructed on the basis of discrete Hermitian derivatives, and the associated Discrete Biharmonic Operator (DBO). It is shown that the matrices associated with the discrete operator belong to a class of quasiseparable matrices of low rank matrices. The application of quasiseparable representation of rank structured matrices yields fast and stable algorithm for variable potentials c(x). Numerical examples corroborate the claim of high order accuracy of the algorithm, with optimal complexity O(N).

Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

Post-training quantization (PTQ) has emerged as a promising technique for mitigating memory consumption and computational costs in large language models (LLMs). However, a systematic examination of various quantization schemes, model families, and quantization bit precision has been absent from the literature. In this paper, we conduct a comprehensive analysis of these factors by investigating the effects of PTQ on weight-only, activation-only, and weight-and-activation quantization using diverse methods such as round-to-nearest (RTN), GPTQ, ZeroQuant, and their variants. We apply these methods to two distinct model families with parameters ranging from 125M to 176B. Our contributions include: (1) a sensitivity analysis revealing that activation quantization is generally more susceptible to weight quantization, with smaller models often outperforming larger models in terms of activation quantization; (2) an evaluation and comparison of existing PTQ methods to optimize model size reduction while minimizing the impact on accuracy, revealing that none of the current methods can achieve the original model quality for quantization with either INT4-weight or INT4-weight-and-INT8-activation; (3) based on these insights, we propose an optimized method called Low-Rank Compensation (LoRC), which employs low-rank matrices to enhance model quality recovery with a minimal increase in model size.

Fixing Inventory Inaccuracies at Scale

Problem definition: Inaccurate records of inventory occur frequently and, by some measures, cost retailers approximately 4% in annual sales. Detecting inventory inaccuracies manually is cost-prohibitive, and existing algorithmic solutions rely almost exclusively on learning from longitudinal data, which is insufficient in the dynamic environment induced by modern retail operations. Instead, we propose a solution based on cross-sectional data over stores and stock-keeping units (SKUs), viewing inventory inaccuracies as a problem of identifying anomalies in a (low-rank) Poisson matrix. State-of-the-art approaches to anomaly detection in low-rank matrices apparently fall short. Specifically, from a theoretical perspective, recovery guarantees for these approaches require that nonanomalous entries be observed with vanishingly small noise (which is not the case in our problem and, indeed, in many applications). Methodology/results: So motivated, we propose a conceptually simple entrywise approach to anomaly detection in low-rank Poisson matrices. Our approach accommodates a general class of probabilistic anomaly models. We show that the cost incurred by our algorithm approaches that of an optimal algorithm at a min-max optimal rate. Using synthetic data and real data from a consumer goods retailer, we show that our approach provides up to a 10× cost reduction over incumbent approaches to anomaly detection. Along the way, we build on recent work that seeks entrywise error guarantees for matrix completion, establishing such guarantees for subexponential matrices, a result of independent interest. Managerial implications: By utilizing cross-sectional data at scale, our novel approach provides a practical solution to the issue of inventory inaccuracies in retail operations. Our method is cost-effective and can help managers detect inventory inaccuracies quickly, leading to increased sales and improved customer satisfaction. In addition, the entrywise error guarantees that we establish are of interest to academics working on matrix-completion problems. History: This paper was selected for Fast Track in M&amp;SOM from the 2022 MSOM Supply Chain Management SIG Conference. Funding: Financial support from the National Science Foundation Division of Civil, Mechanical, and Manufacturing Innovation [Grant CMMI 1727239] is gratefully acknowledged. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2023.0146 .

Quantum Communication Complexity of Linear Regression

Quantum computers may achieve speedups over their classical counterparts for solving linear algebra problems. However, in some cases—such as for low-rank matrices—dequantized algorithms demonstrate that there cannot be an exponential quantum speedup. In this work, we show that quantum computers have provable polynomial and exponential speedups in terms of communication complexity for some fundamental linear algebra problems if there is no restriction on the rank. We mainly focus on solving linear regression and Hamiltonian simulation. In the quantum case, the task is to prepare the quantum state of the result. To allow for a fair comparison, in the classical case, the task is to sample from the result. We investigate these two problems in two-party and multiparty models, propose near-optimal quantum protocols, and prove quantum/classical lower bounds. In this process, we propose an efficient quantum protocol for quantum singular value transformation, which is a powerful technique for designing quantum algorithms. We feel this will be helpful in developing efficient quantum protocols for many other problems.

The effect of smooth parametrizations on nonconvex optimization landscapes

AbstractWe develop new tools to study landscapes in nonconvex optimization. Given one optimization problem, we pair it with another by smoothly parametrizing the domain. This is either for practical purposes (e.g., to use smooth optimization algorithms with good guarantees) or for theoretical purposes (e.g., to reveal that the landscape satisfies a strict saddle property). In both cases, the central question is: how do the landscapes of the two problems relate? More precisely: how do desirable points such as local minima and critical points in one problem relate to those in the other problem? A key finding in this paper is that these relations are often determined by the parametrization itself, and are almost entirely independent of the cost function. Accordingly, we introduce a general framework to study parametrizations by their effect on landscapes. The framework enables us to obtain new guarantees for an array of problems, some of which were previously treated on a case-by-case basis in the literature. Applications include: optimizing low-rank matrices and tensors through factorizations; solving semidefinite programs via the Burer–Monteiro approach; training neural networks by optimizing their weights and biases; and quotienting out symmetries.