Low-rank Factorization Research Articles

A Feasible Method for Solving an SDP Relaxation of the Quadratic Knapsack Problem

In this paper, we consider a semidefinite programming (SDP) relaxation of the quadratic knapsack problem. After applying low-rank factorization, we get a nonconvex problem, whose feasible region is an algebraic variety with certain good geometric properties, which we analyze. We derive a rank condition under which these two formulations are equivalent. This rank condition is much weaker than the classical rank condition if the coefficient matrix has certain special structures. We also prove that, under an appropriate rank condition, the nonconvex problem has no spurious local minima without assuming linearly independent constraint qualification. We design a feasible method that can escape from nonoptimal nonregular points. Numerical experiments are conducted to verify the high efficiency and robustness of our algorithm as compared with other solvers. In particular, our algorithm is able to solve a one-million-dimensional sparse SDP problem accurately in about 20 minutes on a modest computer. Funding: The research of K.-C. Toh is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 3 grant call [Grant MOE-2019-T3-1-010].

Combined Deep Priors With Low-Rank Tensor Factorization for Hyperspectral Image Restoration

Mixed noise pollution severely disturbs hyperspectral image (HSI) processing and applications. Plenty of algorithms have been developed to address this issue via two strategies: model-driven or data-driven strategy. However, model-driven methods exist in the highly time-consuming weakness of iterative optimization and unstable sensitivity of setting parameters. Data-driven methods usually perform poor due to the overfitting effects. To solve these issues, we combine both the deep denoising priors with low-rank tensor factorization (DP-LRTF) for HSI restoration. The proposed method uses Tucker tensor factorization to depict the global spectral low-rank constraint. Then the spectral orthogonal basis and spatial reduced factor are optimized by two deep denoising priors, respectively. Through this integrated strategy, we can simultaneously exploit the intrinsic low-rank property of HSI, and utilize the powerful feature extraction ability by deep learning for HSI restoration. Compared with model-driven and data-driven methods, DP-LRTF outperforms on HSI mixed noise removal and execution efficiency for various simulated/real experiments.

Model distillation for high-level semantic understanding：a survey

Model distillation for high-level semantic understanding：a survey

Optimization Ways in Neural Network Compression

Deep learning is a powerful tool that uses simple representations to express complex ideas and allows computers to mine hidden information and value from experience. It has achieved great success in a variety of fields. However, with the development of deep learning models, the computing resources consumed by training the models increase rapidly, making it difficult to directly deploy the models to portable devices or embedded systems. Therefore, compressing and accelerating DNN models without sacrificing performance has become a crucial area of research in the field of deep learning. In many existing compression techniques, optimization theory and approaches play an important role in their research and implementation. In this paper, we focus on neural network compression from an optimization perspective and review related optimization strategies. Specifically, we summarize optimization techniques emerging from four general categories of commonly used network compression approaches, including network pruning, low-bit quantization, low-rank factorization, and knowledge distillation. Finally, we provide a summary and discuss some possible research directions.

A Survey on Deep-Learning-Based Real-Time SAR Ship Detection

Recently, deep learning has greatly promoted the development of SAR ship detection. But the detectors are usually heavy and computation intensive which hinder the usage on the edge. In order to solve this problem, a lot of lightweight networks and acceleration ideas are proposed. In this survey, we review the papers that about real-time SAR ship detection. We firstly introduce the model compression and acceleration methods. They are pruning, quantization, knowledge distillation, low-rank factorization, lightweight networks and model deployment. They are the source of innovation in real-time SAR ship detection. Then we summarize the real-time object detection methods. They are two-stage, single-stage, anchor free, trained from scratch, model compression and acceleration. Researchers in SAR ship detection usually learn from these ideas. We then spend a lot of content on the review of the 70 real-time SAR ship detection papers. The years, datasets, journals, deep learning frameworks, and hardwares are introduced firstly. After that, the 10 public datasets and the evaluation metrics are shown. Then, we survey the 70 papers according to anchor free, trained from scratch, YOLO series, CFAR+CNN, lightweight backbone, pruning, quantization, knowledge distillation and hardware deployment. The experimental results show that the algorithms have been greatly developed in speed and accuracy. In the end we pointed out the problems of the 70 papers and the directions to be studied in the future. Our work can enable researchers to quickly understand the research status in this field.

Robust Hyperspectral Inpainting via Low-Rank Regularized Untrained Convolutional Neural Network

Over the past decade, many low-rank models, factorizations, or approximations have been applied to the restoration of hyperspectral images (HSIs) (e.g., denoising, inpainting, and super-resolution) from their incomplete and/or noisy measurements. Recently, deep learning (DL) has been shown to be a powerful method for solving inverse problems (including HSI restoration), but a large amount of training data is required. Since this is not possible for HSIs, unlike red green blue (RGB) images, in this work, a novel unsupervised framework for hyperspectral inpainting (HI) is proposed that can be implemented using an untrained convolutional neural network (CNN) for deep image prior (DIP), together with a recently reported differentiable regularization for the data rank and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell _{2}$ </tex-math></inline-formula> -norm squared loss function. Based on the proposed framework, we come up with a novel HI algorithm [denoted as deep low-rank hyperspectral inpainting (DLRHyIn)] and a robust DLRHyIn (denoted as R-DLRHyIn) which is robust against outliers, where the latter differs from the former only in the Huber loss function (HLF) (which has been justified robust to mixed noise) used instead. Then some simulation results and real-data experiments are provided to demonstrate the effectiveness of the proposed DLRHyIn and R-DLRHyIn. Finally, we draw some conclusions.

MOLASS: Software for automatic processing of matrix data obtained from small-angle X-ray scattering and UV-visible spectroscopy combined with size-exclusion chromatography.

Recent small-angle X-ray scattering (SAXS) for biological macromolecules (BioSAXS) is generally combined with size-exclusion chromatography (SEC-SAXS) at synchrotron facilities worldwide. For SEC-SAXS analysis, the final scattering profile for the target molecule is calculated from a large volume of continuously collected data. It would be ideal to automate this process; however, several complex problems exist regarding data measurement and analysis that have prevented automation. Here, we developed the analytical software MOLASS (Matrix Optimization with Low-rank factorization for Automated analysis of SEC-SAXS) to automatically calculate the final scattering profiles for solution structure analysis of target molecules. In this paper, the strategies for automatic analysis of SEC-SAXS data are described, including correction of baseline-drift using a low percentile method, optimization of peak decompositions composed of multiple scattering components using modified Gaussian fitting against the chromatogram, and rank determination for extrapolation to infinite dilution. In order to easily calculate each scattering component, the Moore-Penrose pseudo-inverse matrix is adopted as a basic calculation. Furthermore, this analysis method, in combination with UV-visible spectroscopy, led to better results in terms of accuracy in peak decomposition. Therefore, MOLASS will be able to smoothly suggest to users an accurate scattering profile for the subsequent structural analysis.

Efficient quantum computation of molecular forces and other energy gradients

While most work on the quantum simulation of chemistry has focused on computing energy surfaces, a similarly important application requiring subtly different algorithms is the computation of energy derivatives. Almost all molecular properties can be expressed an energy derivative, including molecular forces, which are essential for applications such as molecular dynamics simulations. Here, we introduce new quantum algorithms for computing molecular energy derivatives with significantly lower complexity than prior methods. Under cost models appropriate for noisy-intermediate scale quantum devices we demonstrate how low rank factorizations and other tomography schemes can be optimized for energy derivative calculations. We perform numerics revealing that our techniques reduce the number of circuit repetitions required by many orders of magnitude for even modest systems. In the context of fault-tolerant algorithms, we develop new methods of estimating energy derivatives with Heisenberg limited scaling incorporating state-of-the-art techniques for block encoding fermionic operators. Our results suggest that the calculation of forces on a single nucleus may be of similar cost to estimating energies of chemical systems, but that further developments are needed for quantum computers to meaningfully assist with molecular dynamics simulations.

Exponential Graph Regularized Non-Negative Low-Rank Factorization for Robust Latent Representation

Non-negative matrix factorization (NMF) is a fundamental theory that has received much attention and is widely used in image engineering, pattern recognition and other fields. However, the classical NMF has limitations such as only focusing on local information, sensitivity to noise and small sample size (SSS) problems. Therefore, how to develop the NMF to improve the performance and robustness of the algorithm is a worthy challenge. Based on the bottlenecks above, we propose an exponential graph regularization non-negative low-rank factorization algorithm (EGNLRF) combining sparseness, low rank and matrix exponential. Firstly, based on the assumption that the data is corroded, we decompose a given raw data item with a data error fitting noise matrix, applying a low-rank constraint to the denoising data. Then, we perform a non-negative factorization on the resulting low-rank matrix, from which we derive the low-dimensional representation of the original matrix. Finally, we use the low-dimensional representation for graph embedding to maintain the geometry between samples. The graph embedding terms are matrix exponentiated to cope with SSS problems and nearest neighbor sensitivity. The above three steps will be incorporated into a joint framework to validate and optimize each other; therefore, we can learn latent data representations that are undisturbed by noise and preserve the local structure of known samples. We conducted simulation experiments on different datasets and verified the effectiveness of the algorithm by comparing the proposed with the lasting ones related to NMF, low rank and graph embedding.

Bayesian robust tensor completion via CP decomposition

The real-world tensor data are inevitably missing and corrupted with noise. Some models of the low-rank tensor factorization (LRTF) add an L1 norm or L2 norm to deal with the sparse or Gaussian noise. However, the real noise are usually complex. We propose a robust Bayesian tensor completion method, called MoG BTC-CP, which could impute the missing data and remove the complex noise simultaneously. The observed tensor is assumed to be the summation of a low-rank tensor and the noise. CP decomposition is proposed to extract the low-rank structure of the tensor. We assume that the noise follows a Mixture of Gaussian (MoG) distribution. A full Bayesian framework together with a Gibbs sampling algorithm is designed to estimate the model. Extensive experiments including synthetic data and real life applications show that MoG BTC-CP outperforms the recently published leading tensor completion and denoising methods.

A low-rank ensemble Kalman filter for elliptic observations

We propose a regularization method for ensemble Kalman filtering (EnKF) with elliptic observation operators. Commonly used EnKF regularization methods suppress state correlations at long distances. For observations described by elliptic partial differential equations, such as the pressure Poisson equation (PPE) in incompressible fluid flows, distance localization should be used cautiously, as we cannot disentangle slowly decaying physical interactions from spurious long-range correlations. This is particularly true for the PPE, in which distant vortex elements couple nonlinearly to induce pressure. Instead, these inverse problems have a low effective dimension: low-dimensional projections of the observations strongly inform a low-dimensional subspace of the state space. We derive a low-rank factorization of the Kalman gain based on the spectrum of the Jacobian of the observation operator. The identified eigenvectors generalize the source and target modes of the multipole expansion, independently of the underlying spatial distribution of the problem. Given rapid spectral decay, inference can be performed in the low-dimensional subspace spanned by the dominant eigenvectors. This low-rank EnKF is assessed on dynamical systems with Poisson observation operators, where we seek to estimate the positions and strengths of point singularities over time from potential or pressure observations. We also comment on the broader applicability of this approach to elliptic inverse problems outside the context of filtering.

A fast direct solver for integral equations on locally refined boundary discretizations and its application to multiphase flow simulations

In transient simulations of particulate Stokes flow, to accurately capture the interaction between the constituent particles and the confining wall, the discretization of the wall often needs to be locally refined in the region approached by the particles. Consequently, standard fast direct solvers lose their efficiency since the linear system changes at each time step. This manuscript presents a new computational approach that avoids this issue by pre-constructing a fast direct solver for the wall ahead of time, computing a low-rank factorization to capture the changes due to the refinement, and solving the problem on the refined discretization via a Woodbury formula. Numerical results illustrate the efficiency of the solver in accelerating particulate Stokes simulations.

RSKNet-MTSP: Effective and portable deep architecture for speaker verification

The convolutional neural network (CNN) based approaches have shown great success for speaker verification (SV) tasks, where modeling long temporal context and reducing information loss of speaker characteristics are two important challenges significantly affecting the verification performance. Previous works have introduced dilated convolution and multi-scale aggregation methods to address above challenges. However, such methods are also hard to make full use of some valuable information, which make it difficult to substantially improve the verification performance. To address above issues, we construct a novel CNN-based architecture for SV, called RSKNet-MTSP, where a residual selective kernel block (RSKBlock) and a multiple time-scale statistics pooling (MTSP) module are proposed. The RSKNet-MTSP can capture both long temporal context and neighbouring information, and gather more speaker-discriminative information from multi-scale features. In order to design a portable model for real applications with limited resources, we then present a lightweight version of RSKNet-MTSP, namely RSKNet-MTSP-L, which employs a combination technique associating the depthwise separable convolutions with low-rank factorization of weight matrices. Extensive experiments are conducted on two public SV datasets, VoxCeleb and Speaker in the Wild (SITW). The results demonstrate that 1) RSKNet-MTSP significantly outperforms the state-of-the-art deep embedding architectures on all five test sets; 2) RSKNet-MTSP-L achieves competitive performance with 17%-44% less network parameters compared to baseline models. The ablation experiments further illustrate that our proposed approaches can achieve substantial improvement over prior methods.

Measurements as a roadblock to near-term practical quantum advantage in chemistry: Resource analysis

Recent advances in quantum computing devices have brought attention to hybrid quantum-classical algorithms like the Variational Quantum Eigensolver (VQE) as a potential route to practical quantum advantage in chemistry. However, it is not yet clear whether such algorithms, even in the absence of device error, could actually achieve quantum advantage for systems of practical interest. We have performed an exhaustive analysis to estimate the number of qubits and number of measurements required to compute the combustion energies of small organic molecules and related systems to within chemical accuracy of experimental values using VQE. We consider several key modern improvements to VQE, including low-rank factorizations of the Hamiltonian. Our results indicate that although these techniques are useful, they will not be sufficient to achieve practical quantum computational advantage for our molecular set, or for similar molecules. This suggests that novel approaches to operator estimation leveraging quantum coherence, such as Enhanced Likelihood Functions [arxiv:2006.09350, arxiv:2006.09349], may be required.

Mixed precision low-rank approximations and their application to block low-rank LU factorization

Abstract We introduce a novel approach to exploit mixed precision arithmetic for low-rank approximations. Our approach is based on the observation that singular vectors associated with small singular values can be stored in lower precisions while preserving high accuracy overall. We provide an explicit criterion to determine which level of precision is needed for each singular vector. We apply this approach to block low-rank (BLR) matrices, most of whose off-diagonal blocks have low rank. We propose a new BLR LU factorization algorithm that exploits the mixed precision representation of the blocks. We carry out the rounding error analysis of this algorithm and prove that the use of mixed precision arithmetic does not compromise the numerical stability of the BLR LU factorization. Moreover, our analysis determines which level of precision is needed for each floating-point operation (flop), and therefore guides us toward an implementation that is both robust and efficient. We evaluate the potential of this new algorithm on a range of matrices coming from real-life problems in industrial and academic applications. We show that a large fraction of the entries in the LU factors and flops to perform the BLR LU factorization can be safely switched to lower precisions, leading to significant reductions of the storage and expected time costs, of up to a factor three using fp64, fp32, and bfloat16 arithmetics.

Neural graph embeddings as explicit low-rank matrix factorization for link prediction

Learning good quality neural graph embeddings has long been achieved by minimzing the pointwise mutual information (PMI) for co-occuring nodes in simulated random walks. This design choice has been mostly popularized by the direct application of the highly-successful word embedding algorithm word2vec to predicting the formation of new links in social, co-citation, and biological networks. However, such a skeuomorphic design of graph embedding methods entails a truncation of information coming from pairs of nodes with low PMI. To circumvent this issue, we propose an improved approach to learning low-rank factorization embeddings that incorporate information from such unlikely pairs of nodes and show that it can improve the link prediction performance of baseline methods from 1.2% to 24.2%. Based on our results and observations, we outline further steps that could improve the design of next graph embedding algorithms that are based on matrix factorizaion.

The Influence of Gauge Conditions on Process and Result of Solving 3-D Problems of Computational Electromagnetism by the Finite Element Method

This article discusses the properties of the practical application results of the Coulomb and the “tree-cotree” gauge conditions for problems based on massive finite element 3-D-meshes (up to 12 million tetrahedrons) and with the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${A}$ </tex-math></inline-formula> – <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${V}$ </tex-math></inline-formula> formulation of the magnetic potential. The influence of the sort of gauging of the magnetic vector potential on the accuracy of the calculated solution of systems of linear algebraic equations (SLAEs) is shown by the example of magnetostatics and quasi-static (frequency domain) benchmark problems. The study was carried out for the cases of solving problems by iterative methods of the Krylov subspace and by direct methods. Iterative methods have shown the complexity of achieving the required solution accuracy with an increase in the number of finite elements and deterioration of the properties of the SLAE. The analysis of the dependence of the iterative process convergence and the required solution accuracy on the number of elements in the mesh and the type of gauging applied is accomplished. For direct methods for solving SLAEs, an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a posteriori</i> estimate of the dependence of the value of the relative forward error on the presence and type of the gauge condition is performed. The possibility and efficiency of using block low-rank (BLR) factorization for solving SLAEs in the presence and absence of gauge conditions are considered. The conditions for using the gauging of the magnetic vector potential are shown, which are suitable for simplifying the process of preparing and solving numerical problems and increasing the accuracy of the calculated values of the magnetic vector potential in the frequency domain.

Space-Fractional Diffusion with Variable Order and Diffusivity: Discretization and Direct Solution Strategies

We consider the multidimensional space-fractional diffusion equations with spatially varying diffusivity and fractional order. Significant computational challenges are encountered when solving these equations due to the kernel singularity in the fractional integral operator and the resulting dense discretized operators, which quickly become prohibitively expensive to handle because of their memory and arithmetic complexities. In this work, we present a singularity-aware discretization scheme that regularizes the singular integrals through a singularity subtraction technique adapted to the spatial variability of diffusivity and fractional order. This regularization strategy is conveniently formulated as a sparse matrix correction that is added to the dense operator, and is applicable to different formulations of fractional diffusion equations. We also present a block low rank representation to handle the dense matrix representations, by exploiting the ability to approximate blocks of the resulting formally dense matrix by low rank factorizations. A Cholesky factorization solver operates directly on this representation using the low rank blocks as its atomic computational tiles, and achieves high performance on multicore hardware. Numerical results show that the singularity treatment is robust, substantially reduces discretization errors, and attains the first-order convergence rate allowed by the regularity of the solutions. They also show that considerable savings are obtained in storage (\(O(N^{1.5})\)) and computational cost (\(O(N^2)\)) compared to dense factorizations. This translates to orders-of-magnitude savings in memory and time on multidimensional problems, and shows that the proposed methods offer practical tools for tackling large nonlocal fractional diffusion simulations.

Hyperspectral Image Denoising Using Factor Group Sparsity-Regularized Nonconvex Low-Rank Approximation

Hyperspectral image (HSI) mixed noise removal is a fundamental problem and an important preprocessing step in remote sensing fields. The low-rank approximation-based methods have been verified effective to encode the global spectral correlation for HSI denoising. However, due to the large scale and complexity of real HSI, previous low-rank HSI denoising techniques encounter several problems, including coarse rank approximation (such as nuclear norm), the high computational cost of singular value decomposition (SVD) (such as Schatten <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> -norm), and adaptive rank selection (such as low-rank factorization). In this article, two novel factor group sparsity-regularized nonconvex low-rank approximation (FGSLR) methods are introduced for HSI denoising, which can simultaneously overcome the mentioned issues of previous works. The FGSLR methods capture the spectral correlation via low-rank factorization, meanwhile utilizing factor group sparsity regularization to further enhance the low-rank property. It is SVD-free and robust to rank selection. Moreover, FGSLR is equivalent to Schatten <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> -norm approximation ( <xref ref-type="theorem" rid="theorem1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Theorem 1</xref> ), and thus FGSLR is tighter than the nuclear norm in terms of rank approximation. To preserve the spatial information of HSI in the denoising process, the total variation regularization is also incorporated into the proposed FGSLR models. Specifically, the proximal alternating minimization is designed to solve the proposed FGSLR models. Experimental results have demonstrated that the proposed FGSLR methods significantly outperform existing low-rank approximation-based HSI denoising methods.

Multichannel SAR Moving Target Detection via RPCA-Net

Ground moving target indication (GMTI), as a challenging task for synthetic aperture radar (SAR) systems, keeps drawing considerable attention. Robust principal component analysis (RPCA) aiming at separating low-rank and sparse components has been successfully employed in SAR systems for GMTI recently. However, its practical application is limited by the heavy computational burden as well as the requirement of manual parameter modification. To cope with this problem, a fast and free of presetting parameters RPCA network (RPCA-Net) is proposed for SAR-GMTI under strong clutter background. In the proposed method, a novel RPCA model is first introduced, where not only the low-rank and sparse terms but also the errors in practical SAR systems are taken into account. Moreover, the low-rank factorization plus scaled gradient descent (ScaledGD) is also employed to acquire low-rank clutter background rather than singular value decomposition (SVD). Then, we parameterize our proposed RPCA model and unfold it as a feedforward neural network (FNN) to acquire the iterative parameters through backpropagation. Compared to the GMTI methods based on traditional RPCA models, our proposed RPCA-Net can provide a higher detection ability and faster convergence without presetting parameters empirically. Experiments on two groups of measured data collected by airborne SAR systems validate the superior performance of the proposed RPCA-Net.