Low-rank Matrix Approximation Research Articles

An integrated approach combining randomized kernel PCA, Gaussian mixture modeling and ICA for fault detection in non-linear processes

Principal component analysis (PCA) and independent component analysis (ICA), as well as their kernel extensions, have been widely applied in the past for industrial fault detection with Gaussian or non-Gaussian process data with linear or non-linear characteristics. Kernel-based techniques lead to computational complexity due to the high dimensionality of the dataset in the feature space. In this work, a randomization approach is used to obtain a low-rank approximation of the high-dimensional kernel matrix. A hybrid machine learning technique is proposed that integrates randomized kernel PCA (RKPCA) with ICA and Gaussian mixture modeling (GMM). The proposed approach, ICA-RKPCA-GMM, addresses the Gaussian and non-Gaussian characteristics of non-linear process data. Another hybrid algorithm combining three basic techniques of ICA, PCA and GMM is also developed (ICA-PCA-GMM). The fault detection performances of the proposed techniques (ICA-RKPCA-GMM and ICA-PCA-GMM) are compared with PCA, ICA, KPCA and combined ICA-PCA techniques by applying the techniques to two benchmark systems. Monitoring performances were evaluated by determining the false alarm rate and fault detection rate for different types of process and sensor faults. The simulation results show that the proposed ICA-RKPCA-GMM approach yields better results than individual ICA, PCA and KPCA techniques, the combined ICA-PCA and the proposed ICA-PCA-GMM technique.

TDOA-Based Indoor Localization via Linear Fusion With Low-Rank Matrix Approximation

TDOA-Based Indoor Localization via Linear Fusion With Low-Rank Matrix Approximation

Fixed-precision randomized quaternion singular value decomposition algorithm for low-rank quaternion matrix approximations

The fixed-precision randomized quaternion singular value decomposition algorithm (FPRQSVD) is presented to compute the low-rank quaternion matrix approximation. The FPRQSVD algorithm estimates the appropriate rank according to the given tolerance without a predetermined rank parameter, and computes the corresponding low-rank quaternion matrix approximation automatically. Error analysis indicates that the FPRQSVD algorithm is mathematically equivalent to the randomized quaternion singular value decomposition algorithm. Numerical experiments are given to demonstrate that the FPRQSVD algorithm is feasible and effective. Moreover, we use it to deal with the problem of color image inpainting.

A Similarity Matrix Low-Rank Approximation and Inconsistency Separation Fusion Approach for Multiview Clustering

In recent years, multi-view clustering algorithms have achieved promising performance by exploiting the com-plementarity and consistency of different views. However, many multi-view spectral clustering methods only focus on the con-sistent information of views, and the time cost of feature decomposition is expensive. Moreover, these methods also require post-processing (e.g., k-means) to obtain the final clustering results. To overcome these limitations simultaneously, we propose a novel multi-view clustering algorithm. Firstly, the method removes the inconsistent information of the views through cross-view measurement to maintain consistent information. These inconsistencies may be caused by noise, corruptions, or view-specific properties and will affect the quality of the similarity matrix. Then, we learn a consensus embedding matrix with non-negative constraints by performing a low-rank decomposition of the consistency information. In this way, we can replace the eigendecomposition of the n × n Laplacian matrix in spectral clustering with the singular value decomposition of a n × c low-rank matrix to reduce the computational burden, where c ≪ n. Furthermore, due to the non-negative constraint, we can directly obtain the clustering results. Also, to consider the diversity of views, adaptive weighting is applied to different view data. Compared to state-of-the-art multi-view clustering methods on five benchmark multi-view datasets, we demonstrate the superiority and effectiveness of our approach. We release the source code at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hulu88/FAMvC</uri> .

Activation-Based Pruning of Neural Networks

We present a novel technique for pruning called activation-based pruning to effectively prune fully connected feedforward neural networks for multi-object classification. Our technique is based on the number of times each neuron is activated during model training. We compare the performance of activation-based pruning with a popular pruning method: magnitude-based pruning. Further analysis demonstrated that activation-based pruning can be considered a dimensionality reduction technique, as it leads to a sparse low-rank matrix approximation for each hidden layer of the neural network. We also demonstrate that the rank-reduced neural network generated using activation-based pruning has better accuracy than a rank-reduced network using principal component analysis. We provide empirical results to show that, after each successive pruning, the amount of reduction in the magnitude of singular values of each matrix representing the hidden layers of the network is equivalent to introducing the sum of singular values of the hidden layers as a regularization parameter to the objective function.

Improved Hybrid Preprocessing Technique for Effective Segmentation of Wheat Canopies in Chlorophyll Fluorescence Images

Precision agriculture heavily relies on accurately segmenting wheat canopies from chlorophyll fluorescence (CHF) images. However, these images often face challenges due to inherent noise and illumination variations, primarily induced by the thermal activity of photons emitting a fluorescence effect. The unique nature of fluorescence introduces variations in illumination, especially during the crop's dark adaptation before experimentation. This adaptation aims to capture the full fluorescence effect, starting from minimum fluorescence and progressing to maximum fluorescence. In the initial stages of fluorescence, images tend to appear darker compared to those progressing towards maximum fluorescence. This variability necessitates the development of a sophisticated hybrid approach to eliminate noise and enhance contrast collaboratively, maximizing the benefits derived from CHF images. This paper introduces a novel hybrid preprocessing approach designed to address these challenges. The proposed method integrates five denoising techniques, namely Discrete Cosine Transform, Block Matching-3D, Low-Rank Matrix Approximation, Wiener Filtering, and Median Filtering, to mitigate the impact of noise in CHF images. Simultaneously, two enhancement techniques, Adaptive Histogram Specification and Gamma Correction, are employed to accentuate critical features, compensating for inherent variations in illumination during the fluorescence process. The hybrid preprocessing technique was proposed after analysing different combinations of denoising and enhancement techniques. Through qualitative and quantitative analysis of the results, it was observed that Block Matching-3D denoising with Gamma Correction produced the best output, with an Average PSNR of 0.54 and Average MSE of 0.07. This cascaded approach not only emphasizes noise reduction but also prioritizes the enhancement of crucial information within CHF images. By synergistically combining denoising and enhancement methods, the proposed approach optimizes the overall quality of the images, laying a foundation for improved wheat canopy segmentation. This research contributes a comprehensive and innovative solution to the challenges associated with CHF images in precision agriculture. The proposed hybrid approach holds promise for advancing the accuracy and reliability of wheat canopy segmentation, thereby enhancing the efficacy of precision agricultural practices.

Hyperspectral Image Denoising: From Model-Driven, Data-Driven, to Model-Data-Driven.

Mixed noise pollution in HSI severely disturbs subsequent interpretations and applications. In this technical review, we first give the noise analysis in different noisy HSIs and conclude crucial points for programming HSI denoising algorithms. Then, a general HSI restoration model is formulated for optimization. Later, we comprehensively review existing HSI denoising methods, from model-driven strategy (nonlocal mean, total variation, sparse representation, low-rank matrix approximation, and low-rank tensor factorization), data-driven strategy 2-D convolutional neural network (CNN), 3-D CNN, hybrid, and unsupervised networks, to model-data-driven strategy. The advantages and disadvantages of each strategy for HSI denoising are summarized and contrasted. Behind this, we present an evaluation of the HSI denoising methods for various noisy HSIs in simulated and real experiments. The classification results of denoised HSIs and execution efficiency are depicted through these HSI denoising methods. Finally, prospects of future HSI denoising methods are listed in this technical review to guide the ongoing road for HSI denoising. The HSI denoising dataset could be found at https://qzhang95.github.io.

Quaternion Chromaticity Contrast Preserving Decolorization Method Based on Adaptive Singular Value Weighting

Color image decolorization can not only simplify the complexity of image processing and analysis, improving computational efficiency, but also help to preserve the key information of the image, enhance visual effects, and meet various practical application requirements. However, with existing decolorization methods it is difficult to simultaneously maintain the local detail features and global smooth features of the image. To address this shortcoming, this paper utilizes singular value decomposition to obtain the hierarchical local features of the image and utilizes quaternion theory to overcome the limitation of existing color image processing methods that ignore the correlation between the three channels of the color image. Based on this, we propose a singular value adaptive weighted fusion quaternion chromaticity contrast preserving decolorization method. This method utilizes the low-rank matrix approximation principle to design a singular value adaptive weighted fusion strategy for the three channels of the color image and implements image decolorization based on singular value adaptive weighting. To address the deficiency of the decolorization result obtained in this step, which cannot maintain global smoothness characteristics well, a contrast preserving decolorization algorithm based on quaternion chromaticity distance is further proposed, and the global weighting strategy obtained by this algorithm is integrated into the image decolorization based on singular value adaptive weighting. The experimental results show that the decolorization method proposed in this paper achieves excellent results in both subjective visual perception and objective evaluation metrics.

Evolving masked low-rank transformer for long text understanding

Long sequence text processing is time-consuming owing to the ultra-large-scale self-attention computing. Recent advances demonstrate the attention in transformer can be accelerated by redundancy removal, and there are various sparse variants for attention in large sequences are proposed, which leads to state-of-the-art performance on language and vision task. Low-rank method achieve outstanding success in the field of efficient transformer. The dynamic token sparsification is efficiently time-saving and cost-saving, which can be easily extended to prune redundant spans and to yield semantic features. Evolutionary algorithm is attractive for selecting hyperparameter which is of significant importance in effectiveness. Motivated by these works, we propose an efficient transformers model, termed EMLT, to alleviate time and cost without sacrificing the accuracy. EMLT effectively combines strengths of Low-rank transformers, dynamic token sparsification and evolutionary algorithm to ulteriorly cut redundant token and meanwhile maintains the original precision, which can achieve a linear memory and time complexity. We compress transformer in three stages. Firstly, sliding window is validated as local attention to capture fine-grained dependency semantics. After that, low-rank approximation of attention matrix is applied as global attention to store long-range dependency semantics, and aggregated with local attention. On this basis, we consistently prune redundant token in accordance with importance score to further sparse the attention operation. Finally, Evolutionary algorithm is utilized to optimize the hyper-parameters of every layer. The results of comprehensive experiments and analysis show that our method can rival others on accuracy, and outperforms others on efficiency by a large margin in terms of the computational complexity.

A systematic solution to 3D anchorless direction estimation using TDOA measurements

The paper examines the anchorless direction estimation problem using signal measurements of time difference of arrival (TDOA) from multiple far-field sources in 3D space. A systematic solution involving coordinate frame construction is proposed to jointly estimate source directions and sensor positions through low-rank matrix approximation. The number of unknown parameters to be estimated is reduced by QR decomposition of the mixing matrix to determine the relevant parameters. Pseudolinear and iterative estimators are developed for the solution through norm constraints. We study the property of TDOA-based anchorless direction estimation, discuss the necessary number of sources and sensors, and derive the constrained Cramér–Rao bound for performance evaluation. Additionally, an exception treatment strategy is proposed in the complex case for pseudolinear estimator. A simulation conducted to validate the proposed solution's effectiveness showed that the solution's estimation accuracy was high, indicating its feasibility and effectiveness.

Data-driven inference of bioprocess models: A low-rank matrix approximation approach

Following the recent advent of Process Analytical Technologies, dataset production has undergone significant leverage. In this new abundance of data, isolating meaningful, informative content is critical for process dynamic modeling. This paper proposes a data-driven algorithm based on low-rank matrix approximation, the so-called successive projection algorithm, to retrieve a minimal set of macroscopic reactions, the corresponding stoichiometry, and a consistent kinetic model structure from the measurements of the trajectories of the species concentrations during cultures in a bioreactor. The proposed method is successfully validated in simulation, considering a case study related to monoclonal antibody (MAb) production with hybridoma cell cultures.

Parameterized Complexity of Feature Selection for Categorical Data Clustering

We develop new algorithmic methods with provable guarantees for feature selection in regard to categorical data clustering. While feature selection is one of the most common approaches to reduce dimensionality in practice, most of the known feature selection methods are heuristics. We study the following mathematical model. We assume that there are some inadvertent (or undesirable) features of the input data that unnecessarily increase the cost of clustering. Consequently, we want to select a subset of the original features from the data such that there is a small-cost clustering on the selected features. More precisely, for given integers ℓ (the number of irrelevant features) and k (the number of clusters), budget B , and a set of n categorical data points (represented by m -dimensional vectors whose elements belong to a finite set of values Σ), we want to select m -ℓ relevant features such that the cost of any optimal k -clustering on these features does not exceed B . Here the cost of a cluster is the sum of Hamming distances (ℓ 0 -distances) between the selected features of the elements of the cluster and its center. The clustering cost is the total sum of the costs of the clusters. We use the framework of parameterized complexity to identify how the complexity of the problem depends on parameters k , B , and |Σ|. Our main result is an algorithm that solves the Feature Selection problem in time f ( k, B , |Σ|)⋅ m g ( k , |Σ|) ⋅ n 2 for some functions f and g . In other words, the problem is fixed-parameter tractable parameterized by B when |Σ| and k are constants. Our algorithm for Feature Selection is based on a solution to a more general problem, Constrained Clustering with Outliers. In this problem, we want to delete a certain number of outliers such that the remaining points could be clustered around centers satisfying specific constraints. One interesting fact about Constrained Clustering with Outliers is that besides Feature Selection, it encompasses many other fundamental problems regarding categorical data such as Robust Clustering and Binary and Boolean Low-rank Matrix Approximation with Outliers. Thus, as a byproduct of our theorem, we obtain algorithms for all these problems. We also complement our algorithmic findings with complexity lower bounds.

Tangent Space Based Alternating Projections for Nonnegative Low Rank Matrix Approximation

In this paper, we develop a new alternating projection method to compute nonnegative low rank matrix approximation for nonnegative matrices. In the nonnegative low rank matrix approximation method, the projection onto the manifold of fixed rank matrices can be expensive as the singular value decomposition is required. We propose to use the tangent space of the point in the manifold to approximate the projection onto the manifold in order to reduce the computational cost. We show that the sequence generated by the alternating projections onto the tangent spaces of the fixed rank matrices manifold and the nonnegative matrix manifold, converge linearly to a point in the intersection of the two manifolds where the convergent point is sufficiently close to optimal solutions. This convergence result based inexact projection onto the manifold is new and is not studied in the literature. Numerical examples in data clustering, pattern recognition and hyperspectral data analysis are given to demonstrate that the performance of the proposed method is better than that of nonnegative matrix factorization methods in terms of computational time and accuracy.

Randomized block Krylov subspace algorithms for low-rank quaternion matrix approximations

Randomized block Krylov subspace algorithms for low-rank quaternion matrix approximations

Pass-efficient randomized LU algorithms for computing low-rank matrix approximation

Abstract Low-rank matrix approximation is extremely useful in the analysis of data that arises in scientific computing, engineering applications, and data science. However, as data sizes grow, traditional low-rank matrix approximation methods, such as singular value decomposition (SVD) and column pivoting QR decomposition (CPQR), are either prohibitively expensive or cannot provide sufficiently accurate results. A solution is to use randomized low-rank matrix approximation methods such as randomized SVD, and randomized LU decomposition on extremely large data sets. In this paper, we focus on the randomized LU decomposition method. Then we propose a novel randomized LU algorithm, called SubspaceLU, for the fixed low-rank approximation problem. SubspaceLU is based on the sketch of the co-range of input matrices and allows for an arbitrary number of passes of the input matrix, v ≥ 2 {v\geq 2} . Numerical experiments on CPU show that our proposed SubspaceLU is generally faster than the existing randomized LU decomposition, while remaining accurate. Experiments on GPU shows that our proposed SubspaceLU can gain more speedup than the existing randomized LU decomposition. We also propose a version of SubspaceLU, called SubspaceLU_FP, for the fixed precision low-rank matrix approximation problem. SubspaceLU_FP is a post-processing step based on an efficient blocked adaptive rank determination Algorithm 5 proposed in this paper. We present numerical experiments that show that SubspaceLU_FP can achieve close results to SVD but faster in speed. We finally propose a single-pass algorithm based on LU factorization. Tests show that the accuracy of our single-pass algorithm is comparable with the existing single-pass algorithms.

Using low-rank approximations of gridded data for spline surface fitting

The paper is devoted to the problem of finding bivariate tensor-product spline (or similar) functions that approximate gridded data in the least-squares sense. We propose to apply a low rank approximation of matrices to the data, to find the result by solving a sequence of univariate fitting problems that can be handled efficiently. This can be seen as a generalization of the method proposed by Georgieva and Hofreither (2017) that combines cross approximation (which is a particular method for low rank matrix approximation) with spline interpolation.While the algorithm yields the best least-squares approximation after r steps, where r denotes the rank of the data matrix, terminating it earlier yields a low-rank approximation, which often provides a sufficient level of accuracy. We also present a stopping criterion (based on a lower error estimate) that allows to use the method efficiently in the situation when the required number of degrees of freedom is not known in advance.

Sparse low-rank approximation of matrix and local preservation for unsupervised image feature selection

Sparse low-rank approximation of matrix and local preservation for unsupervised image feature selection

Effective elastic properties of three-dimensional multiple crack problems with the isogeometric boundary element parallel fast direct solver

We propose an isogeometric boundary element (IGABEM) parallel fast direct solver based on OpenMP to solve three-dimensional multiple crack problems. In this paper, the orderly and randomly distributed cracks are studied to observe the influence of the multiple crack interaction on the effective elastic properties. This paper designs the parallel implementation of the matrix low rank approximation and the matrix assembly for the hierarchical off-diagonal low rank (HODLR) matrix. In the matrix low rank approximation, we propose the m-n tree to improve the accelerated hybrid algorithm to balance the CPU performance and memory usage. The numerical results show that the effective elastic properties can respond to the strong interaction to a certain extent, but the maximum stress intensity factors (SIFs) of multiple cracks still need to be calculated to diagnose the possible crack initiation points. On the other hand, the numerical results also show that the IGABEM parallel fast direct solver can greatly improve the computational efficiency on the premise of ensuring the computational accuracy.

Hybrid Threshold Denoising Framework Using Singular Value Decomposition for Side-Channel Analysis Preprocessing.

The traces used in side-channel analysis are essential to breaking the key of encryption and the signal quality greatly affects the correct rate of key guessing. Therefore, the preprocessing of side-channel traces plays an important role in side-channel analysis. The process of side-channel leakage signal acquisition is usually affected by internal circuit noise, external environmental noise, and other factors, so the collected signal is often mixed with strong noise. In order to extract the feature information of side-channel signals from very low signal-to-noise ratio traces, a hybrid threshold denoising framework using singular value decomposition is proposed for side-channel analysis preprocessing. This framework is based on singular value decomposition and introduces low-rank matrix approximation theory to improve the rank selection methods of singular value decomposition. This paper combines the hard threshold method of truncated singular value decomposition with the soft threshold method of singular value shrinkage damping and proposes a hybrid threshold denoising framework using singular value decomposition for the data preprocessing step of side-channel analysis as a general preprocessing method for non-profiled side-channel analysis. The data used in the experimental evaluation are from the raw traces of the public database of DPA contest V2 and AES_HD. The success rate curve of non-profiled side-channel analysis further confirms the effectiveness of the proposed framework. Moreover, the signal-to-noise ratio of traces is significantly improved after preprocessing, and the correlation with the correct key is also significantly enhanced. Experimental results on DPA v2 and AES_HD show that the proposed noise reduction framework can be effectively applied to the side-channel analysis preprocessing step, and can successfully improve the signal-to-noise ratio of the traces and the attack efficiency.

Hierarchical off-diagonal low-rank approximation of Hessians in inverse problems, with application to ice sheet model initialization

Obtaining lightweight and accurate approximations of discretized objective functional Hessians in inverse problems governed by partial differential equations (PDEs) is essential to make both deterministic and Bayesian statistical large-scale inverse problems computationally tractable. The cubic computational complexity of dense linear algebraic tasks, such as Cholesky factorization, that provide a means to sample Gaussian distributions and determine solutions of Newton linear systems is a computational bottleneck at large-scale. These tasks can be reduced to log-linear complexity by utilizing hierarchical off-diagonal low-rank (HODLR) matrix approximations. In this work, we show that a class of Hessians that arise from inverse problems governed by PDEs are well approximated by the HODLR matrix format. In particular, we study inverse problems governed by PDEs that model the instantaneous viscous flow of ice sheets. In these problems, we seek a spatially distributed basal sliding parameter field such that the flow predicted by the ice sheet model is consistent with ice sheet surface velocity observations. We demonstrate the use of HODLR Hessian approximation to efficiently sample the Laplace approximation of the posterior distribution with covariance further approximated by HODLR matrix compression. Computational studies are performed which illustrate ice sheet problem regimes for which the Gauss–Newton data-misfit Hessian is more efficiently approximated by the HODLR matrix format than the low-rank (LR) format. We then demonstrate that HODLR approximations can be favorable, when compared to global LR approximations, for large-scale problems by studying the data-misfit Hessian associated with inverse problems governed by the first-order Stokes flow model on the Humboldt glacier and Greenland ice sheet.