Low-rank Tensor Decomposition Research Articles

Multidimensional Data Processing With Bayesian Inference via Structural Block Decomposition.

How to handle large multidimensional datasets, such as hyperspectral images and video information, efficiently and effectively plays a critical role in big-data processing. The characteristics of low-rank tensor decomposition in recent years demonstrate the essentials in describing the tensor rank, which often leads to promising approaches. However, most current tensor decomposition models consider the rank-1 component simply to be the vector outer product, which may not fully capture the correlated spatial information effectively for large-scale and high-order multidimensional datasets. In this article, we develop a new novel tensor decomposition model by extending it to the matrix outer product or called Bhattacharya-Mesner product, to form an effective dataset decomposition. The fundamental idea is to decompose tensors structurally in a compact manner as much as possible while retaining data spatial characteristics in a tractable way. By incorporating the framework of the Bayesian inference, a new tensor decomposition model on the subtle matrix unfolding outer product is established for both tensor completion and robust principal component analysis problems, including hyperspectral image completion and denoising, traffic data imputation, and video background subtraction. Numerical experiments on real-world datasets demonstrate the highly desirable effectiveness of the proposed approach.

Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention

Deep convolutional neural networks have a large number of parameters and require a significant number of floating-point operations during computation, which limits their deployment in situations where the storage space is limited and computational resources are insufficient, such as in mobile phones and small robots. Many network compression methods have been proposed to address the aforementioned issues, including pruning, low-rank decomposition, quantization, etc. However, these methods typically fail to achieve a significant compression ratio in terms of the parameter count. Even when high compression rates are achieved, the network’s performance is often significantly deteriorated, making it difficult to perform tasks effectively. In this study, we propose a more compact representation for neural networks, named Quantized Low-Rank Tensor Decomposition (QLTD), to super compress deep convolutional neural networks. Firstly, we employed low-rank Tucker decomposition to compress the pre-trained weights. Subsequently, to further exploit redundancies within the core tensor and factor matrices obtained through Tucker decomposition, we employed vector quantization to partition and cluster the weights. Simultaneously, we introduced a self-attention module for each core tensor and factor matrix to enhance the training responsiveness in critical regions. The object identification results in the CIFAR10 experiment showed that QLTD achieved a compression ratio of 35.43×, with less than 1% loss in accuracy and a compression ratio of 90.61×, with less than a 2% loss in accuracy. QLTD was able to achieve a significant compression ratio in terms of the parameter count and realize a good balance between compressing parameters and maintaining identification accuracy.

Randomized two-sided subspace iteration for low-rank matrix and tensor decomposition

The low-rank approximation of big data matrices and tensors plays a pivotal role in many modern applications. Although, a truncated version of the singular value decomposition (SVD) furnishes the best approximation, its computation is challenging on modern, multicore architectures. Recently, the randomized subspace iteration has shown to be a powerful tool in approximating large-scale matrices. In this paper we present a two-sided variant of the randomized subspace iteration. Novel in our work is the exploitation of the unpivoted QR factorization, rather than the SVD, for factorizing the compressed matrix. Hence our algorithm is a randomized rank-revealing URV decomposition. We prove the rank-revealingness of our algorithm by establishing bounds for the singular values as well as the other blocks of the compressed matrix. We further provide bounds on the error of the low-rank approximations of the proposed algorithm, in both 2- and Frobenius norm. In addition, we employ the proposed algorithm to efficiently compute low rank tensor decompositions: we present two randomized algorithms, one for the truncated higher-order SVD, and the other for the tensor SVD. We conduct numerical tests on (i) various classes of matrices, and (ii) synthetic tensors and real datasets to demonstrate the efficacy of the proposed algorithms.

Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity

Modern convolutional neural networks (CNNs) play a crucial role in computer vision applications. The intricacy of the application scenarios and the growing dataset both significantly raise the complexity of CNNs. As a result, they are often overparameterized and have significant computational costs. One potential solution for optimizing and compressing the CNNs is to replace convolutional layers with low-rank tensor decomposition. The most suitable technique for this is Canonical Polyadic (CP) decomposition. However, there are two primary issues with CP decomposition that lead to a significant loss in accuracy. Firstly, the selection of tensor ranks for CP decomposition is an unsolved issue. Secondly, degeneracy and instability are common problems in the CP decomposition of contractional tensors, which makes fine-tuning the compressed model difficult. In this study, a novel approach was proposed for compressing CNNs by using CP decomposition. The first step involves using the sensitivity of convolutional layers to determine the tensor ranks for CP decomposition effectively. Subsequently, to address the degeneracy issue and enhance the stability of the CP decomposition, two novel techniques were incorporated: optimization with sensitivity constraints and iterative fine-tuning based on sensitivity order. Finally, the proposed method was examined on common CNN structures for image classification tasks and demonstrated that it provides stable performance and significantly fewer reductions in classification accuracy.

Semi-parametric tensor factor analysis by iteratively projected singular value decomposition

Abstract This paper introduces a general framework of Semi-parametric TEnsor Factor Analysis (STEFA) that focuses on the methodology and theory of low-rank tensor decomposition with auxiliary covariates. Semi-parametric TEnsor Factor Analysis models extend tensor factor models by incorporating auxiliary covariates in the loading matrices. We propose an algorithm of iteratively projected singular value decomposition (IP-SVD) for the semi-parametric estimation. It iteratively projects tensor data onto the linear space spanned by the basis functions of covariates and applies singular value decomposition on matricized tensors over each mode. We establish the convergence rates of the loading matrices and the core tensor factor. The theoretical results only require a sub-exponential noise distribution, which is weaker than the assumption of sub-Gaussian tail of noise in the literature. Compared with the Tucker decomposition, IP-SVD yields more accurate estimators with a faster convergence rate. Besides estimation, we propose several prediction methods with new covariates based on the STEFA model. On both synthetic and real tensor data, we demonstrate the efficacy of the STEFA model and the IP-SVD algorithm on both the estimation and prediction tasks.

Removal of Mixed Noise in Hyperspectral Images Based on Subspace Representation and Nonlocal Low-Rank Tensor Decomposition.

Hyperspectral images (HSIs) contain abundant spectral and spatial structural information, but they are inevitably contaminated by a variety of noises during data reception and transmission, leading to image quality degradation and subsequent application hindrance. Hence, removing mixed noise from hyperspectral images is an important step in improving the performance of subsequent image processing. It is a well-established fact that the data information of hyperspectral images can be effectively represented by a global spectral low-rank subspace due to the high redundancy and correlation (RAC) in the spatial and spectral domains. Taking advantage of this property, a new algorithm based on subspace representation and nonlocal low-rank tensor decomposition is proposed to filter the mixed noise of hyperspectral images. The algorithm first obtains the subspace representation of the hyperspectral image by utilizing the spectral low-rank property and obtains the orthogonal basis and representation coefficient image (RCI). Then, the representation coefficient image is grouped and denoised using tensor decomposition and wavelet decomposition, respectively, according to the spatial nonlocal self-similarity. Afterward, the orthogonal basis and denoised representation coefficient image are optimized using the alternating direction method of multipliers (ADMM). Finally, iterative regularization is used to update the image to obtain the final denoised hyperspectral image. Experiments on both simulated and real datasets demonstrate that the algorithm proposed in this paper is superior to related mainstream methods in both quantitative metrics and intuitive vision. Because it is denoising for image subspace, the time complexity is greatly reduced and is lower than related denoising algorithms in terms of computational cost.

Smooth Tensor Qatar Riyal Decomposition for Dynamic MRI Reconstruction.

Dynamic magnetic resonance imaging (dMRI) speed and imaging quality have always been a crucial issue in medical imaging research. Most existing methods characterize the tensor rank-based minimization to reconstruct dMRI from sampling k- t space data. However, (1) these approaches that unfold the tensor along each dimension destroy the inherent structure of dMR images. (2) they focus on preserving global information only, while ignoring the local details reconstruction such as the spatial piece-wise smoothness and sharp boundaries. To overcome these obstacles, we suggest a novel low-rank tensor decomposition approach by integrating tensor Qatar Riyal (QR) decomposition, low-rank tensor nuclear norm, and asymmetric total variation to reconstruct dMRI, named TQRTV. Specifically, while preserving the tensor inherent structure by utilizing tensor nuclear norm minimization to approximate tensor rank, QR decomposition reduces the dimensions in the low-rank constraint term, thereby improving the reconstruction performance. TQRTV further exploits the asymmetric total variation regularizer to capture local details. Numerical experiments demonstrate that the proposed reconstruction approach is superior to the existing ones.

Adaptive Regularized Low-Rank Tensor Decomposition for Hyperspectral Image Denoising and Destriping

Adaptive Regularized Low-Rank Tensor Decomposition for Hyperspectral Image Denoising and Destriping

Kiosk 11R-TB-11 - Non-rigid Motion Corrected Reconstruction with Nonlocally Low-rank Tensor Decomposition Based Regularization for High-resolution Cartesian First-pass Myocardial Perfusion Imaging at 3 Tesla

Kiosk 11R-TB-11 - Non-rigid Motion Corrected Reconstruction with Nonlocally Low-rank Tensor Decomposition Based Regularization for High-resolution Cartesian First-pass Myocardial Perfusion Imaging at 3 Tesla

Simulation-aided infrared thermography with decomposition-based noise reduction for detecting defects in ancient polyptychs

In recent years, the conservation and protection of ancient cultural heritage have received increasing attention, and non-destructive testing (NDT), which can minimize the damage done to the test subject, plays an integral role therein. For instance, NDT through active infrared thermal imaging can be applied to ancient polyptychs, which can realize accurate detection of damage and defects existing on the surface and interior of the polyptychs. In this study, infrared thermography is used for non-invasive investigation and evaluation of two polyptych samples with different pigments and artificial defects, but both reproduced based on a painting by Pietro Lorenzetti (1280/85–1348) using the typical tempera technique of the century. It is noted that, to avoid as far as possible secondary damages done to the ancient cultural heritages, repeated damage-detection experiments are rarely carried out on the test subjects. To that end, numerical simulation is used to reveal the heat transfer properties and temperature distributions, as to perform procedural verification and reduce the number of experiments that need to be conducted on actual samples. Technique-wise, to improve the observability of the experimental results, a total variation regularized low-rank tensor decomposition algorithm is implemented to reduce the background noise and improve the contrast of the images. Furthermore, the efficacy of image processing is quantified through the structural-similarity evaluation.

HMC: Hybrid model compression method based on layer sensitivity grouping.

Previous studies have shown that deep models are often over-parameterized, and this parameter redundancy makes deep compression possible. The redundancy of model weight is often manifested as low rank and sparsity. Ignoring any part of the two or the different distributions of these two characteristics in the model will lead to low accuracy and a low compression rate of deep compression. To make full use of the difference between low-rank and sparsity, a unified framework combining low-rank tensor decomposition and structured pruning is proposed: a hybrid model compression method based on sensitivity grouping (HMC). This framework unifies the existing additive hybrid compression method (AHC) and the non-additive hybrid compression method (NaHC) proposed by us into one model. The latter group the network according to the sensitivity difference of the convolutional layer to different compression methods, which can better integrate the low rank and sparsity of the model compared with the former. Experiments show that our approach achieves a better trade-off between test accuracy and compression ratio when compressing the ResNet family of models than other recent compression methods using a single strategy or additive hybrid compression.

Post-Kohn-Sham Random-Phase Approximation and Correction Terms in the Expectation-Value Coupled-Cluster Formulation.

Using expectation-value coupled-cluster theory and many-body perturbation theory (MBPT), we formulate a series of corrections to the post-Kohn-Sham (post-KS) random-phase approximation (RPA) energy. The beyond-RPA terms are of two types: those accounting for the non-Hartree-Fock reference and those introducing the coupled-cluster doubles non-ring contractions. The contributions of the former type, introduced via the semicanonical orbital basis, drastically reduce the binding strength in noncovalent systems. The good accuracy is recovered by the attractive third-order doubles correction referred to as Ec2g. The existing RPA approaches based on KS orbitals neglect most of the proposed corrections but can perform well thanks to error cancellation. The proposed method accounts for every contribution in the state-of-the-art renormalized second-order perturbation theory (rPT2) approach but adds additional terms which initially contribute in the third order of MBPT. The cost of energy evaluation scales as noniterative in the implementation with low-rank tensor decomposition. The numerical tests of the proposed approach demonstrate accurate results for noncovalent dimers of polar molecules and for the challenging many-body noncovalent cluster of CH4···(H2O)20.

Performance Implication of Tensor Irregularity and Optimization for Distributed Tensor Decomposition

Tensors are used by a wide variety of applications to represent multi-dimensional data; tensor decompositions are a class of methods for latent data analytics, data compression, and so on. Many of these applications generate large tensors with irregular dimension sizes and nonzero distribution. CANDECOMP/PARAFAC decomposition ( Cpd ) is a popular low-rank tensor decomposition for discovering latent features. The increasing overhead on memory and execution time of Cpd for large tensors requires distributed memory implementations as the only feasible solution. The sparsity and irregularity of tensors hinder the improvement of performance and scalability of distributed memory implementations. While previous works have been proved successful in Cpd for tensors with relatively regular dimension sizes and nonzero distribution, they either deliver unsatisfactory performance and scalability for irregular tensors or require significant time overhead in preprocessing. In this work, we focus on medium-grained tensor distribution to address their limitation for irregular tensors. We first thoroughly investigate through theoretical and experimental analysis. We disclose that the main cause of poor Cpd performance and scalability is the imbalance of multiple types of computations and communications and their tradeoffs; and sparsity and irregularity make it challenging to achieve their balances and tradeoffs. Irregularity of a sparse tensor is categorized based on two aspects: very different dimension sizes and a non-uniform nonzero distribution. Typically, focusing on optimizing one type of load imbalance causes other ones more severe for irregular tensors. To address such challenges, we propose irregularity-aware distributed Cpd that leverages the sparsity and irregularity information to identify the best tradeoff between different imbalances with low time overhead. We materialize the idea with two optimization methods: the prediction-based grid configuration and matrix-oriented distribution policy, where the former forms the global balance among computations and communications, and the latter further adjusts the balances among computations. The experimental results show that our proposed irregularity-aware distributed Cpd is more scalable and outperforms the medium- and fine-grained distributed implementations by up to 4.4 × and 11.4 × on 1,536 processors, respectively. Our optimizations support different sparse tensor formats, such as compressed sparse fiber (CSF), coordinate (COO), and Hierarchical Coordinate (HiCOO), and gain good scalability for all of them.

Hybrid Low-Rank Tensor CP and Tucker Decomposition with Total Variation Regularization for HSI Noise Removal

The acquired hyperspectral images (HSIs) are affected by a mixture of several types of noise, which often suffer from information missing. Corrupted HSIs limit the precision of the subsequent processing. In this paper, the weighted Total Variation-regularized Hybrid Model of CP and Tucker (WTV-HMCT) is proposed to accurately identify the intrinsic structures of the clean HSIs. By jointly minimizing CP rank and Tucker rank in the low-rank tensor approximation, WTV-HMCT fully exploits the high-dimensional structure correlations of HSI. To ensure the piecewise smoothness of the recovered image, the hybrid low-rank tensor decomposition approach integrates the weighted spatial spectral total variation regularization for the separation of the noise-free HSI and mixed noise. By the Alternating Direction Method of Multipliers (ADMM), the optimization model is transformed into two subproblems. Finally, an efficient proximal alternating minimization algorithm is developed to optimize the proposed hybrid low-rank tensor decomposition efficiently. The experimental results show that the proposed model effectively handles Gauss noise, striping noise, and mixed noise and that it outperforms the most advanced methods in terms of evaluation metrics and visual evaluation.

Multi-view Ensemble Clustering via Low-rank and Sparse Decomposition: From Matrix to Tensor

As a significant extension of classical clustering methods, ensemble clustering first generates multiple basic clusterings and then fuses them into one consensus partition by solving a problem concerning graph partition with respect to the co-association matrix. Although the collaborative cluster structure among basic clusterings can be well discovered by ensemble clustering, most advanced ensemble clustering utilizes the self-representation strategy with the constraint of low-rank to explore a shared consensus representation matrix in multiple views. However, they still encounter two challenges: (1) high computational cost caused by both the matrix inversion operation and singular value decomposition of large-scale square matrices; (2) less considerable attention on high-order correlation attributed to the pursue of the two-dimensional pair-wise relationship matrix. In this article, based on low-rank and sparse decomposition from both matrix and tensor perspectives, we propose two novel multi-view ensemble clustering methods, which tangibly decrease computational complexity. Specifically, our first method utilizes low-rank and sparse matrix decomposition to learn one common co-association matrix, while our last method constructs all co-association matrices into one third-order tensor to investigate the high-order correlation among multiple views by low-rank and sparse tensor decomposition. We adopt the alternating direction method of multipliers to solve two convex models by dividing them into several subproblems with closed-form solution. Experimental results on ten real-world datasets prove the effectiveness and efficiency of the proposed two multi-view ensemble clustering methods by comparing them with other advanced ensemble clustering methods.

Tensor Train-Based Higher-Order Dynamic Mode Decomposition for Dynamical Systems

Higher-order dynamic mode decomposition (HODMD) has proved to be an efficient tool for the analysis and prediction of complex dynamical systems described by data-driven models. In the present paper, we propose a realization of HODMD that is based on the low-rank tensor decomposition of potentially high-dimensional datasets. It is used to compute the HODMD modes and eigenvalues to effectively reduce the computational complexity of the problem. The proposed extension also provides a more efficient realization of the ordinary dynamic mode decomposition with the use of the tensor-train decomposition. The high efficiency of the tensor-train-based HODMD (TT-HODMD) is illustrated by a few examples, including forecasting the load of a power system, which provides comparisons between TT-HODMD and HODMD with respect to the computing time and accuracy. The developed algorithm can be effectively used for the prediction of high-dimensional dynamical systems.

Multi-view spectral clustering based on adaptive neighbor learning and low-rank tensor decomposition

Multi-view spectral clustering based on adaptive neighbor learning and low-rank tensor decomposition

Quantization and application of low-rank tensor decomposition based on the deep learning model.

Watching the presentation of a large-scale network is very important for network state tracking, performance optimization, traffic engineering, anomaly detection, fault analysis, etc. In this paper, we try to develop deep learning technology to solve the defect problem of tensor filling based on inner product interaction. To solve the limitations of the existing tensor-filling algorithms, a new neural tensor-filling (NTC) model is proposed. NTC model can effectively type the third-order communication between data landscapes through outer creation operation. It creates the third-order interaction mapping tensor. On this basis, the interaction between local features of the 3D neural network is studied. In this paper, another fusion neural tensor filling (Fu NTC) model is proposed to solve the problem that the NTC model can only extract the nonlinear complex structural information between potential feature dimensions. In the framework of the neural network, the NTC model and tensor decomposition model share the same potential feature embedding. It can effectively extract nonlinear feature information and linear feature information at the same time. It achieves higher precision data recovery.

Low Rank Tensor Decompositions and Approximations

AbstractThere exist linear relations among tensor entries of low rank tensors. These linear relations can be expressed by multi-linear polynomials, which are called generating polynomials. We use generating polynomials to compute tensor rank decompositions and low rank tensor approximations. We prove that this gives a quasi-optimal low rank tensor approximation if the given tensor is sufficiently close to a low rank one.

ANLPT: Self-Adaptive and Non-Local Patch-Tensor Model for Infrared Small Target Detection

Infrared small target detection is widely used for early warning, aircraft monitoring, ship monitoring, and so on, which requires the small target and its background to be represented and modeled effectively to achieve their complete separation. Low-rank sparse decomposition based on the structural features of infrared images has attracted much attention among many algorithms because of its good interpretability. Based on our study, we found some shortcomings in existing baseline methods, such as redundancy of constructing tensors and fixed compromising factors. A self-adaptive low-rank sparse tensor decomposition model for infrared dim small target detection is proposed in this paper. In this model, the entropy of image block is used for fast matching of non-local similar blocks to construct a better sparse tensor for small targets. An adaptive strategy of low-rank sparse tensor decomposition is proposed for different background environments, which adaptively determines the weight coefficient to achieve effective separation of background and small targets in different background environments. Tensor robust principal component analysis (TRPCA) was applied to achieve low-rank sparse tensor decomposition to reconstruct small targets and their backgrounds separately. Sufficient experiments on the various types data sets show that the proposed method is competitive.