Low-rank Tensor Decomposition Research Articles

Dual-Grained Lightweight Strategy.

Removing redundant parameters and computations before the model training has attracted a great interest as it can effectively reduce the storage space of the model, speed up the training and inference of the model, and save energy consumption during the running of the model. In addition, the simplification of deep neural network models can enable high-performance network models to be deployed to resource-constrained edge devices, thus promoting the development of the intelligent world. However, current pruning at initialization methods exhibit poor performance at extreme sparsity. In order to improve the performance of the model under extreme sparsity, this paper proposes a dual-grained lightweight strategy-TEDEPR. This is the first time that TEDEPR has used tensor theory in the pruning at initialization method to optimize the structure of a sparse sub-network model and improve its performance. Specifically, first, at the coarse-grained level, we represent the weight matrix or weight tensor of the model as a low-rank tensor decomposition form and use multi-step chain operations to enhance the feature extraction capability of the base module to construct a low-rank compact network model. Second, unimportant weights are pruned at a fine-grained level based on the trainability of the weights in the low-rank model before the training of the model, resulting in the final compressed model. To evaluate the superiority of TEDEPR, we conducted extensive experiments on MNIST, UCF11, CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet datasets with LeNet, LSTM, VGGNet, ResNet and Transformer architectures, and compared with state-of-the-art methods. The experimental results show that TEDEPR has higher accuracy, faster training and inference, and less storage space than other pruning at initialization methods under extreme sparsity.

TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor Decomposition

Most state-of-the-art convolutional neural networks (CNNs) are characterised by excessive parameterisation, leading to a high computational burden. Tensor decomposition has emerged as a model reduction technique for compressing deep neural networks. Previous approaches have predominantly relied on either Tucker decomposition or Canonical Polyadic (CP) decomposition for CNNs. However, CP decomposition exhibits exceptional compression capabilities in comparison to Tucker decomposition, which results in a more pronounced accuracy loss. This paper introduces an efficient model compression method, termed TEC-CNN, designed to achieve significant compression while preserving accuracy levels comparable to those of the original models. In TEC-CNN, convolutional layers are identified to obtain convolutional kernels by analysing given models under the principles of low-rank tensor decomposition, and then, calculating the ranks of convolutional kernels. Furthermore, an efficient decomposition schema for the convolutional kernel is proposed with approximate kernel tensor for reducing parameters. Additionally, a novel format of a convolutional sequence is presented and constructed with a reduced number of parameters to replace the original convolutional layers. Finally, the effectiveness of TEC-CNN is assessed across a range of computer vision tasks. For instance, in CIFAR-100 classification, ResNet18 is compressed to 4.1 MB, while Unext, when applied to image segmentation using the International Skin Imaging Collaboration (ISIC) dataset, is reduced to 3.419 MB. When employed for fire object detection with Yolov7, TEC-CNN achieves a model size reduction of 71.6 MB. Comprehensive experimental results underscore that our approach achieves significant model compression while preserving model performance.

Enhanced MRI brain tumor detection and classification via topological data analysis and low-rank tensor decomposition

The advent of artificial intelligence in medical imaging has paved the way for significant advancements in the diagnosis of brain tumors. This study presents a novel ensemble approach that uses magnetic resonance imaging (MRI) to identify and categorize common brain cancers, such as pituitary, meningioma, and glioma. The proposed workflow is composed of a two-fold approach: firstly, it employs non-trivial image enhancement techniques in data preprocessing, low-rank Tucker decomposition for dimensionality reduction, and machine learning (ML) classifiers to detect and predict the type of brain tumor. Secondly, persistent homology (PH), a topological data analysis (TDA) technique, is exploited to extract potential critical areas in MRI scans. When paired with the ML classifier output, this additional information can help domain experts to identify areas of interest that might contain tumor signatures, improving the interpretability of ML predictions. When compared to automated diagnoses, this transparency adds another level of confidence and is essential for clinical acceptance. The performance of the system was quantitatively evaluated on a well-known MRI dataset, with an overall classification accuracy of 97.28% using an extremely randomized trees model. The promising results show that the integration of TDA, ML, and low-rank approximation methods is a successful approach for brain tumor identification and categorization, providing a solid foundation for further study and clinical application.

Bound-state stability of Coulomb three-body systems using numerical tensor methods

In this paper we present a unified treatment of three-body atoms and molecules using numerical tensor methods. The Schrödinger equation in perimetric coordinates is recast in a canonical tensor format. It is shown that the Schrödinger equation can be solved in this full-tensor format but that by using a low-rank tensor decomposition, in particular the tensor-train and quantized-tensor-train formats, energies accurate to at least the nanohartree can be obtained for the He atom (in which the mass of the uniquely charged particle is much greater than the other two particles), the positronium negative ion (Ps−, in which all the masses are equal), and the non-Born-Oppenheimer H2+ molecule (in which the mass of the uniquely charged particle is much smaller than the other two particles). Published by the American Physical Society 2024

Multidimensional Data Processing With Bayesian Inference via Structural Block Decomposition.

How to handle large multidimensional datasets, such as hyperspectral images and video information, efficiently and effectively plays a critical role in big-data processing. The characteristics of low-rank tensor decomposition in recent years demonstrate the essentials in describing the tensor rank, which often leads to promising approaches. However, most current tensor decomposition models consider the rank-1 component simply to be the vector outer product, which may not fully capture the correlated spatial information effectively for large-scale and high-order multidimensional datasets. In this article, we develop a new novel tensor decomposition model by extending it to the matrix outer product or called Bhattacharya-Mesner product, to form an effective dataset decomposition. The fundamental idea is to decompose tensors structurally in a compact manner as much as possible while retaining data spatial characteristics in a tractable way. By incorporating the framework of the Bayesian inference, a new tensor decomposition model on the subtle matrix unfolding outer product is established for both tensor completion and robust principal component analysis problems, including hyperspectral image completion and denoising, traffic data imputation, and video background subtraction. Numerical experiments on real-world datasets demonstrate the highly desirable effectiveness of the proposed approach.

Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention

Deep convolutional neural networks have a large number of parameters and require a significant number of floating-point operations during computation, which limits their deployment in situations where the storage space is limited and computational resources are insufficient, such as in mobile phones and small robots. Many network compression methods have been proposed to address the aforementioned issues, including pruning, low-rank decomposition, quantization, etc. However, these methods typically fail to achieve a significant compression ratio in terms of the parameter count. Even when high compression rates are achieved, the network’s performance is often significantly deteriorated, making it difficult to perform tasks effectively. In this study, we propose a more compact representation for neural networks, named Quantized Low-Rank Tensor Decomposition (QLTD), to super compress deep convolutional neural networks. Firstly, we employed low-rank Tucker decomposition to compress the pre-trained weights. Subsequently, to further exploit redundancies within the core tensor and factor matrices obtained through Tucker decomposition, we employed vector quantization to partition and cluster the weights. Simultaneously, we introduced a self-attention module for each core tensor and factor matrix to enhance the training responsiveness in critical regions. The object identification results in the CIFAR10 experiment showed that QLTD achieved a compression ratio of 35.43×, with less than 1% loss in accuracy and a compression ratio of 90.61×, with less than a 2% loss in accuracy. QLTD was able to achieve a significant compression ratio in terms of the parameter count and realize a good balance between compressing parameters and maintaining identification accuracy.

Randomized two-sided subspace iteration for low-rank matrix and tensor decomposition

The low-rank approximation of big data matrices and tensors plays a pivotal role in many modern applications. Although, a truncated version of the singular value decomposition (SVD) furnishes the best approximation, its computation is challenging on modern, multicore architectures. Recently, the randomized subspace iteration has shown to be a powerful tool in approximating large-scale matrices. In this paper we present a two-sided variant of the randomized subspace iteration. Novel in our work is the exploitation of the unpivoted QR factorization, rather than the SVD, for factorizing the compressed matrix. Hence our algorithm is a randomized rank-revealing URV decomposition. We prove the rank-revealingness of our algorithm by establishing bounds for the singular values as well as the other blocks of the compressed matrix. We further provide bounds on the error of the low-rank approximations of the proposed algorithm, in both 2- and Frobenius norm. In addition, we employ the proposed algorithm to efficiently compute low rank tensor decompositions: we present two randomized algorithms, one for the truncated higher-order SVD, and the other for the tensor SVD. We conduct numerical tests on (i) various classes of matrices, and (ii) synthetic tensors and real datasets to demonstrate the efficacy of the proposed algorithms.

Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity

Modern convolutional neural networks (CNNs) play a crucial role in computer vision applications. The intricacy of the application scenarios and the growing dataset both significantly raise the complexity of CNNs. As a result, they are often overparameterized and have significant computational costs. One potential solution for optimizing and compressing the CNNs is to replace convolutional layers with low-rank tensor decomposition. The most suitable technique for this is Canonical Polyadic (CP) decomposition. However, there are two primary issues with CP decomposition that lead to a significant loss in accuracy. Firstly, the selection of tensor ranks for CP decomposition is an unsolved issue. Secondly, degeneracy and instability are common problems in the CP decomposition of contractional tensors, which makes fine-tuning the compressed model difficult. In this study, a novel approach was proposed for compressing CNNs by using CP decomposition. The first step involves using the sensitivity of convolutional layers to determine the tensor ranks for CP decomposition effectively. Subsequently, to address the degeneracy issue and enhance the stability of the CP decomposition, two novel techniques were incorporated: optimization with sensitivity constraints and iterative fine-tuning based on sensitivity order. Finally, the proposed method was examined on common CNN structures for image classification tasks and demonstrated that it provides stable performance and significantly fewer reductions in classification accuracy.

Semi-parametric tensor factor analysis by iteratively projected singular value decomposition

Abstract This paper introduces a general framework of Semi-parametric TEnsor Factor Analysis (STEFA) that focuses on the methodology and theory of low-rank tensor decomposition with auxiliary covariates. Semi-parametric TEnsor Factor Analysis models extend tensor factor models by incorporating auxiliary covariates in the loading matrices. We propose an algorithm of iteratively projected singular value decomposition (IP-SVD) for the semi-parametric estimation. It iteratively projects tensor data onto the linear space spanned by the basis functions of covariates and applies singular value decomposition on matricized tensors over each mode. We establish the convergence rates of the loading matrices and the core tensor factor. The theoretical results only require a sub-exponential noise distribution, which is weaker than the assumption of sub-Gaussian tail of noise in the literature. Compared with the Tucker decomposition, IP-SVD yields more accurate estimators with a faster convergence rate. Besides estimation, we propose several prediction methods with new covariates based on the STEFA model. On both synthetic and real tensor data, we demonstrate the efficacy of the STEFA model and the IP-SVD algorithm on both the estimation and prediction tasks.

Removal of Mixed Noise in Hyperspectral Images Based on Subspace Representation and Nonlocal Low-Rank Tensor Decomposition.

Hyperspectral images (HSIs) contain abundant spectral and spatial structural information, but they are inevitably contaminated by a variety of noises during data reception and transmission, leading to image quality degradation and subsequent application hindrance. Hence, removing mixed noise from hyperspectral images is an important step in improving the performance of subsequent image processing. It is a well-established fact that the data information of hyperspectral images can be effectively represented by a global spectral low-rank subspace due to the high redundancy and correlation (RAC) in the spatial and spectral domains. Taking advantage of this property, a new algorithm based on subspace representation and nonlocal low-rank tensor decomposition is proposed to filter the mixed noise of hyperspectral images. The algorithm first obtains the subspace representation of the hyperspectral image by utilizing the spectral low-rank property and obtains the orthogonal basis and representation coefficient image (RCI). Then, the representation coefficient image is grouped and denoised using tensor decomposition and wavelet decomposition, respectively, according to the spatial nonlocal self-similarity. Afterward, the orthogonal basis and denoised representation coefficient image are optimized using the alternating direction method of multipliers (ADMM). Finally, iterative regularization is used to update the image to obtain the final denoised hyperspectral image. Experiments on both simulated and real datasets demonstrate that the algorithm proposed in this paper is superior to related mainstream methods in both quantitative metrics and intuitive vision. Because it is denoising for image subspace, the time complexity is greatly reduced and is lower than related denoising algorithms in terms of computational cost.

Co-Exploring Structured Sparsification and Low-Rank Tensor Decomposition for Compact DNNs

Co-Exploring Structured Sparsification and Low-Rank Tensor Decomposition for Compact DNNs

Smooth Tensor Qatar Riyal Decomposition for Dynamic MRI Reconstruction.

Dynamic magnetic resonance imaging (dMRI) speed and imaging quality have always been a crucial issue in medical imaging research. Most existing methods characterize the tensor rank-based minimization to reconstruct dMRI from sampling k- t space data. However, (1) these approaches that unfold the tensor along each dimension destroy the inherent structure of dMR images. (2) they focus on preserving global information only, while ignoring the local details reconstruction such as the spatial piece-wise smoothness and sharp boundaries. To overcome these obstacles, we suggest a novel low-rank tensor decomposition approach by integrating tensor Qatar Riyal (QR) decomposition, low-rank tensor nuclear norm, and asymmetric total variation to reconstruct dMRI, named TQRTV. Specifically, while preserving the tensor inherent structure by utilizing tensor nuclear norm minimization to approximate tensor rank, QR decomposition reduces the dimensions in the low-rank constraint term, thereby improving the reconstruction performance. TQRTV further exploits the asymmetric total variation regularizer to capture local details. Numerical experiments demonstrate that the proposed reconstruction approach is superior to the existing ones.

Adaptive Regularized Low-Rank Tensor Decomposition for Hyperspectral Image Denoising and Destriping

Adaptive Regularized Low-Rank Tensor Decomposition for Hyperspectral Image Denoising and Destriping

Kiosk 11R-TB-11 - Non-rigid Motion Corrected Reconstruction with Nonlocally Low-rank Tensor Decomposition Based Regularization for High-resolution Cartesian First-pass Myocardial Perfusion Imaging at 3 Tesla

Kiosk 11R-TB-11 - Non-rigid Motion Corrected Reconstruction with Nonlocally Low-rank Tensor Decomposition Based Regularization for High-resolution Cartesian First-pass Myocardial Perfusion Imaging at 3 Tesla

Simulation-aided infrared thermography with decomposition-based noise reduction for detecting defects in ancient polyptychs

In recent years, the conservation and protection of ancient cultural heritage have received increasing attention, and non-destructive testing (NDT), which can minimize the damage done to the test subject, plays an integral role therein. For instance, NDT through active infrared thermal imaging can be applied to ancient polyptychs, which can realize accurate detection of damage and defects existing on the surface and interior of the polyptychs. In this study, infrared thermography is used for non-invasive investigation and evaluation of two polyptych samples with different pigments and artificial defects, but both reproduced based on a painting by Pietro Lorenzetti (1280/85–1348) using the typical tempera technique of the century. It is noted that, to avoid as far as possible secondary damages done to the ancient cultural heritages, repeated damage-detection experiments are rarely carried out on the test subjects. To that end, numerical simulation is used to reveal the heat transfer properties and temperature distributions, as to perform procedural verification and reduce the number of experiments that need to be conducted on actual samples. Technique-wise, to improve the observability of the experimental results, a total variation regularized low-rank tensor decomposition algorithm is implemented to reduce the background noise and improve the contrast of the images. Furthermore, the efficacy of image processing is quantified through the structural-similarity evaluation.

HMC: Hybrid model compression method based on layer sensitivity grouping.

Previous studies have shown that deep models are often over-parameterized, and this parameter redundancy makes deep compression possible. The redundancy of model weight is often manifested as low rank and sparsity. Ignoring any part of the two or the different distributions of these two characteristics in the model will lead to low accuracy and a low compression rate of deep compression. To make full use of the difference between low-rank and sparsity, a unified framework combining low-rank tensor decomposition and structured pruning is proposed: a hybrid model compression method based on sensitivity grouping (HMC). This framework unifies the existing additive hybrid compression method (AHC) and the non-additive hybrid compression method (NaHC) proposed by us into one model. The latter group the network according to the sensitivity difference of the convolutional layer to different compression methods, which can better integrate the low rank and sparsity of the model compared with the former. Experiments show that our approach achieves a better trade-off between test accuracy and compression ratio when compressing the ResNet family of models than other recent compression methods using a single strategy or additive hybrid compression.

Post-Kohn-Sham Random-Phase Approximation and Correction Terms in the Expectation-Value Coupled-Cluster Formulation.

Using expectation-value coupled-cluster theory and many-body perturbation theory (MBPT), we formulate a series of corrections to the post-Kohn-Sham (post-KS) random-phase approximation (RPA) energy. The beyond-RPA terms are of two types: those accounting for the non-Hartree-Fock reference and those introducing the coupled-cluster doubles non-ring contractions. The contributions of the former type, introduced via the semicanonical orbital basis, drastically reduce the binding strength in noncovalent systems. The good accuracy is recovered by the attractive third-order doubles correction referred to as Ec2g. The existing RPA approaches based on KS orbitals neglect most of the proposed corrections but can perform well thanks to error cancellation. The proposed method accounts for every contribution in the state-of-the-art renormalized second-order perturbation theory (rPT2) approach but adds additional terms which initially contribute in the third order of MBPT. The cost of energy evaluation scales as noniterative in the implementation with low-rank tensor decomposition. The numerical tests of the proposed approach demonstrate accurate results for noncovalent dimers of polar molecules and for the challenging many-body noncovalent cluster of CH4···(H2O)20.

Performance Implication of Tensor Irregularity and Optimization for Distributed Tensor Decomposition

Tensors are used by a wide variety of applications to represent multi-dimensional data; tensor decompositions are a class of methods for latent data analytics, data compression, and so on. Many of these applications generate large tensors with irregular dimension sizes and nonzero distribution. CANDECOMP/PARAFAC decomposition ( Cpd ) is a popular low-rank tensor decomposition for discovering latent features. The increasing overhead on memory and execution time of Cpd for large tensors requires distributed memory implementations as the only feasible solution. The sparsity and irregularity of tensors hinder the improvement of performance and scalability of distributed memory implementations. While previous works have been proved successful in Cpd for tensors with relatively regular dimension sizes and nonzero distribution, they either deliver unsatisfactory performance and scalability for irregular tensors or require significant time overhead in preprocessing. In this work, we focus on medium-grained tensor distribution to address their limitation for irregular tensors. We first thoroughly investigate through theoretical and experimental analysis. We disclose that the main cause of poor Cpd performance and scalability is the imbalance of multiple types of computations and communications and their tradeoffs; and sparsity and irregularity make it challenging to achieve their balances and tradeoffs. Irregularity of a sparse tensor is categorized based on two aspects: very different dimension sizes and a non-uniform nonzero distribution. Typically, focusing on optimizing one type of load imbalance causes other ones more severe for irregular tensors. To address such challenges, we propose irregularity-aware distributed Cpd that leverages the sparsity and irregularity information to identify the best tradeoff between different imbalances with low time overhead. We materialize the idea with two optimization methods: the prediction-based grid configuration and matrix-oriented distribution policy, where the former forms the global balance among computations and communications, and the latter further adjusts the balances among computations. The experimental results show that our proposed irregularity-aware distributed Cpd is more scalable and outperforms the medium- and fine-grained distributed implementations by up to 4.4 × and 11.4 × on 1,536 processors, respectively. Our optimizations support different sparse tensor formats, such as compressed sparse fiber (CSF), coordinate (COO), and Hierarchical Coordinate (HiCOO), and gain good scalability for all of them.

Hybrid Low-Rank Tensor CP and Tucker Decomposition with Total Variation Regularization for HSI Noise Removal

The acquired hyperspectral images (HSIs) are affected by a mixture of several types of noise, which often suffer from information missing. Corrupted HSIs limit the precision of the subsequent processing. In this paper, the weighted Total Variation-regularized Hybrid Model of CP and Tucker (WTV-HMCT) is proposed to accurately identify the intrinsic structures of the clean HSIs. By jointly minimizing CP rank and Tucker rank in the low-rank tensor approximation, WTV-HMCT fully exploits the high-dimensional structure correlations of HSI. To ensure the piecewise smoothness of the recovered image, the hybrid low-rank tensor decomposition approach integrates the weighted spatial spectral total variation regularization for the separation of the noise-free HSI and mixed noise. By the Alternating Direction Method of Multipliers (ADMM), the optimization model is transformed into two subproblems. Finally, an efficient proximal alternating minimization algorithm is developed to optimize the proposed hybrid low-rank tensor decomposition efficiently. The experimental results show that the proposed model effectively handles Gauss noise, striping noise, and mixed noise and that it outperforms the most advanced methods in terms of evaluation metrics and visual evaluation.

Multi-view Ensemble Clustering via Low-rank and Sparse Decomposition: From Matrix to Tensor

As a significant extension of classical clustering methods, ensemble clustering first generates multiple basic clusterings and then fuses them into one consensus partition by solving a problem concerning graph partition with respect to the co-association matrix. Although the collaborative cluster structure among basic clusterings can be well discovered by ensemble clustering, most advanced ensemble clustering utilizes the self-representation strategy with the constraint of low-rank to explore a shared consensus representation matrix in multiple views. However, they still encounter two challenges: (1) high computational cost caused by both the matrix inversion operation and singular value decomposition of large-scale square matrices; (2) less considerable attention on high-order correlation attributed to the pursue of the two-dimensional pair-wise relationship matrix. In this article, based on low-rank and sparse decomposition from both matrix and tensor perspectives, we propose two novel multi-view ensemble clustering methods, which tangibly decrease computational complexity. Specifically, our first method utilizes low-rank and sparse matrix decomposition to learn one common co-association matrix, while our last method constructs all co-association matrices into one third-order tensor to investigate the high-order correlation among multiple views by low-rank and sparse tensor decomposition. We adopt the alternating direction method of multipliers to solve two convex models by dividing them into several subproblems with closed-form solution. Experimental results on ten real-world datasets prove the effectiveness and efficiency of the proposed two multi-view ensemble clustering methods by comparing them with other advanced ensemble clustering methods.