Tensor Train Decomposition Research Articles

Compact lossy compression of tensors via neural tensor-train decomposition

AbstractMany real-world datasets are represented as tensors, i.e., multi-dimensional arrays of numerical values. Storing them without compression often requires substantial space, which grows exponentially with the order. While many tensor compression algorithms are available, many of them rely on strong data assumptions regarding its order, sparsity, rank, and smoothness. In this work, we propose TensorCodec, a lossy compression algorithm for general tensors that do not necessarily adhere to strong input data assumptions.TensorCodec incorporates three key ideas. The first idea is neural tensor-train decomposition (NTTD) where we integrate a recurrent neural network into Tensor-Train Decomposition to enhance its expressive power and alleviate the limitations imposed by the low-rank assumption. Another idea is to fold the input tensor into a higher-order tensor to reduce the space required by NTTD. Finally, the mode indices of the input tensor are reordered to reveal patterns that can be exploited by NTTD for improved approximation. In addition, we extend TensorCodec to enable the lossy compression of tensors with missing entries, often found in real-world datasets. Our analysis and experiments on 8 real-world datasets demonstrate that TensorCodec is (a) Concise: it gives up to $$7.38 \times $$ 7.38 × more compact compression than the best competitor with similar reconstruction error, (b) Accurate: given the same budget for compressed size, it yields up to $$3.33\times $$ 3.33 × more accurate reconstruction than the best competitor, (c) Scalable: Its empirical compression time is linear in the number of tensor entries, and it reconstructs each entry in logarithmic time. Our code and datasets are available at https://github.com/kbrother/TensorCodec.

Bayesian poisson regression tensor train decomposition model for learning mortality pattern changes during COVID-19 pandemic

COVID-19 has led to excess deaths around the world. However, the impact on mortality rates from other causes of death during this time remains unclear. To understand the broader impact of COVID-19 on other causes of death, we analyze Italian official data covering monthly mortality counts from January 2015 to December 2020. To handle the high-dimensional nature of the data, we developed a model that combines Poisson regression with tensor train decomposition to explore the lower-dimensional residual structure of the data. Our Bayesian approach incorporates prior information on model parameters and utilizes an efficient Metropolis-Hastings within Gibbs algorithm for posterior inference. Simulation studies were conducted to validate our approach. Our method not only identifies differential effects of interventions on cause-specific mortality rates through Poisson regression but also provides insights into the relationship between COVID-19 and other causes of death. Additionally, it uncovers latent classes related to demographic characteristics, temporal patterns, and causes of death.

IDAD: An improved tensor train based distributed DDoS attack detection framework and its application in complex networks

With the vigorous development of Internet technology, the scale of systems in the network has increased sharply, which provides a great opportunity for potential attacks, especially the Distributed Denial of Service (DDoS) attack. In this case, detecting DDoS attacks is critical to system security. However, current detection methods exhibit limitations, leading to compromises in accuracy and efficiency. To cope with it, three key strategies are implemented in this paper: (i) Using tensors to model large-scale and heterogeneous data in complex networks; (ii) Proposing a denoising algorithm based on the improved and distributed tensor train (IDTT) decomposition, which optimizes the tensor train(TT) decomposition in terms of parallel computation and low-rank estimation; (iii) Combining (i), (ii) and Light Gradient Boosting Machine (LightGBM) classification model, an efficient DDoS attack detection framework is proposed. Datasets CIC-DDoS2019 and NSL-KDD are used to evaluate the framework, and results demonstrate that accuracy can reach 99.19% while having the characteristics of low storage consumption and well speedup ratio.

Robust tensor recovery via a nonconvex approach with ket augmentation and auto‐weighted strategy

AbstractIn this article, we introduce a nonconvex tensor recovery approach, which employs the powerful ket augmentation technique to expand a low order tensor into a high‐order one so that we can exploit the advantage of tensor train (TT) decomposition tailored for high‐order tensors. Moreover, we define a new nonconvex surrogate function to approximate the tensor rank, and develop an auto‐weighted mechanism to adjust the weights of the resulting high‐order tensor's TT ranks. To make our approach robust, we add two mode‐unfolding regularization terms to enhance the model for the purpose of exploring spatio‐temporal continuity and self‐similarity of the underlying tensors. Also, we propose an implementable algorithm to solve the proposed optimization model in the sense that each subproblem enjoys a closed‐form solution. A series of numerical results demonstrate that our approach works well on recovering color images and videos.

A weighted prior tensor train decomposition method for community detection in multi-layer networks

Community detection in multi-layer networks stands as a prominent subject within network analysis research. However, the majority of existing techniques for identifying communities encounter two primary constraints: they lack suitability for high-dimensional data within multi-layer networks and fail to fully leverage additional auxiliary information among communities to enhance detection accuracy. To address these limitations, a novel approach named weighted prior tensor training decomposition (WPTTD) is proposed for multi-layer network community detection. Specifically, the WPTTD method harnesses the tensor feature optimization techniques to effectively manage high-dimensional data in multi-layer networks. Additionally, it employs a weighted flattened network to construct prior information for each dimension of the multi-layer network, thereby continuously exploring inter-community connections. To preserve the cohesive structure of communities and to harness comprehensive information within the multi-layer network for more effective community detection, the common community manifold learning (CCML) is integrated into the WPTTD framework for enhancing the performance. Experimental evaluations conducted on both artificial and real-world networks have verified that this algorithm outperforms several mainstream multi-layer network community detection algorithms.

TENSOR QUANTILE REGRESSION WITH LOW-RANK TENSOR TRAIN ESTIMATION.

Neuroimaging studies often involve predicting a scalar outcome from an array of images collectively called tensor. The use of magnetic resonance imaging (MRI) provides a unique opportunity to investigate the structures of the brain. To learn the association between MRI images and human intelligence, we formulate a scalar-on-image quantile regression framework. However, the high dimensionality of the tensor makes estimating the coefficients for all elements computationally challenging. To address this, we propose a low-rank coefficient array estimation algorithm based on tensor train (TT) decomposition which we demonstrate can effectively reduce the dimensionality of the coefficient tensor to a feasible level while ensuring adequacy to the data. Our method is more stable and efficient compared to the commonly used, Canonic Polyadic rank approximation-based method. We also propose a generalized Lasso penalty on the coefficient tensor to take advantage of the spatial structure of the tensor, further reduce the dimensionality of the coefficient tensor, and improve the interpretability of the model. The consistency and asymptotic normality of the TT estimator are established under some mild conditions on the covariates and random errors in quantile regression models. The rate of convergence is obtained with regularization under the total variation penalty. Extensive numerical studies, including both synthetic and real MRI imaging data, are conducted to examine the empirical performance of the proposed method and its competitors.

Randomized density matrix renormalization group algorithm for low-rank tensor train decomposition

Randomized density matrix renormalization group algorithm for low-rank tensor train decomposition

Prediction of wind energy with the use of tensor‐train based higher order dynamic mode decomposition

AbstractAs the international energy market pays more and more attention to the development of clean energy, wind power is gradually attracting the attention of various countries. Wind power is a sustainable and environmentally friendly resource of energy. However, it is unstable. Therefore, it is important to develop algorithms for its prediction. In this paper, we apply a recently developed algorithm that effectively combines the tensor train decomposition with the higher order dynamic mode decomposition (TT‐HODMD). The dynamic mode decomposition (DMD) is a data‐driven technique that does not need a prior mathematical model. It is based on the measurement data or time slots. As demonstrated, for prediction it is important to use the higher order DMD (HODMD). In turn, HODMD might lead to very large scale arrays that are sparse. The tensor train decomposition provides a highly efficient way to work with such arrays. It is demonstrated that the combined TT‐HODMD algorithm is capable of providing quite accurate prediction of wind power for months ahead.

Lightweight tensorial convolutional neural network for lunar impact crater detection

Impact crater detection is one of the most important and critical studies of the geological history of planetary surfaces. Recent literature on crater identification demonstrates the powerful potential of deep-learning algorithms. However, these methods potentially assumed that the detection is conducted in devices with sufficient computing resources, and used networks with massive learnable parameters to obtain superb performance. In this paper, we proposed a lightweight tensorial convolutional neural network (CNN) for lunar impact crater detection. Specifically, we develop a tensorial you only look once (TYOLO) network by representing the convolutional kernels in tensor train (TT) decomposition. Thus, the parameter redundancy of TYOLO can be greatly reduced. To the best of our knowledge, this is the first work that considers the lightweight CNN model for impact crater detection. We demonstrate the lunar impact crater detection results over the ChangE-2 lunar digital orthophotos model (DOM) and digital evaluated model (DEM). Experimental results show that the proposed TYOLO network can obtain 1.44×speed up and 39× compression with 2.4% improvement in crater detection. Overall, the proposed network demonstrates great potential in spacecraft with emerging computing resource constraints.

A Practical Approach for Employing Tensor Train Decomposition in Edge Devices

Deep Neural Networks (DNN) have made significant advances in various fields including speech recognition and image processing. Typically, modern DNNs are both compute and memory intensive, therefore their deployment in low-end devices is a challenging task. A well-known technique to address this problem is Low-Rank Factorization (LRF), where a weight tensor is approximated by one or more lower-rank tensors, reducing both the memory size and the number of executed tensor operations. However, the employment of LRF is a multi-parametric optimization process involving a huge design space where different design points represent different solutions trading-off the number of FLOPs, the memory size, and the prediction accuracy of the DNN models. As a result, extracting an efficient solution is a complex and time-consuming process. In this work, a new methodology is presented that formulates the LRF problem as a (FLOPs vs. memory vs. prediction accuracy) Design Space Exploration (DSE) problem. Then, the DSE space is drastically pruned by removing inefficient solutions. Our experimental results prove that the design space can be efficiently pruned, therefore extract only a limited set of solutions with improved accuracy, memory, and FLOPs compared to the original (non-factorized) model. Our methodology has been developed as a stand-alone, parameterized module integrated into T3F library of TensorFlow 2.X.

SVSeq2Seq: An Efficient Computational Method for State Vectors in Sequence-to-Sequence Architecture Forecasting

This study proposes an efficient method for computing State Vectors in Sequence-to-Sequence (SVSeq2Seq) architecture to improve the performance of sequence data forecasting, which associates each element with other elements instead of relying only on nearby elements. First, the dependency between two elements is adaptively captured by calculating the relative importance between hidden layers. Second, tensor train decomposition is used to address the issue of dimensionality catastrophe. Third, we further select seven instantiated baseline models for data prediction and compare them with our proposed model on six real-world datasets. The results show that the Mean Square Error (MSE) and Mean Absolute Error (MAE) of our SVSeq2Seq model exhibit significant advantages over the other seven baseline models in predicting the three datasets, i.e., weather, electricity, and PEMS, with MSE/MAE values as low as 0.259/0.260, 0.186/0.285 and 0.113/0.222, respectively. Furthermore, the ablation study demonstrates that the SVSeq2Seq model possesses distinct advantages in sequential forecasting tasks. It is observed that replacing SVSeq2Seq with LPRcode and NMTcode resulted in an increase under an MSE of 18.05 and 10.11 times, and an increase under an MAE of 16.54 and 9.8 times, respectively. In comparative experiments with support vector machines (SVM) and random forest (RF), the performance of the SVSeq2Seq model is improved by 56.88 times in the weather dataset and 73.78 times in the electricity dataset under the MSE metric, respectively. The above experimental results demonstrate both the exceptional rationality and versatility of the SVSeq2Seq model for data forecasting.

Active Fault Detection Based on Tensor Train Decomposition

The paper deals with active fault detection of stochastic systems based on tensor train representation of the Bellman function. The faulty and faulty-free behavior of the system is represented using multiple models. The active fault detection problem is treated as an optimal design problem similar to optimal stochastic control. The original problem is reformulated as a perfect state information problem by introducing an information state that contains statistics computed by a state estimator. The Bellman function is computed using the value iteration algorithm over a rectilinear grid set up in the information state space. Within the value iteration algorithm, the Bellman function is represented using the tensor train decomposition, and considerable attention is devoted to designing a rectilinear grid that respects the constraints placed on the elements of the information state.

End-to-End Optimized Image Compression With Deep Gaussian Process Regression

End-to-end optimization via deep neural networks has facilitated lossy image compression. Existing neural network-based entropy models for end-to-end optimized image compression are limited by parameterized Gaussian distributions with deterministic mean and variance and cannot achieve accurate rate estimation for bottleneck representation with varying statistics. In this paper, we propose a novel entropy model based on deep Gaussian process regression (DGPR) to address this problem. Specifically, the proposed entropy model leverages autoregressive DGPR to flexibly predict the channel-wise posterior distributions of high-dimensional bottleneck representation for entropy coding. Consequently, we develop a well-established bit-rate estimation scheme via posterior inference of DGPR using the learned probabilistic distribution. Furthermore, scalable training is achieved via tensor train decomposition and Monte Carlo sampling to enable tractable variational inference of DGPR. To our best knowledge, this paper is the first attempt to develop the learnable probabilistic model for flexible parameter estimation in entropy modeling. Experimental results show that the proposed model outperforms conventional image compression methods ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., JPEG2000 and BPG) as well as recent end-to-end optimized methods on the Kodak and Tecnick datasets in terms of rate-distortion performance.

Tensor-Enabled Communication-Efficient and Trustworthy Federated Learning for Heterogeneous Intelligent Space–Air–Ground-Integrated IoT

Federated Learning (FL) could provide a promising privacy-preserving intelligent learning paradigm for Space-Air-Ground Integrated Internet of Things (SAGI-IoT) by breaking down data islands and solving the dilemma between data privacy and data sharing. Currently, adaptivity, communication efficiency and model security are the three main challenges faced by FL, and they are rarely considered by existing works simultaneously. Concretely, most existing FL works assume that local models share the same architecture with global model, which is less adaptive and cannot meet the heterogeneous requirements of SAGI-IoT. Exchanging numerous model parameters not only generates massive communication overhead but also poses the risk of privacy leakage. The security of FL based on homomorphic encryption with single private key is weak as well. Given this, this paper proposes a tensor-empowered communication-efficient and trustworthy heterogeneous FL, where various participants could choose suitable heterogeneous local models according to their actual computing and communication environment, so that clients with different capabilities could do what they are good at. Additionally, tensor train decomposition is leveraged to reduce communication parameters while maintaining model performance. The storage requirements and communication overhead for heterogeneous clients are reduced further. Finally, the homomorphic encryption with double trapdoor property is utilized to provide a robust and trustworthy environment, which can defend against the inference attacks from malicious external attackers, honest-but-curious server and internal participating clients. Extensive experimental results show that the proposed approach is more adaptive and can improve communication efficiency as well as protect model security compared with the state-of-the-art.

Multi-level deep domain adaptive adversarial model based on tensor-train decomposition for industrial time series forecasting

The polyester industry is a complex process industry, building a time series prediction model for new production lines or equipment with new sensors can be challenging due to a lack of historical data. The time-series data collected from sensors cross-production-line often exhibit varying distributions. Current domain adaptation (DA) approaches in data-driven time series forecasting primarily concentrate on adjusting either the features or the models, neglecting the intricacies of industrial time series data. Furthermore, constructing deep neural networks for industrial data necessitates substantial computational resources and runtime due to their large and high-dimensional nature. In order to tackle these obstacles, we propose a novel Multi-level deep domain adaptive adversarial model based on tensor-train decomposition (TT-MDAM). Our model aims to strike a dynamic balance between prediction accuracy and runtime efficiency. By integrating multiple perspectives at the feature, trend, and model levels, we leverage DA to enhance the prediction accuracy of our model in the target domain. Additionally, by analyzing the weight matrix of the neural network, we generate a low-rank model to improve operational efficiency. The application of the proposed TT-MDAM approach to both the three-phase flow facility process (TPFF) dataset and a real-world polyester esterification process dataset reveals promising results, outperforming state-of-the-art methodologies in terms of prediction performance. The results indicate that the approach provides a viable solution for building time series prediction models in industrial processes with new equipment or production lines.

Quaternion tensor train rank minimization with sparse regularization in a transformed domain for quaternion tensor completion

The tensor train rank (TT-rank) has achieved promising results in tensor completion due to its ability to capture the global low-rankness of higher-order (>3) tensors. On the other hand, in recent times, quaternions have demonstrated their remarkable suitability as a framework for encoding color pixels, leading to exceptional performance across a range of color image processing tasks. In this paper, we encode the three channels of color pixels using the three imaginary parts of quaternions, leveraging the structural advantages of quaternions to fully preserve the potential relationships between color pixel channels. Subsequently, we extend the TT-rank to higher-order quaternion tensors to capture the global low-rank structure of higher-dimensional data. Specifically, the quaternion tensor train (QTT) decomposition is presented, and based on that the quaternion TT-rank (QTT-rank) is naturally defined. In addition, to utilize the local sparse prior of the quaternion tensor, a general and flexible transform framework is defined. Combining both the global low-rank and local sparse priors of the quaternion tensor, we propose a novel quaternion tensor completion model, i.e., QTT-rank minimization with sparse regularization in a transformed domain. Furthermore, to facilitate QTT-rank minimization for processing color images and enhancing its performance with color videos, we extend KA, a tensor augmentation method, to quaternion tensors, introducing quaternion KA (QKA). Numerical experiments conducted on color image and color video inpainting tasks showcase the superiority of the proposed method over state-of-the-art alternatives.

Tensor shape search for efficient compression of tensorized data and neural networks

Compressing big data and model parameters via tensor decomposition such as the tensor train (TT) format has gained great success in recent years. The application of tensor compression methods requires the data be high dimensional. However, not all the real-world data primarily are high-dimensional, and sometimes reshaping is necessary before the application of tensor compression methods. Meantime, reordering and reshaping data may affect the efficiency of the compression. This work utilizes tensor reshaping to improve the efficiency of tensor compression using the TT format. An optimization model is proposed that maximizes the space-saving of tensor compression with respect to the shape of a given tensor while the compression error is bounded. The study is narrowed down to the TT decomposition and the TT-SVD algorithm is linked with a genetic algorithm (GA) to find an optimal tensor shape. The proposed method is applied to compress RGB images and a neural network to exemplify its capability. The results of the proposed tensor shape search using the GA are also compared with a purely random search. The results demonstrate that the proposed tensor shape search method significantly improves the space-saving and compression ratio of the data compression and enhances the efficiency of tensorized neural networks using the TT decomposition.

A non-local grouping tensor train decomposition model for travel demand analysis concerning categorical independent variables

With the booming of transportation big data, plenty of spatial and temporal features become available for developing travel demand analysis models. However, existing machine learning based travel demand fitting models could face significant model complexity issues once there are multiple discrete/categorical independent variables. This paper embeds a non-local grouping (clustering) process into the tensor train decomposition (NLG-TTD) to characterize the higher-order and nonlinear relationship between categorical variables and travel demand. That is, the transportation dataset with multiple discrete/categorical independent variables and travel demand is treated as a multi-mode tensor, wherein levels of a discrete/categorical variable are regarded as indexes of a mode in the tensor, leading to a “regression on index”. The proposed NLG-TTD model divides the full tensor into non-local sub-tensors, clusters sub-tensors with similarity, conducts tensor train decomposition (TTD) within each cluster, and aggregates cluster-specific learning results. Compared with a conventional TTD that generally works on image completions, NLG-TTD can reveal non-Euclidean and non-linear patterns between categorical/discrete levels in independent variables. It is expected to discover similar travel demand patterns among categorical levels and provide high-level goodness-of-fit. For numerical studies, we use bike-sharing usage data and metro demand data in New York City (NYC), in which several widely-used machine learning models are trained for comparison. The results demonstrate the good performance of the proposed model. Moreover, we apply the NLG-TTD model to analyze and predict the impact of land use shifts on travel demand.

Tensor-Train Decomposition-Based Hybrid Beamforming for Millimeter-Wave Massive Multiple-Input Multiple-Output/Free-Space Optics in Unmanned Aerial Vehicles with Reconfigurable Intelligent Surface Networks

Unmanned aerial vehicles (UAVs) can support low-cost, highly mobile communications while making it possible to establish dedicated terrestrial networks. To overcome the pointing error (PE) and beam misalignment of millimeter-wave large-scale multiple-input multiple-output/free-space optics (MIMO/FSO) caused by UAV jitter, a millimeter-wave massive MIMO/FSO hybrid beamforming method based on tensor train decomposition is proposed. This approach is used for reconfigurable intelligent surface (RIS) network-assisted UAV millimeter-wave massive MIMO/FSO to improve system spectral efficiency. Firstly, the high-dimensional channel of RIS-assisted millimeter-wave massive MIMO/FSO in UAV is represented as a low-dimensional channel by tensor training decomposition. Secondly, the two-way gated recursive unit attention neural network model can effectively solve the FSO PE caused by UAV jitter, and the fast fading channel and Doppler frequency shift are estimated by the Fast Cyclic Tensor Power Method (FCTPM) based on tensor training decomposition. Finally, the RIS phase shift matrix is optimized by singular value decomposition. The hybrid beamforming and RIS phase-shift matrices were estimated by using the low-complexity phase extraction alternating minimization method to solve the beam misalignment problem. Simulation results show that by using the proposed method, the spectrum utilization rate is improved by 23.6% compared with other methods.

A novel recursive least-squares adaptive method for streaming tensor-train decomposition with incomplete observations

Tensor tracking which is referred to as online (adaptive) decomposition of streaming tensors has recently gained much attention in the signal processing community due to the fact that many modern applications generate a huge number of multidimensional data streams over time. In this paper, we propose an effective tensor tracking method via the tensor-train format for decomposing high-order incomplete streaming tensors. On the arrival of new data, the proposed algorithm minimizes a weighted least-squares objective function accounting for both missing values and time-variation constraints on the underlying tensor-train cores, thanks to the recursive least-squares filtering technique and the block coordinate descent framework. Our algorithm is fully capable of tensor tracking from noisy, incomplete, and high-dimensional observations in both static and time-varying environments. Its tracking ability is validated with several experiments on both synthetic and real data.