Tucker Tensor Research Articles

Large-scale tucker Tensor factorization for sparse and accurate decomposition

Large-scale tucker Tensor factorization for sparse and accurate decomposition

Robust Tensor Recovery in Impulsive Noise Based on Correntropy and Hybrid Tensor Sparsity

Robust tensor recovery aims to reconstruct a multidimensional tensor from its observations contaminated by noise. In this brief, a new formulation based on correntropy and hybrid tensor sparsity measure is proposed for robust tensor recovery in the environment of impulsive noise. The robust correntropy measure has shown satisfactory robustness against large outliers in various scenarios recently. Meanwhile, in this formulation, the hybrid tensor sparsity measure combining the advantages of Tucker and CP tensor rank can better characterize the tensor sparsity. To solve the proposed formulation effectively, an efficient large-scale optimization algorithm is derived based on the framework of alternating direction method of multipliers (ADMM) and half-quadratic optimization technique. The results of multispectral image (MSI) data recovery indicate that the proposed algorithm can achieve robust tensor recovery in the environment of impulsive noise.

Algorithm and Hardware Co-Design of Energy-Efficient LSTM Networks for Video Recognition with Hierarchical Tucker Tensor Decomposition

Long short-term memory (LSTM) is a type of powerful deep neural network that has been widely used in many sequence analysis and modeling applications. However, the large model size problem of LSTM networks make their practical deployment still very challenging, especially for the video recognition tasks that require high-dimensional input data. Aiming to overcome this limitation and fully unlock the potentials of LSTM models, in this paper we propose to perform algorithm and hardware co-design towards high-performance energy-efficient LSTM networks. At algorithm level, we propose to develop <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fully decomposed hierarchical Tucker (FDHT)</i> structure-based LSTM, namely FDHT-LSTM, which enjoys ultra-low model complexity while still achieving high accuracy. In order to fully reap such attractive algorithmic benefit, we further develop the corresponding customized hardware architecture to support the efficient execution of the proposed FDHT-LSTM model. With the delicate design of memory access scheme, the complicated matrix transformation can be efficiently supported by the underlying hardware without any access conflict in an on-the-fly way. Our evaluation results show that both the proposed ultra-compact FDHT-LSTM models and the corresponding hardware accelerator achieve very high performance. Compared with the state-of-the-art compressed LSTM models, FDHT-LSTM enjoys both order-of-magnitude reduction (more than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1000 \times$</tex-math></inline-formula> ) in model size and significant accuracy improvement (0.6% to 12.7%) across different video recognition datasets. Meanwhile, compared with the state-of-the-art tensor decomposed model-oriented hardware TIE, our proposed FDHT-LSTM architecture achieve <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2.5\times$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.46\times$</tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2.41\times$</tex-math></inline-formula> increase in throughput, area efficiency and energy efficiency, respectively on LSTM-Youtube workload. For LSTM-UCF workload, our proposed design also outperforms TIE with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.9\times$</tex-math></inline-formula> higher throughput, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.83\times$</tex-math></inline-formula> higher energy efficiency and comparable area efficiency.

Blind Unmixing of Hyperspectral Images Based on L 1 Norm and Tucker Tensor Decomposition

Most of the traditional hyperspectral unmixing methods are based on the matrix and often ignore the spatial information of hyperspectral images (HSIs). In recent years, tensor-based methods have been gradually used in hyperspectral unmixing, owing to their ability to completely preserve the real spatial structure of HSIs. A blind unmixing method for HSIs based on an L₁ regular term and tucker tensor decomposition (BUTTDL1) is proposed, which describes the low rank of abundance by tucker tensor decomposition in the form of a third-order tensor and increases the sparse characterization of abundance by the L₁ norm. A comparative experiment is performed on two simulation datasets. Compared with the latest method unmixing with low-rank tensor regularization algorithm accounting for endmembers variability (ULTRA-V), in the simulation dataset Data Cube 1 (DC1), the endmember mean square error (MSE) of BUTTDL1 is decreased by 1.1, and the abundance MSE is decreased by 8.6. In the simulation dataset DC2, the endmember MSE is decreased by 2.4, and the abundance MSE is decreased by 6.63.

Analysis of Tensor Approximation Schemes for Continuous Functions

In this article, we analyze tensor approximation schemes for continuous functions. We assume that the function to be approximated lies in an isotropic Sobolev space and discuss the cost when approximating this function in the continuous analogue of the Tucker tensor format or of the tensor train format. We especially show that the cost of both approximations are dimension-robust when the Sobolev space under consideration provides appropriate dimension weights.

Hyperspectral and Multispectral Image Fusion Using Coupled Non-Negative Tucker Tensor Decomposition

Fusing a low spatial resolution hyperspectral image (HSI) with a high spatial resolution multispectral image (MSI), aiming to produce a super-resolution hyperspectral image, has recently attracted increasing research interest. In this paper, a novel approach based on coupled non-negative tensor decomposition is proposed. The proposed method performs a tucker tensor factorization of a low resolution hyperspectral image and a high resolution multispectral image under the constraint of non-negative tensor decomposition (NTD). The conventional matrix factorization methods essentially lose spatio-spectral structure information when stacking the 3D data structure of a hyperspectral image into a matrix form. Moreover, the spectral, spatial, or their joint structural features have to be imposed from the outside as a constraint to well pose the matrix factorization problem. The proposed method has the advantage of preserving the spatio-spectral structure of hyperspectral images. In this paper, the NTD is directly imposed on the coupled tensors of the HSI and MSI. Hence, the intrinsic spatio-spectral structure of the HSI is represented without loss, and spatial and spectral information can be interdependently exploited. Furthermore, multilinear interactions of different modes of the HSIs can be exactly modeled with the core tensor of the Tucker tensor decomposition. The proposed method is straightforward and easy to implement. Unlike other state-of-the-art approaches, the complexity of the proposed approach is linear with the size of the HSI cube. Experiments on two well-known datasets give promising results when compared with some recent methods from the literature.

Efficient propagation of the hierarchical equations of motion using the Tucker and hierarchical Tucker tensors.

We develop new methods to efficiently propagate the hierarchical equations of motion (HEOM) by using the Tucker and hierarchical Tucker (HT) tensors to represent the reduced density operator and auxiliary density operators. We first show that by employing the split operator method, the specific structure of the HEOM allows a simple propagation scheme using the Tucker tensor. When the number of effective modes in the HEOM increases and the Tucker representation becomes intractable, the split operator method is extended to the binary tree structure of the HT representation. It is found that to update the binary tree nodes related to a specific effective mode, we only need to propagate a short matrix product state constructed from these nodes. Numerical results show that by further employing the mode combination technique commonly used in the multi-configuration time-dependent Hartree approaches, the binary tree representation can be applied to study excitation energy transfer dynamics in a fairly large system including over 104 effective modes. The new methods may thus provide a promising tool in simulating quantum dynamics in condensed phases.

An unconventional robust integrator for dynamical low-rank approximation

We propose and analyse a numerical integrator that computes a low-rank approximation to large time-dependent matrices that are either given explicitly via their increments or are the unknown solution to a matrix differential equation. Furthermore, the integrator is extended to the approximation of time-dependent tensors by Tucker tensors of fixed multilinear rank. The proposed low-rank integrator is different from the known projector-splitting integrator for dynamical low-rank approximation, but it retains the important robustness to small singular values that has so far been known only for the projector-splitting integrator. The new integrator also offers some potential advantages over the projector-splitting integrator: It avoids the backward time integration substep of the projector-splitting integrator, which is a potentially unstable substep for dissipative problems. It offers more parallelism, and it preserves symmetry or anti-symmetry of the matrix or tensor when the differential equation does. Numerical experiments illustrate the behaviour of the proposed integrator.

Efficient multi-way sparsity estimation for hyperspectral image processing

Sparsity and low rankness are essential properties of real-world data. The sparsity attribute mentions that the data contain very few non-zero elements. On the other hand, the low-rank property characterizes redundancy in the data. The concept of sparsity and low rankness is central to numerous signal processing tasks such as signal inversion, data compression, and noise removal. With the advent of multi-sensor and multi-dimensional data, we frequently encounter multi-way data or tensors with sparse and low-rank structure. Prevalent vector or matrix-based sparsity estimation criterias cannot conveniently characterize the sparsity structure of multi-way data. Hence, we require efficient criteria to quantify the sparsity structure of multi-way data. Sparsity plays a crucial aspect in hyperspectral image processing tasks such as unmixing and denoising. Hyperspectral images contain an inherent multi-way structure; we perform multi-way sparsity estimation in hyperspectral image processing. We introduce an efficient criterion to estimate multi-way sparsity, facilitating signal inversion, and noise removal tasks. The proposed sparsity criteria combine the sparsity of the core tensor obtained by Tucker tensor decomposition, the low rankness of the tensor, and the approximated rank. Since the core tensor obtained by tucker decomposition is highly sparse, the core tensor sparsity reflects the sparsity of the whole tensor indirectly. On the other hand, the nuclear norm and Stein’s unbiased risk estimate quantifies the low rankness of the tensor. Unlike other approaches, our proposed measure takes both low rankness and sparsity attribute into account. In addition, our proposed sparsity measure also satisfies some of the desirable properties of ideal sparsity measures. We demonstrate the efficacy of our proposed sparsity quantification measure in applications such as hyperspectral image denoising, hyperspectral unmixing tasks as demonstrated in the real image experiments.

Tensor-structured algorithm for reduced-order scaling large-scale Kohn\u2013Sham density functional theory calculations

We present a tensor-structured algorithm for efficient large-scale density functional theory (DFT) calculations by constructing a Tucker tensor basis that is adapted to the Kohn–Sham Hamiltonian and localized in real-space. The proposed approach uses an additive separable approximation to the Kohn–Sham Hamiltonian and an L1 localization technique to generate the 1-D localized functions that constitute the Tucker tensor basis. Numerical results show that the resulting Tucker tensor basis exhibits exponential convergence in the ground-state energy with increasing Tucker rank. Further, the proposed tensor-structured algorithm demonstrated sub-quadratic scaling with system-size for both systems with and without a gap, and involving many thousands of atoms. This reduced-order scaling has also resulted in the proposed approach outperforming plane-wave DFT implementation for systems beyond 2000 electrons.

Multiview deep learning based on tensor decomposition and its application in fault detection of overhead contact systems

This article mainly focuses on the most common types of high-speed railways malfunctions in overhead contact systems, namely, unstressed droppers, foreign-body invasions, and pole number-plate malfunctions, to establish a deep-network detection model. By fusing the feature maps of the shallow and deep layers in the pretraining network, global and local features of the malfunction area are combined to enhance the network's ability of identifying small objects. Further, in order to share the fully connected layers of the pretraining network and reduce the complexity of the model, Tucker tensor decomposition is used to extract features from the fused-feature map. The operation greatly reduces training time. Through the detection of images collected on the Lanxin railway line, experiments result show that the proposed multiview Faster R-CNN based on tensor decomposition had lower miss probability and higher detection accuracy for the three types faults. Compared with object-detection methods YOLOv3, SSD, and the original Faster R-CNN, the average miss probability of the improved Faster R-CNN model in this paper is decreased by 37.83%, 51.27%, and 43.79%, respectively, and average detection accuracy is increased by 3.6%, 9.75%, and 5.9%, respectively.

Krylov-Levenberg-Marquardt Algorithm for Structured Tucker Tensor Decompositions

Structured Tucker tensor decomposition models complete or incomplete multiway data sets (tensors), where the core tensor and the factor matrices can obey different constraints. The model includes block-term decomposition or canonical polyadic decomposition as special cases. We propose a very flexible optimization method for the structured Tucker decomposition problem, based on the second-order Levenberg-Marquardt optimization, using an approximation of the Hessian matrix by the Krylov subspace method. An algorithm with limited sensitivity of the decomposition is included. The proposed algorithm is shown to perform well in comparison to existing tensor decomposition methods.

Hyperspectral image, video compression using sparse tucker tensor decomposition

Hyperspectral image and videos provide rich spectral information content, which facilitates accurate classification, unmixing, temporal change detection, and so on. However, with the rapid improvements in technology, the data size has increased many folds. To properly handle the enormous data volume, efficient methods are required to compress the data. This paper proposes a multi-way approach for compression of the hyperspectral image or video sequence. In this approach, a differential representation of the data is first obtained. In the case of hyperspectral images, the difference between consecutive bands is obtained and in case of videos, the difference between consecutive frames is computed. In the next step, a sparse Tucker tensor decomposition is performed and the sparse core tensor obtained. Finally, the core tensor and the corresponding factor matrices are truncated and the data encoded to obtain the compressed version for transmission. The compression method utilises the multi-way structure of the data and hence can be extended for hyperspectral videos. Experimental results on several real data imply that the proposed compression approach obtains better efficiency in terms of compression ratio, signal to noise ratio.

Time Integration of Tree Tensor Networks

Dynamical low-rank approximation by tree tensor networks is studied for the data-sparse approximation of large time-dependent data tensors and unknown solutions to tensor differential equations. A time integration method for tree tensor networks of prescribed tree rank is presented and analyzed. It extends the known projector-splitting integrators for dynamical low-rank approximation by matrices and rank-constrained Tucker tensors and is shown to inherit their favorable properties. The integrator is based on recursively applying the low-rank Tucker tensor integrator. In every time step, the integrator climbs up and down the tree: it uses a recursion that passes from the root to the leaves of the tree for the construction of initial value problems on subtree tensor networks using appropriate restrictions and prolongations, and another recursion that passes from the leaves to the root for the update of the factors in the tree tensor network. The integrator reproduces given time-dependent tree tensor networks of the specified tree rank exactly and is robust to the typical presence of small singular values in matricizations of the connection tensors, in contrast to standard integrators applied to the differential equations for the factors in the dynamical low-rank approximation by tree tensor networks.

Spectral-Image Decomposition With Energy-Fusion Sensing for Spectral CT Reconstruction

Spectral-computed tomography (CT) has been demonstrating its great advantages in lesion detection, tissue characterization, and material decomposition. However, the quality of images is often significantly corrupted with various noises, which brings a great challenge for its applications. Because the channel-wise images from different energy interval share similar structure and physical message, the spatial sparsity, global correlation across the spectrum (GCS), and nonlocal self-similarity (NSS) as three important characteristics are employed to spectral CT reconstruction. In this study, we propose a spectral-image decomposition with energy-fusion sensing (SIDES) reconstruction method, which encourages to obtain better quality spectral images and material decomposition results by establishing a unified tensor decomposition model. First, considering the noise distribution in channel-wise and the difference of linear attenuation coefficients within channel-cross, an adaptive weighted full-spectrum prior image as additional supervised information is incorporated to formulate a new weighted prior image-based tensor. Cooperating with original image tensor, they fully explore the spatial sparsity, GCS, and NSS properties. Then, we formulate nonlocal similar patch-based tensor groups to encode the NSS property from image-domain and residual-image-domain (which is expanded by prior-image and image-self). Next, low-rank regularized Tucker tensor decomposition is employed to fully explore the intrinsic knowledge with the help of prior-image supervision. Finally, the relaxed convex optimization model is optimized by dividing reconstruction model into several subproblem using split-Bregman method. Numerical simulations and real experiments are designed to validate and evaluate the SIDES method and the results demonstrate that the SIDES reconstruction outperforms the state-of-the-art.

Non-adiabatic quantum dynamics without potential energy surfaces based on second-quantized electrons: Application within the framework of the MCTDH method.

A first principles quantum formalism to describe the non-adiabatic dynamics of electrons and nuclei based on a second quantization representation (SQR) of the electronic motion combined with the usual representation of the nuclear coordinates is introduced. This procedure circumvents the introduction of potential energy surfaces and non-adiabatic couplings, providing an alternative to the Born-Oppenheimer approximation. An important feature of the molecular Hamiltonian in the mixed first quantized representation for the nuclei and the SQR representation for the electrons is that all degrees of freedom, nuclear positions and electronic occupations, are distinguishable. This makes the approach compatible with various tensor decomposition Ansätze for the propagation of the nuclear-electronic wavefunction. Here, we describe the application of this formalism within the multi-configuration time-dependent Hartree framework and its multilayer generalization, corresponding to Tucker and hierarchical Tucker tensor decompositions of the wavefunction, respectively. The approach is applied to the calculation of the photodissociation cross section of the HeH+ molecule under extreme ultraviolet irradiation, which features non-adiabatic effects and quantum interferences between the two possible fragmentation channels, He + H+ and He+ + H. These calculations are compared with the usual description based on ab initio potential energy surfaces and non-adiabatic coupling matrix elements, which fully agree. The proof-of-principle calculations serve to illustrate the advantages and drawbacks of this formalism, which are discussed in detail, as well as possible ways to overcome them. We close with an outlook of possible application domains where the formalism might outperform the usual approach, for example, in situations that combine a strong static correlation of the electrons with non-adiabatic electronic-nuclear effects.

Hybrid tensor decomposition in neural network compression

Deep neural networks (DNNs) have enabled impressive breakthroughs in various artificial intelligence (AI) applications recently due to its capability of learning high-level features from big data. However, the current demand of DNNs for computational resources especially the storage consumption is growing due to that the increasing sizes of models are being required for more and more complicated applications. To address this problem, several tensor decomposition methods including tensor-train (TT) and tensor-ring (TR) have been applied to compress DNNs and shown considerable compression effectiveness. In this work, we introduce the hierarchical Tucker (HT), a classical but rarely-used tensor decomposition method, to investigate its capability in neural network compression. We convert the weight matrices and convolutional kernels to both HT and TT formats for comparative study, since the latter is the most widely used decomposition method and the variant of HT. We further theoretically and experimentally discover that the HT format has better performance on compressing weight matrices, while the TT format is more suited for compressing convolutional kernels. Based on this phenomenon we propose a strategy of hybrid tensor decomposition by combining TT and HT together to compress convolutional and fully connected parts separately and attain better accuracy than only using the TT or HT format on convolutional neural networks (CNNs). Our work illuminates the prospects of hybrid tensor decomposition for neural network compression.

Multi-aspect renewable energy forecasting

The increasing presence of renewable energy plants has created new challenges such as grid integration, load balancing and energy trading, making it fundamental to provide effective prediction models. Recent approaches in the literature have shown that exploiting spatio-temporal autocorrelation in data coming from multiple plants can lead to better predictions. Although tensor models and techniques are suitable to deal with spatio-temporal data, they have received little attention in the energy domain. In this paper, we propose a new method based on the Tucker tensor decomposition, capable of extracting a new feature space for the learning task. For evaluation purposes, we have investigated the performance of predictive clustering trees with the new feature space, compared to the original feature space, in three renewable energy datasets. The results are favorable for the proposed method, also when compared with state-of-the-art algorithms.

Sparse regularized low-rank tensor regression with applications in genomic data analysis

Many applications in biomedical informatics deal with data in the tensor form. Traditional regression methods which take vectors as covariates may encounter difficulties in handling tensors due to their ultrahigh dimensionality and complex structure. In this paper, we introduce a novel sparse regularized Tucker tensor regression model to exploit the structure of tensor covariates and perform feature selection on tensor data. Based on Tucker decomposition of the regression coefficient tensor, we reduce the ultrahigh dimensionality to a manageable level. To make our model identifiable, we impose the orthonormality constraint on the factor matrices. Unlike previous tensor regression models that impose sparse penalty on the factor matrices of the coefficient tensor, our model directly imposes sparse penalty on the coefficient tensor to select the relevant features on tensor data. An efficient optimization algorithm based on alternating direction method of multiplier (ADMM) algorithm is designed to solve our proposed model. The performance of our model is evaluated on both synthetic and real genomic data. Experiment results on synthetic data demonstrate that our model could identify the true related signals more accurately than other state-of-the-art regression models. The analysis on genomic data of melanoma demonstrates that our model can achieve better prediction performance and identify markers with important implications. Our model and the associated studies can provide useful insights to disease or pathogenesis mechanisms, and will benefit further studies in variable selection.

PMU Missing Data Recovery Using Tensor Decomposition

The paper proposes a new approach for the recovery of missing data from phasor measurement units (PMUs). The approach is based on the application of tensor decomposition to PMU data organized as three-dimensional tensors with respect to time, location and type of variables. The organization of the PMU data is in the form of a bus-oriented data structure and a branch-oriented data structure. Two versions of the approach are introduced. The first version uses polyadic tensor decomposition and the second version uses Tucker tensor decomposition. Several case studies are conducted validating the proposed approaches including measured PMU data in the New York transmission system. Comparison with existing approaches is performed to demonstrate the improvement in missing data recovery accuracy achieved by the new approach.