Tucker Decomposition Research Articles

Three-Order Tucker Decomposition and Reconstruction Detector for Unsupervised Hyperspectral Change Detection

Change detection from multitemporal hyperspectral images has attracted great attention. Most traditional methods using spectral information for change detection treat a hyperspectral image as a two-dimensional matrix and do not take into account inherently structure information of spectrum, which leads to limited detection accuracy. To better approximate both spectral and spatial information, a novel three-order Tucker decomposition and reconstruction detector is proposed for hyperspectral change detection. Initially, Tucker decomposition and reconstruction strategies are used to eliminate the influence of various factors in a multitemporal dataset. Specifically, a singular value accumulation strategy is used to determine principal components in factor matrices. Meanwhile, a spectral angle is used to analyze spectral change after tensor processing in different domains. Finally, a new detector is designed to further improve the detection accuracy. Experiments conducted on five real hyperspectral datasets demonstrate that the proposed detector achieves a better detection performance.

Cross Tensor Approximation Methods for Compression and Dimensionality Reduction

Cross Tensor Approximation (CTA) is a generalization of Cross/skeleton matrix and CUR Matrix Approximation (CMA) and is a suitable tool for fast low-rank tensor approximation. It facilitates interpreting the underlying data tensors and decomposing/compressing tensors so that their structures, such as nonnegativity, smoothness, or sparsity, can be potentially preserved. This paper reviews and extends state-of-the-art deterministic and randomized algorithms for CTA with intuitive graphical illustrations. We discuss several possible generalizations of the CMA to tensors, including CTAs: based on fiber selection, slice-tube selection, and lateral-horizontal slice selection. The main focus is on the CTA algorithms using Tucker and tubal SVD (t-SVD) models while we provide references to other decompositions such as Tensor Train (TT), Hierarchical Tucker (HT), and Canonical Polyadic (CP) decompositions. We evaluate the performance of the CTA algorithms by extensive computer simulations to compress color and medical images and compare their performance.

Hyperspectral Image Superresolution Using Global Gradient Sparse and Nonlocal Low-Rank Tensor Decomposition With Hyper-Laplacian Prior

This article presents a novel global gradient sparse and nonlocal low-rank tensor decomposition model with a hyper-Laplacian prior for hyperspectral image (HSI) superresolution to produce a high-resolution HSI (HR-HSI) by fusing a low-resolution HSI (LR-HSI) with an HR multispectral image (HR-MSI). Inspired by the investigated hyper-Laplacian distribution of the gradients of the difference images between the upsampled LR-HSI and latent HR-HSI, we formulate the relationship between these two datasets as a $\ell _{p}$ $(0 -norm term to enforce spectral preservation. Then, the relationship between the HR-MSI and latent HR-HSI is built using a tensor-based fidelity term to recover the spatial details. To effectively capture the high spatio-spectral-nonlocal similarities of the latent HR-HSI, we design a novel nonlocal low-rank Tucker decomposition to model the 3-D regular tensors constructed from the grouped nonlocal similar HR-HSI cubes. The global spatial-spectral total variation regularization is then adopted to ensure the global spatial piecewise smoothness and spectral consistency of the reconstructed HR-HSI from nonlocal low-rank cubes. Finally, an alternating direction method of multipliers-based algorithm is designed to efficiently solve the optimization problem. Experiments on both the synthetic and real datasets collected by different sensors show the effectiveness of the proposed method, from visual and quantitative assessments.

Randomized Algorithms for Computation of Tucker Decomposition and Higher Order SVD (HOSVD)

Big data analysis has become a crucial part of new emerging technologies such as the internet of things, cyber-physical analysis, deep learning, anomaly detection, etc. Among many other techniques, dimensionality reduction plays a key role in such analyses and facilitates feature selection and feature extraction. Randomized algorithms are efficient tools for handling big data tensors. They accelerate decomposing large-scale data tensors by reducing the computational complexity of deterministic algorithms and the communication among different levels of memory hierarchy, which is the main bottleneck in modern computing environments and architectures. In this article, we review recent advances in randomization for computation of Tucker decomposition and Higher Order SVD (HOSVD). We discuss random projection and sampling approaches, single-pass and multi-pass randomized algorithms and how to utilize them in the computation of the Tucker decomposition and the HOSVD. Simulations on synthetic and real datasets are provided to compare the performance of some of best and most promising algorithms.

Orthogonal Nonnegative Tucker Decomposition

In this paper, we study nonnegative tensor data and propose an orthogonal nonnegative Tucker decomposition (ONTD). We discuss some properties of ONTD and develop a convex relaxation algorithm of th...

SGD_Tucker: A Novel Stochastic Optimization Strategy for Scalable Parallel Sparse Tucker Decomposition

Sparse Tucker Decomposition (STD) algorithms learn a core tensor and a group of factor matrices to obtain an optimal low-rank representation feature for the High-Order, High-Dimension, and Sparse Tensor (HOHDST). However, existing STD algorithms face the problem of intermediate variables explosion which results from the fact that the formation of those variables, i.e., matrices Khatri-Rao product, Kronecker product, and matrix-matrix multiplication, follows the whole elements in sparse tensor. The above problems prevent deep fusion of efficient computation and big data platforms. To overcome the bottleneck, a novel stochastic optimization strategy (SGD Tucker) is proposed for STD which can automatically divide the high-dimension intermediate variables into small batches of intermediate matrices. Specifically, SGD Tucker only follows the randomly selected small samples rather than the whole elements, while maintaining the overall accuracy and convergence rate. In practice, SGD Tucker features the two distinct advancements over the state of the art. First, SGD Tucker can prune the communication overhead for the core tensor in distributed settings. Second, the low data-dependence of SGD Tucker enables fine-grained parallelization, which makes SGD Tucker obtaining lower computational overheads with the same accuracy. Experimental results show that SGD Tucker runs at least 2X faster than the state of the art.

A tucker decomposition based knowledge distillation for intelligent edge applications

Knowledge distillation(KD) has been proven an effective method in intelligent edge computing and have achieved extensive study in recent deep learning research. However, when the teacher network is too stronger compared to the student network, the effect of knowledge distillation is not ideal. Aiming at resolving this problem, an improved method of knowledge distillation (TDKD) is proposed, which enables to transfer the complex mapping functions learned by cumbersome models to relatively simpler models. Firstly, the tucker-2 decomposition was performed on the convolutional layers of the original teacher model to reduce the capacity variance between the teacher network and student network. Then, the decomposed model will be used as a new teacher to participate in knowledge distillation for the student model. The experimental results show that the TDKD method can effectively solve the problem of poor distillation performance, which not only get better results if the KD method is effective, but also can reactivate the invalid KD method to some extents.

Hyperspectral Image Restoration Combining Intrinsic Image Characterization With Robust Noise Modeling

In hyperspectral image (HSI) processing, a fundamental issue is to restore HSI data from various degradations such as noise corruption and information missing. However, most existing methods more or less ignore the abundant prior knowledge on HSIs and the embedded noise, leading to suboptimal performance in practice. In this article, we propose a novel HSI restoration method by fully considering the intrinsic image structures and the complex noise characteristics. For HSIs, the global correlation is captured by the Kronecker-basis-representation-based tensor low-rankness measure, which integrates the insights delivered by both CP and Tucker decompositions; the local regularity is depicted by a plug-and-play spatial-spectral convolutional neural network with strong fitting ability to complex image features. For realistic noise, its statistical characteristics are encoded by a nonidentical and nonindependent distributed mixture of Gaussians distribution with flexible fitting capability. Then, we incorporate these image and noise priors into a probabilistic model based on the maximum a posteriori principle, and develop a solving scheme by combining expectation-maximization and alternating direction method of multipliers. Extensive experimental results on both simulated and real scenarios demonstrate the effectiveness of the proposed method and its superiority over the compared state-of-the- arts.

A Feature Tensor-Based Epileptic Detection Model Based on Improved Edge Removal Approach for Directed Brain Networks.

Electroencephalograph (EEG) plays a significant role in the diagnostics process of epilepsy, but the detection rate is unsatisfactory when the length of interictal EEG signals is relatively short. Although the deliberate attacking theories for undirected brain network based on node removal method can extract potential network features, the node removal method fails to sufficiently consider the directionality of brain electrical activities. To solve the problems above, this study proposes a feature tensor-based epileptic detection method of directed brain networks. First, a directed functional brain network is constructed by calculating the transfer entropy of EEG signals between different electrodes. Second, the edge removal method is used to imitate the disruptions of brain connectivity, which may be related to the disorder of brain diseases, to obtain a sequence of residual networks. After that, topological features of these residual networks are extracted based on graph theory for constructing a five-way feature tensor. To exploit the inherent interactions among multiple modes of the feature tensor, this study uses the Tucker decomposition method to get a core tensor which is finally reshaped into a vector and input into the support vectors machine (SVM) classifier. Experiment results suggest that the proposed method has better epileptic screening performance for short-term interictal EEG data.

Nonnegative and Nonlocal Sparse Tensor Factorization-Based Hyperspectral Image Super-Resolution

Hyperspectral image (HSI) super-resolution refers to enhancing the spatial resolution of a 3-D image with many spectral bands (slices). It is a seriously ill-posed problem when the low-resolution (LR) HSI is the only input. It is better solved by fusing the LR HSI with a high-resolution (HR) multispectral image (MSI) for a 3-D image with both high spectral and spatial resolution. In this article, we propose a novel nonnegative and nonlocal 4-D tensor dictionary learning-based HSI super-resolution model using group-block sparsity. By grouping similar 3-D image cubes into clusters and then conduct super-resolution cluster by cluster using 4-D tensor structure, we not only preserve the structure but also achieve sparsity within the cluster due to the collection of similar cubes. We use 4-D tensor Tucker decomposition and impose nonnegative constraints on the dictionaries and group-block sparsity. Numerous experiments demonstrate that the proposed model outperforms many state-of-the-art HSI super-resolution methods.

Tensor methods for multisensor signal processing

Over the last two decades, tensor-based methods have received growing attention in the signal processing community. In this work, we propose a comprehensive overview of tensor-based models and methods for multisensor signal processing. We present for instance the Tucker decomposition, the Canonical Polyadic Decomposition (CPD), the Tensor-Train Decomposition (TTD), the Structured TTD, including Nested Tucker Train (NTT), as well as the associated optimization strategies. More precisely, we give synthetic descriptions of state-of-art estimators as the Alternating Least Square (ALS) algorithm, the High-Order SVD (HOSVD), and of more advanced algorithms as the Rectified ALS, the TT-SVD/TT-HSVD and the Joint dImensionally Reduction And Factor retrieval Estimator (JIRAFE) scheme. We illustrate the efficiency of the introduced methodological and algorithmic concepts in the context of three important and timely signal processing-based applications: the Direction-Of-Arrival (DOA) estimation based on sensor arrays, multidimensional harmonic retrieval and MIMO wireless communication systems.

Optimizing the Linear Fascicle Evaluation Algorithm for Multi-core and Many-core Systems

Sparse matrix-vector multiplication ( SpMV ) operations are commonly used in various scientific and engineering applications. The performance of the SpMV operation often depends on exploiting regularity patterns in the matrix. Various representations and optimization techniques have been proposed to minimize the memory bandwidth bottleneck arising from the irregular memory access pattern involved. Among recent representation techniques, tensor decomposition is a popular one used for very large but sparse matrices. Post sparse-tensor decomposition, the new representation involves indirect accesses, making it challenging to optimize for multi-cores and even more demanding for the massively parallel architectures, such as on GPUs. Computational neuroscience algorithms often involve sparse datasets while still performing long-running computations on them. The Linear Fascicle Evaluation (LiFE) application is a popular neuroscience algorithm used for pruning brain connectivity graphs. The datasets employed herein involve the Sparse Tucker Decomposition (STD)—a widely used tensor decomposition method. Using this decomposition leads to multiple indirect array references, making it very difficult to optimize on both multi-core and many-core systems. Recent implementations of the LiFE algorithm show that its SpMV operations are the key bottleneck for performance and scaling. In this work, we first propose target-independent optimizations to optimize the SpMV operations of LiFE decomposed using the STD technique, followed by target-dependent optimizations for CPU and GPU systems. The target-independent techniques include: (1) standard compiler optimizations to prevent unnecessary and redundant computations, (2) data restructuring techniques to minimize the effects of indirect array accesses, and (3) methods to partition computations among threads to obtain coarse-grained parallelism with low synchronization overhead. Then, we present the target-dependent optimizations for CPUs such as: (1) efficient synchronization-free thread mapping and (2) utilizing BLAS calls to exploit hardware-specific speed. Following that, we present various GPU-specific optimizations to optimally map threads at the granularity of warps, thread blocks, and grid. Furthermore, to automate the CPU-based optimizations developed for this algorithm, we also extend the PolyMage domain-specific language, embedded in Python. Our highly optimized and parallelized CPU implementation obtains a speedup of 6.3× over the naive parallel CPU implementation running on 16-core Intel Xeon Silver (Skylake-based) system. In addition to that, our optimized GPU implementation achieves a speedup of 5.2× over a reference-optimized GPU code version on NVIDIA’s GeForce RTX 2080 Ti GPU and a speedup of 9.7× over our highly optimized and parallelized CPU implementation.

Hyperspectral and Multispectral Image Fusion via Nonlocal Low-Rank Tensor Decomposition and Spectral Unmixing

Hyperspectral (HS) imaging has shown its superiority in many real applications. However, it is usually difficult to obtain high-resolution (HR) HS images through existing imaging techniques due to the hardware limitations. To improve the spatial resolution of HS images, this article proposes an effective HS-multispectral (HS-MS) image fusion method by combining the ideas of nonlocal low-rank tensor modeling and spectral unmixing. To be more precise, instead of unfolding the HS image into a matrix as done in the literature, we directly represent it as a tensor, then a designed nonlocal Tucker decomposition is used to model its underlying spatial–spectral correlation and the spatial self-similarity. The MS image serves mainly as a data constraint to maintain spatial consistency. To further reduce the spectral distortions in spatial enhancement, endmembers, and abundances from the spectral are used for spectral regularization. An efficient algorithm based on the alternating direction method of multipliers (ADMM) is developed to solve the resulting model. Extensive experiments on four HS image data sets demonstrate the superiority of the proposed method over several state-of-the-art HS-MS image fusion methods.

Enhanced Sparsity Prior Model for Low-Rank Tensor Completion.

Conventional tensor completion (TC) methods generally assume that the sparsity of tensor-valued data lies in the global subspace. The so-called global sparsity prior is measured by the tensor nuclear norm. Such assumption is not reliable in recovering low-rank (LR) tensor data, especially when considerable elements of data are missing. To mitigate this weakness, this article presents an enhanced sparsity prior model for LRTC using both local and global sparsity information in a latent LR tensor. In specific, we adopt a doubly weighted strategy for nuclear norm along each mode to characterize global sparsity prior of tensor. Different from traditional tensor-based local sparsity description, the proposed factor gradient sparsity prior in the Tucker decomposition model describes the underlying subspace local smoothness in real-world tensor objects, which simultaneously characterizes local piecewise structure over all dimensions. Moreover, there is no need to minimize the rank of a tensor for the proposed local sparsity prior. Extensive experiments on synthetic data, real-world hyperspectral images, and face modeling data demonstrate that the proposed model outperforms state-of-the-art techniques in terms of prediction capability and efficiency.

Quality Aware Compression of Multilead Electrocardiogram Signal using 2-mode Tucker Decomposition and Steganography

In this article, a quality controlled compression of multilead electrocardiogram (MECG) is proposed, based on tensor analysis, and implemented upon 3D beat tensor of MECG. To reduce computational complications and execution time, a new approach of principal component analysis (PCA), based on 2-mode Tucker Decomposition, is introduced. In order to maintain relevant features of MECG after reconstruction, multi agent supervised learning system (MASLS) based optimal quantization of each fiber of core tensor is introduced, to limit the percentage root mean squared difference (PRD) within a specified value, maintaining high compression ratio (CR). The MASLS is previously trained offline, using features of tensor fibers, along with optimized quantization levels of those fibers, obtained from a particle swarm optimization (PSO), as reference. In addition, to hide patient’s confidential information, steganography is performed within the core tensor followed by generation of a’ secret key’, which is necessary, while decrypting those information during reconstruction. The whole algorithm is implemented on several MECG records, available in PTB Diagnostic ECG database, and compression result is compared by formation of n-beat tensor separately, using ‘n’ number of successive (‘n’ = 5, 10 and 15) beats. After testing on 547 data, average CR of 22, 41.5, 55.4, PRD of 3.62, 4.96, 5.59 and PRD normalized (PRDN) of 3.61, 4.94, 5.57 are achieved for 5, 10, 15-beat tensor, respectively. This proposed algorithm has provided superior result as compared to recently published works on MECG data compression.

On the Compression of Translation Operator Tensors in FMM-FFT-Accelerated SIE Simulators via Tensor Decompositions

Tensor decomposition methodologies are proposed to reduce the memory requirement of translation operator tensors arising in the fast multipole method-fast Fourier transform (FMM-FFT)-accelerated surface integral equation (SIE) simulators. These methodologies leverage Tucker, hierarchical Tucker (H-Tucker), and tensor train (TT) decompositions to compress the FFT'ed translation operator tensors stored in three-dimensional (3D) and four-dimensional (4D) array formats. Extensive numerical tests are performed to demonstrate the memory saving achieved by and computational overhead introduced by these methodologies for different simulation parameters. Numerical results show that the H-Tucker-based methodology for 4D array format yields the maximum memory saving while Tucker-based methodology for 3D array format introduces the minimum computational overhead. For many practical scenarios, all methodologies yield a significant reduction in the memory requirement of translation operator tensors while imposing negligible/acceptable computational overhead.

Sparse Tucker Tensor Decomposition on a Hybrid FPGA–CPU Platform

Recommendation systems, social network analysis, medical imaging, and data mining often involve processing sparse high-dimensional data. Such high-dimensional data are naturally represented as tensors, and they cannot be efficiently processed by conventional matrix or vector computations. Sparse Tucker decomposition is an important algorithm for compressing and analyzing these sparse high-dimensional data sets. When energy efficiency and data privacy are major concerns, hardware accelerators on resource-constraint platforms become crucial for the deployment of tensor algorithms. In this work, we propose a hybrid computing framework containing CPU and FPGA to accelerate sparse Tucker factorization. This algorithm has three main modules: tensor-times-matrix (TTM), Kronecker products, and QR decomposition with column pivoting (QRP). In addition, we accelerate the former two modules on a Xilinx FPGA and the latter one on a CPU. Our hybrid platform achieves $23.6 \times \sim 1091\times$ speedup and over $93.519\% \sim 99.514 \%$ energy savings compared with CPU on the synthetic and real-world datasets.

MERACLE: Constructive Layer-Wise Conversion of a Tensor Train into a MERA

In this article, two new algorithms are presented that convert a given data tensor train into either a Tucker decomposition with orthogonal matrix factors or a multi-scale entanglement renormalization ansatz (MERA). The Tucker core tensor is never explicitly computed but stored as a tensor train instead, resulting in both computationally and storage efficient algorithms. Both the multilinear Tucker-ranks as well as the MERA-ranks are automatically determined by the algorithm for a given upper bound on the relative approximation error. In addition, an iterative algorithm with low computational complexity based on solving an orthogonal Procrustes problem is proposed for the first time to retrieve optimal rank-lowering disentangler tensors, which are a crucial component in the construction of a low-rank MERA. Numerical experiments demonstrate the effectiveness of the proposed algorithms together with the potential storage benefit of a low-rank MERA over a tensor train.

Hybrid tensor decomposition in neural network compression

Deep neural networks (DNNs) have enabled impressive breakthroughs in various artificial intelligence (AI) applications recently due to its capability of learning high-level features from big data. However, the current demand of DNNs for computational resources especially the storage consumption is growing due to that the increasing sizes of models are being required for more and more complicated applications. To address this problem, several tensor decomposition methods including tensor-train (TT) and tensor-ring (TR) have been applied to compress DNNs and shown considerable compression effectiveness. In this work, we introduce the hierarchical Tucker (HT), a classical but rarely-used tensor decomposition method, to investigate its capability in neural network compression. We convert the weight matrices and convolutional kernels to both HT and TT formats for comparative study, since the latter is the most widely used decomposition method and the variant of HT. We further theoretically and experimentally discover that the HT format has better performance on compressing weight matrices, while the TT format is more suited for compressing convolutional kernels. Based on this phenomenon we propose a strategy of hybrid tensor decomposition by combining TT and HT together to compress convolutional and fully connected parts separately and attain better accuracy than only using the TT or HT format on convolutional neural networks (CNNs). Our work illuminates the prospects of hybrid tensor decomposition for neural network compression.

The correlation-based tucker decomposition for hyperspectral image compression

Tucker decomposition (TD) is widely used in hyperspectral image (HSI) processing. Generally, the performance of TD-based method depends on the core tensor and factor matrices, while the construction of core tensor and factor matrices is still a research topic. We give the detailed discussion about the correlation and performance of TD-based methods in this paper. Since TD is solved by singular value decomposition (SVD), the construction of core tensor and factor matrices should be determined by the distribution of singular energy of each mode-n matricization. Depending on the discussion, we propose a correlation-based Tucker decomposition (CBTD) method to construct the core tensor and factor matrices. As a general method, this proposed CBTD can be employed in any TD-based method of Nth-order tensor. The analysis on real HSI data verifies our conclusion about correlation and good performance of CBTD. Besides, the proposed CBTD method has better ability to improve the performance of HSI compression than other state-of-the-art methods.