Higher-order Tensor Decomposition Research Articles

BATUDE: Budget-Aware Neural Network Compression Based on Tucker Decomposition

Model compression is very important for the efficient deployment of deep neural network (DNN) models on resource-constrained devices. Among various model compression approaches, high-order tensor decomposition is particularly attractive and useful because the decomposed model is very small and fully structured. For this category of approaches, tensor ranks are the most important hyper-parameters that directly determine the architecture and task performance of the compressed DNN models. However, as an NP-hard problem, selecting optimal tensor ranks under the desired budget is very challenging and the state-of-the-art studies suffer from unsatisfied compression performance and timing-consuming search procedures. To systematically address this fundamental problem, in this paper we propose BATUDE, a Budget-Aware TUcker DEcomposition-based compression approach that can efficiently calculate optimal tensor ranks via one-shot training. By integrating the rank selecting procedure to the DNN training process with a specified compression budget, the tensor ranks of the DNN models are learned from the data and thereby bringing very significant improvement on both compression ratio and classification accuracy for the compressed models. The experimental results on ImageNet dataset show that our method enjoys 0.33% top-5 higher accuracy with 2.52X less computational cost as compared to the uncompressed ResNet-18 model. For ResNet-50, the proposed approach enables 0.37% and 0.55% top-5 accuracy increase with 2.97X and 2.04X computational cost reduction, respectively, over the uncompressed model.

Higher-order tensor decomposition based scalp-to-intracranial EEG projection for detection of interictal epileptiform discharges

Objective. Interictal epileptiform discharges (IEDs) occur between two seizures onsets. IEDs are mainly captured by intracranial recordings and are often invisible over the scalp. This study proposes a model based on tensor factorization to map the time-frequency (TF) features of scalp EEG (sEEG) to the TF features of intracranial EEG (iEEG) in order to detect IEDs from over the scalp with high sensitivity. Approach. Continuous wavelet transform is employed to extract the TF features. Time, frequency, and channel modes of IED segments from iEEG recordings are concatenated into a four-way tensor. Tucker and CANDECOMP/PARAFAC decomposition techniques are employed to decompose the tensor into temporal, spectral, spatial, and segmental factors. Finally, TF features of both IED and non-IED segments from scalp recordings are projected onto the temporal components for classification. Main results. The model performance is obtained in two different approaches: within- and between-subject classification approaches. Our proposed method is compared with four other methods, namely a tensor-based spatial component analysis method, TF-based method, linear regression mapping model, and asymmetric–symmetric autoencoder mapping model followed by convolutional neural networks. Our proposed method outperforms all these methods in both within- and between-subject classification approaches by respectively achieving 84.2% and 72.6% accuracy values. Significance. The findings show that mapping sEEG to iEEG improves the performance of the scalp-based IED detection model. Furthermore, the tensor-based mapping model outperforms the autoencoder- and regression-based mapping models.

Tensor Data Imputation by PARAFAC with Updated Chaotic Biases by Adam Optimizer

The big data pattern analysis suffers from incorrect responses due to missing data entries in the real world. Data collected for digital movie platforms like Netflix and intelligent transportation systems is Spatio-temporal data. Extracting the latent and explicit features from this data is a challenge. We present the high dimensional data imputation problem as a higher-order tensor decomposition. The regularized and biased PARAFAC decomposition is proposed to generate the missing data entries. The biases are created and updated by a chaotic exponential factor in Adam’s optimization, which reduces the imputation error. This chaotic perturbed exponentially update in the learning rate replaces the fixed learning rate in the bias update by Adam optimization. The idea has experimented with Netflix and traffic datasets from Guangzhou, China.

Higher order tensor decomposition for proportional myoelectric control based on muscle synergies

Muscle synergies have recently been utilised in myoelectric control systems. Thus far, all proposed synergy-based systems rely on matrix factorisation methods. However, this is limited in terms of task-dimensionality. Here, the potential application of higher-order tensor decomposition as a framework for proportional myoelectric control is demonstrated. A novel constrained Tucker decomposition (consTD) technique of synergy extraction is proposed for synergy-based myoelectric control model and compared with state-of-the-art matrix factorisation models. The extracted synergies were used to estimate control signals for the wrist's Degree of Freedom (DoF) through direct projection. The consTD model was able to estimate the control signals for each DoF by utilising all data in one 3rd-order tensor. This is contrast with matrix factorisation models where data are segmented for each DoF and then the synergies often have to be realigned. Moreover, the consTD method offers more information by providing additional shared synergies, unlike matrix factorisation methods. The extracted control signals were fed to a ridge regression to estimate the wrist's kinematics based on real glove data. The Coefficient of Determination (R2) for the reconstructed wrist position showed that the proposed consTD was higher than matrix factorisation methods. In sum, this study provides the first proof of concept for the use of higher-order tensor decomposition in proportional myoelectric control and it highlights the potential of tensors to provide an objective and direct approach to identify synergies.

Spatial Channel Covariance Estimation for Hybrid Architectures Based on Tensor Decompositions

Spatial channel covariance information can replace full instantaneous channel state information for the analog precoder design in hybrid analog/digital architectures. Obtaining spatial channel covariance estimation, however, is challenging in the hybrid structure due to the use of fewer radio frequency (RF) chains than the number of antennas. In this paper, we propose a spatial channel covariance estimation method based on higher-order tensor decomposition for spatially sparse time-varying frequency-selective channels. The proposed method leverages the fact that the channel can be represented as a low-rank higher-order tensor. We also derive the Cramer-Rao lower bound on the estimation accuracy of the proposed method. Numerical results and theoretical analysis show that the proposed tensor-based approach achieves higher estimation accuracy in comparison with prior compressive-sensing-based approaches or conventional angle-of-arrival estimation approaches.

Individualized Multilayer Tensor Learning With an Application in Imaging Analysis

This work is motivated by multimodality breast cancer imaging data, which is quite challenging in that the signals of discrete tumor-associated microvesicles are randomly distributed with heterogeneous patterns. This imposes a significant challenge for conventional imaging regression and dimension reduction models assuming a homogeneous feature structure. We develop an innovative multilayer tensor learning method to incorporate heterogeneity to a higher-order tensor decomposition and predict disease status effectively through utilizing subject-wise imaging features and multimodality information. Specifically, we construct a multilayer decomposition which leverages an individualized imaging layer in addition to a modality-specific tensor structure. One major advantage of our approach is that we are able to efficiently capture the heterogeneous spatial features of signals that are not characterized by a population structure as well as integrating multimodality information simultaneously. To achieve scalable computing, we develop a new bi-level block improvement algorithm. In theory, we investigate both the algorithm convergence property, tensor signal recovery error bound and asymptotic consistency for prediction model estimation. We also apply the proposed method for simulated and human breast cancer imaging data. Numerical results demonstrate that the proposed method outperforms other existing competing methods. Supplementary materials for this article are available online.

A higher-order tensor voting-based approach for road junction detection and delineation from airborne LiDAR data

The location and delineation of road junctions produce valuable inputs for road network understanding and topology reconstruction; the results also serve as basic information for road construction planning, emergency rescue, and large-scale data registration. This study addresses fully automatic road junction detection and delineation from airborne lidar (or LiDAR – light detection and ranging) data. For this purpose, a higher-order tensor voting-based approach is developed, which can locate road junctions by identifying multidirectional features in encoded higher-order tensor models. This approach comprises three main steps. First, a roughness-enhanced Gabor filter is implemented to process both the lidar-derived intensity image and the normalized digital surface model to extract homogeneous elongated structural keypoints. Second, these keypoints are fed into a higher-order tensor voting algorithm to recognize road junction candidates. We modified the method for encoding higher-order tensor models to enable the algorithm to deal with both antipodal symmetric and nonsymmetric junctions. After a higher-order tensor decomposition, the number of road branches and their directions are obtained; road junction candidates were then located. Third, by matching the results with a geometric template, road junction delineation is performed to retrieve the junction center position and the road branch directions and widths. We tested our method in two study areas and assessed the quality of the results by comparing them with ground-truth data. The results proved the validity of this approach in both urban and semiurban scenarios. We also compared our method with other junction detection algorithms and analyzed the parameter influence of the approach to essential parameters.

Crossing fibers detection with an analytical high order tensor decomposition.

Diffusion magnetic resonance imaging (dMRI) is the only technique to probe in vivo and noninvasively the fiber structure of human brain white matter. Detecting the crossing of neuronal fibers remains an exciting challenge with an important impact in tractography. In this work, we tackle this challenging problem and propose an original and efficient technique to extract all crossing fibers from diffusion signals. To this end, we start by estimating, from the dMRI signal, the so-called Cartesian tensor fiber orientation distribution (CT-FOD) function, whose maxima correspond exactly to the orientations of the fibers. The fourth order symmetric positive definite tensor that represents the CT-FOD is then analytically decomposed via the application of a new theoretical approach and this decomposition is used to accurately extract all the fibers orientations. Our proposed high order tensor decomposition based approach is minimal and allows recovering the whole crossing fibers without any a priori information on the total number of fibers. Various experiments performed on noisy synthetic data, on phantom diffusion, data and on human brain data validate our approach and clearly demonstrate that it is efficient, robust to noise and performs favorably in terms of angular resolution and accuracy when compared to some classical and state-of-the-art approaches.

A Survey on PARATREE and SUSVD Decomposition Techniques and Their Use in Array Signal Processing

The present manuscript is intended to review few applications of tensor decomposition model in array signal processing. Tensor decomposition models like HOSVD, SVD and PARAFAC are useful in signal processing. In this paper we shall use higher order tensor decomposition in signal processing. Also, a novel orthogonal non-iterative tensor decomposition technique (SUSVD), which is scalable to arbitrary high dimensional tensor, has been applied in MIMO channel estimation. The SUSVD provides a tensor model with hierarchical tree structure between the factors in different dimensions. We shall use a new model known as PARATREE, which is related to PARAFAC tensor models. The PARAFAC and PARATREE both describe a tensor as a sum of rank-1 tensors, but PARATREE has several advantages over PARAFAC, when it is applied as a lower rank approximation technique. PARATREE is orthogonal, fast and reliable to compute, and the order of the decomposition can be adaptively adjusted. The low rank PARATREE approximation has been applied to measure noise suppression for tensor valued MIMO channel sounding measurements.

A New Tensorial Framework for Single-Shell High Angular Resolution Diffusion Imaging

Single-shell high angular resolution diffusion imaging data (HARDI) may be decomposed into a sum of eigenpolynomials of the Laplace-Beltrami operator on the unit sphere. The resulting representation combines the strengths hitherto offered by higher order tensor decomposition in a tensorial framework and spherical harmonic expansion in an analytical framework, but removes some of the conceptual weaknesses of either. In particular it admits analytically closed form expressions for Tikhonov regularization schemes and estimation of an orientation distribution function via the Funk-Radon Transform in tensorial form, which previously required recourse to spherical harmonic decomposition. As such it provides a natural point of departure for a Riemann-Finsler extension of the geometric approach towards tractography and connectivity analysis as has been stipulated in the context of diffusion tensor imaging (DTI), while at the same time retaining the natural coarse-to-fine hierarchy intrinsic to spherical harmonic decomposition.

A Multilinear Singular Value Decomposition

We discuss a multilinear generalization of the singular value decomposition. There is a strong analogy between several properties of the matrix and the higher-order tensor decomposition; uniqueness, link with the matrix eigenvalue decomposition, first-order perturbation effects, etc., are analyzed. We investigate how tensor symmetries affect the decomposition and propose a multilinear generalization of the symmetric eigenvalue decomposition for pair-wise symmetric tensors.