Canonical Polyadic Tensor Research Articles

A canonical polyadic tensor basis for fast Bayesian estimation of multi-subject brain activation patterns.

Task-evoked functional magnetic resonance imaging studies, such as the Human Connectome Project (HCP), are a powerful tool for exploring how brain activity is influenced by cognitive tasks like memory retention, decision-making, and language processing. A fast Bayesian function-on-scalar model is proposed for estimating population-level activation maps linked to the working memory task. The model is based on the canonical polyadic (CP) tensor decomposition of coefficient maps obtained for each subject. This decomposition effectively yields a tensor basis capable of extracting both common features and subject-specific features from the coefficient maps. These subject-specific features, in turn, are modeled as a function of covariates of interest using a Bayesian model that accounts for the correlation of the CP-extracted features. The dimensionality reduction achieved with the tensor basis allows for a fast MCMC estimation of population-level activation maps. This model is applied to one hundred unrelated subjects from the HCP dataset, yielding significant insights into brain signatures associated with working memory.

Machine learning with tree tensor networks, CP rank constraints, and tensor dropout.

Tensor networks developed in the context of condensed matter physics try to approximate order-N tensors with a reduced number of degrees of freedom that is only polynomial in N and arranged as a network of partially contracted smaller tensors. As we have recently demonstrated in the context of quantum many-body physics, computation costs can be further substantially reduced by imposing constraints on the canonical polyadic (CP) rank of the tensors in such networks. Here, we demonstrate how tree tensor networks (TTN) with CP rank constraints and tensor dropout can be used in machine learning. The approach is found to outperform other tensor-network-based methods in Fashion-MNIST image classification. A low-rank TTN classifier with branching ratio b = 4 reaches a test set accuracy of 90.3% with low computation costs. Consisting of mostly linear elements, tensor network classifiers avoid the vanishing gradient problem of deep neural networks. The CP rank constraints have additional advantages: The number of parameters can be decreased and tuned more freely to control overfitting, improve generalization properties, and reduce computation costs. They allow us to employ trees with large branching ratios, substantially improving the representation power.

LINK PREDICTION USING TENSOR DECOMPOSITION

In recent years, tensor decomposition has gained increasing interest in the field of link prediction, which aims to estimate the likelihood of new connections forming between nodes in a network. This study highlights the potential of the Canonical Polyadic tensor decomposition in enhancing link prediction in complex networks. It suggests effective tensor decomposition algorithms that not only take into account the structural characteristics of the network but also its temporal evolution. During the process of tensor decomposition, the initial tensor is decomposed into two-way tensors, also known as factor matrices, representing different modes of the data. These factor matrices capture the underlying patterns or relationships within the network, providing insights into the structure and dynamics of the network. For evaluation, we examine a dataset derived from the WSDM. After preprocessing, the data is represented as a multi-way tensor, with each mode representing different aspects such as users, items, and time. Our primary objective is to make precise predictions about the links between users and items within specific time periods. The experimental results demonstrate that our approach significantly improves prediction accuracy for evolving networks, as measured by the AUC.

Computing vibrational energy levels using a canonical polyadic tensor method with a fixed rank and a contraction tree.

In this paper, we use the previously introduced Canonical Polyadic (CP)-Multiple Shift Block Inverse Iteration (MSBII) eigensolver [S. D. Kallullathil and T. Carrington, J. Chem. Phys. 155, 234105 (2021)] in conjunction with a contraction tree to compute vibrational spectra. The CP-MSBII eigensolver uses the CP format. The memory cost scales linearly with the number of coordinates. A tensor in CP format represents a wavefunction constrained to be a sum of products (SOP). An SOP wavefunction can be made more accurate by increasing the number of terms, the rank. When the required rank is large, the runtime of a calculation in CP format is long, although the memory cost is small. To make the method more efficient, we break the full problem into pieces using a contraction tree. The required rank for each of the sub-problems is small. To demonstrate the effectiveness of the ideas, we computed vibrational energy levels of acetonitrile (12-D) and ethylene oxide (15-D).

Multi-level optimization of the canonical polyadic tensor decomposition at large-scale: Application to the stratification of social networks through deflation

Tensors are multi-dimensional mathematical objects that allow to model complex relationships and to perform decompositions for analytical purpose. They are used in a wide range of data mining applications. In social network analysis, tensor decompositions give interesting insights by taking into consideration multiple characteristics of data. However, the power-law distribution of such data forces the decomposition to reveal only the strong signals that hide information of interest having a lighter intensity. To reveal hidden information, we propose a method to stratify the signal, by gathering clusters of similar intensity in each stratum. It is an iterative process, in which the CANDECOMP/PARAFAC (CP) decomposition is applied and its result is used to deflate the tensor, i.e., by removing from the tensor the clusters found with the decomposition. As the CP decomposition is computationally demanding, it is also necessary to optimize its algorithm, to apply it on large-scale data with a reasonable execution time, even with the several executions needed by the iterative process of the stratification. Therefore, we propose an algorithm that uses both dense and sparse data structures and that leverages coarse and fine grained optimizations in addition to incremental computations in order to achieve large scale CP tensor decomposition. Our implementation outperforms the baseline of large-scale CP decomposition libraries by several orders of magnitude. We validate our stratification method and our optimized algorithm on a Twitter dataset about COVID vaccines.

Practical Leverage-Based Sampling for Low-Rank Tensor Decomposition

The low-rank canonical polyadic tensor decomposition is useful in data analysis and can be computed by solving a sequence of overdetermined least squares subproblems. Motivated by consideration of sparse tensors, we propose sketching each subproblem using leverage scores to select a subset of the rows, with probabilistic guarantees on the solution accuracy. We randomly sample rows proportional to leverage score upper bounds that can be efficiently computed using the special Khatri--Rao subproblem structure inherent in tensor decomposition. Crucially, for a $(d+1)$-way tensor, the number of rows in the sketched system is $O(r^d/\epsilon)$ for a decomposition of rank $r$ and $\epsilon$-accuracy in the least squares solve, independent of both the size and the number of nonzeros in the tensor. Along the way, we provide a practical solution to the generic matrix sketching problem of sampling overabundance for high-leverage-score rows, proposing to include such rows deterministically and combine repeated samples in the sketched system; we conjecture that this can lead to improved theoretical bounds. Numerical results on real-world large-scale tensors show the method is significantly faster than deterministic methods at nearly the same level of accuracy.

Conformal IRS-Empowered MIMO-OFDM: Channel Estimation and Environment Mapping

We consider the channel estimation and environment mapping problems in multiple-input multiple-output orthogonal frequency division multiplexing systems empowered by intelligent reconfigurable surfaces (IRSs). In order to acquire more in-depth environmental information, as well as, to flexibly take into account existing real-life infrastructure, we propose a novel three-dimensional conformal IRS architecture consisting of reflective unit cells distributed on curved surfaces. We model the training signal as a third-order canonical polyadic tensor and construct a tensor factorization problem. Given specific conditions on the allocated temporal-frequency training resources, we develop four channel estimation approaches, i.e., least squares, direct, wideband direct and wideband subspace methods, by leveraging tensor techniques and nonlinear system solvers. By fully exploiting the characteristics of conformal IRSs, we propose two decoupling modes to precisely recover the multipath parameters without ambiguities, which cannot be supported by the traditional IRS planar topologies. We implement scatterer mapping and user positioning tasks based on precise parameter estimates. Simulation results indicate that the proposed conformal IRS structure and estimation schemes can recover the channel state information with remarkable accuracy, thereby offering a centimeter-level resolution of environment mapping.

Channel Estimation and User Localization for IRS-Assisted MIMO-OFDM Systems

We consider the channel estimation problem and the channel-based wireless applications in multiple-input multiple-output orthogonal frequency division multiplexing systems assisted by intelligent reconfigurable surfaces (IRSs). To obtain the necessary channel parameters, i.e., angles, delays and gains, for environment mapping and user localization, we propose a novel twin-IRS structure consisting of two IRS planes with a relative spatial rotation. We model the training signal from the user equipment to the base station via IRSs as a third-order canonical polyadic tensor with a maximal tensor rank equal to the number of IRS unit cells. We present four designs of IRS training coefficients, i.e., random, structured, grouping and sparse patterns, and analyze the corresponding uniqueness conditions of channel estimation. We extract the cascaded channel parameters by leveraging array signal processing and atomic norm denoising techniques. Based on the characteristics of the twin-IRS structures, we formulate a nonlinear equation system to exactly recover the multipath parameters by two efficient decoupling modes. We realize environment mapping and user localization based on the estimated channel parameters. Simulation results indicate that the proposed twin-IRS structure and estimation schemes can recover the channel state information with remarkable accuracy, thereby offering a centimeter-level resolution of user positioning.

Tensor-based approach for blind separation of interleave-NOMA 5G system

Nowadays, wireless broadband communication has experienced tremendous growth. The requirement for various new and advanced multiple access techniques is constantly increasing. The interleave division multiple access (IDMA) technique has attracted the attention of the scientific community as an opportune code domain non-orthogonal multiple access (NOMA) for future communication standards. In this system, the detection of the signals received from different users is done by specific interleavers.In this paper, and regarding the transmission-rate loss caused by the use of spreading codes and interleavers in the IDMA systems, we are going to place ourselves in a blind separation context for IDMA signals. For this purpose, a study of a purely deterministic approach, which is the tensor decomposition, as well as a theoretical study of the Canonical Polyadic (CP) model’s existence and uniqueness issues, is also carried out. Then, a tensor modeling of the IDMA emitter and the proposal of a new form of the interleavers will be undertaken. The separation and equalization of received signals using the CP tensor decomposition compelled us to examine the methods utilized to carry out this decomposition, as well as suggest two iterative blind receivers that take use of the algebraic features of the suggested model. On the other hand, the application of an exact performance criterion will be performed, taken into account the scale and permutation indeterminacy constraints.

Dynamic brain effective connectivity analysis based on low-rank canonical polyadic decomposition: application to epilepsy.

In this paper, a new method to track brain effective connectivity networks in the context of epilepsy is proposed. It relies on the combination of partial directed coherence with a constrained low-rank canonical polyadic tensor decomposition. With such combination being established, the most dominating directed graph structures underlying each time window of intracerebral electroencephalographic signals are optimally inferred. Obtained time and frequency signatures of inferred brain networks allow respectively to track the time evolution of these networks and to define frequency bands on which they are operating. Besides, the proposed method allows also to track brain connectivity networks through several epileptic seizures of the same patient. Understanding the most dominating directed graph structures over epileptic seizures and investigating their behavior over time and frequency plans are henceforth possible. Since only few but the the most important directed connections in the graph structure are of interest and also for a meaningful interpretation of obtained signatures to be guaranteed, the low-rank canonical polyadic tensor decomposition is prompted respectively by the sparsity and the non-negativity constraints on the tensor loading matrices. The main objective of this contribution is to propose a new way of tracking brain networks in the context of epileptic iEEG data by identifying the most dominant effective connectivity patterns underlying the observed iEEG signals at each time window. The performance of the proposed method is firstly evaluated on simulated data imitating brain activities and secondly on real intracerebral electroencephalographic signals obtained from an epileptic patient. The partial directed coherence-based tensor has been decomposed into space, time, and frequency signatures in accordance with the expected ground truth for each consecutive sequence of the simulated data. The method is also in accordance with the clinical expertise for iEEG epileptic signals, where the signatures were investigated through a simultaneous multi-seizure analysis.

Robust brain network identification from multi-subject asynchronous fMRI data

We describe a novel method for robust identification of common brain networks and their corresponding temporal dynamics across subjects from asynchronous functional MRI (fMRI) using tensor decomposition. We first temporally align asynchronous fMRI data using the orthogonal BrainSync transform, allowing us to study common brain networks across sessions and subjects. We then map the synchronized fMRI data into a 3D tensor (vertices × time × subject/session). Finally, we apply Nesterov-accelerated adaptive moment estimation (Nadam) within a scalable and robust sequential Canonical Polyadic (CP) decomposition framework to identify a low rank tensor approximation to the data. As a result of CP tensor decomposition, we successfully identified twelve known brain networks with their corresponding temporal dynamics from 40 subjects using the Human Connectome Project's language task fMRI data without any prior information regarding the specific task designs. Seven of these networks show distinct subjects’ responses to the language task with differing temporal dynamics; two show sub-components of the default mode network that exhibit deactivation during the tasks; the remaining three components reflect non-task-related activities. We compare results to those found using group independent component analysis (ICA) and canonical ICA. Bootstrap analysis demonstrates increased robustness of networks found using the CP tensor approach relative to ICA-based methods.

Using the proximal gradient and the accelerated proximal gradient as a canonical polyadic tensor decomposition algorithms in difficult situations

Canonical Polyadic (CP) tensor decomposition is useful in many real-world applications due to its uniqueness, and the ease of interpretation of its factor matrices. This work addresses the problem of calculating the CP decomposition of tensors in difficult cases where the factor matrices in one or all modes are almost collinear – i.e. bottleneck or swamp problems arise. This is done by introducing a constraint on the coherences of the factor matrices that ensures the existence of a best low-rank approximation, which makes it possible to estimate these highly correlated factors. Two new algorithms optimizing the CP decomposition based on proximal methods are proposed. Simulation results are provided and demonstrate the good behaviour of these algorithms, as well as a better compromise between accuracy and convergence speed than other algorithms in the literature.

Research Spotlights

Tensors, or multiway arrays, are often used for storage of high-dimensional data. In order to have compression, completion, or interpretation of such data, the data tensor is factored according to a proposed model consistent with some belief about the data. The first Research Spotlights article, “Generalized Canonical Polyadic Tensor Decomposition," by David Hong, Tamara G. Kolda, and Jed A. Duersch, treats the problem of finding a low-rank tensor decomposition that minimizes a cost functional defined from a componentwise loss function. The difference between the proposed generalized canonical polyadic (GCP) decomposition and the well-known CP decomposition is the flexibility in the choice of loss function, which can be tailored to reflect the appropriate statistical likelihood of a model given the data: low rankness of the tensor is enforced as a constraint. The main result of the paper is in Theorem 3, which addresses the computation of the gradients of a cost function defined in terms of any loss function that is continuously differentiable with respect to its second argument. Their result reveals an elegant expression for the gradients that enables the use of existing kernels for finding CP decompositions to be employed in their algorithm to compute the GCP. The authors use extensive numerical testing, highlighting the flexibility of their algorithm and its ability to efficiently handle missing data while simultaneously illustrating the potential gain to the data analyst in choosing loss functions more consistent with the data. A variety of applications are considered, including analysis of a chat network and explaining factors for mouse neural activity. A “sticky” diffusion process, as defined in the second Research Spotlights article, “Sticky Brownian Motion and Its Numerical Solution," refers to solutions to stochastic differential equations (SDEs) that can spend finite time on a lower-dimensional boundary. Materials like concrete, paint, and toothpaste have something in common: in the limit, the dynamics of the collection of mesoscale particles of which each is comprised approaches a sticky diffusion process. Sticky Brownian motion (SBM), the simplest example of a sticky diffusion process, is the primary focus of the present work. The goals of authors Nawaf Bou-Rabee and Miranda C. Holmes-Cerfon are to call the attention of the greater applied mathematics community to the topic of sticky diffusions as well as to offer a numerical method to allow for simulation of a sticky diffusion with a relatively large time step. These goals are evident in the presentation of material. The mathematical setup of an SBM is given in section 2. Beginning with the Brownian dynamics equations and assumptions on the potential energy function, the dynamics of the probability density are discussed, including asymptotic behavior in the “sticky limit." The authors discuss how to find the generator of the SBM and give some examples. The generator is the basis for the numerical method they develop in section 3, which the authors claim is orders of magnitude faster than alternative methods to simulate a sticky Brownian motion. The authors complement their presentation with a mathematical literature review of some additional methods for characterizing SBM. The final section serves as a well-considered road map for the reader interested in extending this work to a higher-dimensional setting.

A canonical polyadic deep convolutional computation model for big data feature learning in Internet of Things

Abstract In recent years, the Internet of Things is more widely deployed with increasing amounts of data gathered. These data are of high volume, velocity, veracity and variety, posing a vast challenge on the data analysis, especially with respect to variety and velocity. To address this challenge, a canonical polyadic deep convolutional computation model is introduced to efficiently and effectively capture the hierarchical representation of the big data by employing the canonical polyadic decomposition to factorize the deep convolutional computation. In particular, to speed up the learning of local topologies hidden in the big data, a canonical polyadic convolutional kernel is devised by compacting the tensor convolutional kernel into the linear combination of the principle rank-1 tensors. Furthermore, the canonical polyadic tensor fully-connected weight is used to efficiently map the correlation in the fully-connected layer. After that, the canonical polyadic high-order back-propagation is devised to train the canonical polyadic deep convolutional computation model. At last, detailed experiments are carried out on two well-known datasets. And results illustrate that the introduced model achieves higher performance than a competing model.

Software for Sparse Tensor Decomposition on Emerging Computing Architectures

In this paper, we develop software for decomposing sparse tensors that is portable to and performant on a variety of multicore, manycore, and GPU computing architectures. The result is a single code whose performance matches optimized architecture-specific implementations. The key to a portable approach is to determine multiple levels of parallelism that can be mapped in different ways to different architectures, and we explain how to do this for the matricized tensor times Khatri--Rao product (MTTKRP), which is the key kernel in canonical polyadic tensor decomposition. Our implementation leverages the Kokkos framework, which enables a single code to achieve high performance across multiple architectures that differ in how they approach fine-grained parallelism. We also introduce a new construct for portable thread-local arrays, which we call compile-time polymorphic arrays. Not only are the specifics of our approaches and implementation interesting for tuning tensor computations, but they also provide a roadmap for developing other portable high-performance codes. As a last step in optimizing performance, we modify the MTTKRP algorithm itself to do a permuted traversal of tensor nonzeros to reduce atomic-write contention. We test the performance of our implementation on 16- and 68-core Intel CPUs and the K80 and P100 NVIDIA GPUs, showing that we are competitive with state-of-the-art architecture-specific codes while having the advantage of being able to run on a variety of architectures.

Few‐shot Learning of Homogeneous Human Locomotion Styles

AbstractUsing neural networks for learning motion controllers from motion capture data is becoming popular due to the natural and smooth motions they can produce, the wide range of movements they can learn and their compactness once they are trained. Despite these advantages, these systems require large amounts of motion capture data for each new character or style of motion to be generated, and systems have to undergo lengthy retraining, and often reengineering, to get acceptable results. This can make the use of these systems impractical for animators and designers and solving this issue is an open and rather unexplored problem in computer graphics. In this paper we propose a transfer learning approach for adapting a learned neural network to characters that move in different styles from those on which the original neural network is trained. Given a pretrained character controller in the form of a Phase‐Functioned Neural Network for locomotion, our system can quickly adapt the locomotion to novel styles using only a short motion clip as an example. We introduce a canonical polyadic tensor decomposition to reduce the amount of parameters required for learning from each new style, which both reduces the memory burden at runtime and facilitates learning from smaller quantities of data. We show that our system is suitable for learning stylized motions with few clips of motion data and synthesizing smooth motions in real‐time.

An intertwined method for making low-rank, sum-of-product basis functions that makes it possible to compute vibrational spectra of molecules with more than 10 atoms.

We propose a method for solving the vibrational Schrödinger equation with which one can compute spectra for molecules with more than ten atoms. It uses sum-of-product (SOP) basis functions stored in a canonical polyadic tensor format and generated by evaluating matrix-vector products. By doing a sequence of partial optimizations, in each of which the factors in a SOP basis function for a single coordinate are optimized, the rank of the basis functions is reduced as matrix-vector products are computed. This is better than using an alternating least squares method to reduce the rank, as is done in the reduced-rank block power method. Partial optimization is better because it speeds up the calculation by about an order of magnitude and allows one to significantly reduce the memory cost. We demonstrate the effectiveness of the new method by computing vibrational spectra of two molecules, ethylene oxide (C2H4O) and cyclopentadiene (C5H6), with 7 and 11 atoms, respectively.

Partitioned Alternating Least Squares Technique for Canonical Polyadic Tensor Decomposition

Canonical polyadic decomposition (CPD), also known as parallel factor analysis, is a representation of a given tensor as a sum of rank-one components. Traditional method for accomplishing CPD is the alternating least squares (ALS) algorithm. Convergence of ALS is known to be slow, especially when some factor matrices of the tensor contain nearly collinear columns. We propose a novel variant of this technique, in which the factor matrices are partitioned into blocks, and each iteration jointly updates blocks of different factor matrices. Each partial optimization is quadratic and can be done in closed form. The algorithm alternates between different random partitionings of the matrices. As a result, a faster convergence is achieved. Another improvement can be obtained when the method is combined with the enhanced line search of Rajih et al. Complexity per iteration is between those of the ALS and the Levenberg–Marquardt (damped Gauss–Newton) method. It is important, however, that the idea of alternating quadratic optimization with partitioned factor matrices is general and can be applied to other variants of the tensor decomposition problems, e.g., when non-Gaussian additive noise is considered.

Newton-based optimization for Kullback–Leibler nonnegative tensor factorizations

Tensor factorizations with nonnegativity constraints have found application in analysing data from cyber traffic, social networks, and other areas. We consider application data best described as being generated by a Poisson process (e.g. count data), which leads to sparse tensors that can be modelled by sparse factor matrices. In this paper, we investigate efficient techniques for computing an appropriate canonical polyadic tensor factorization based on the Kullback–Leibler divergence function. We propose novel subproblem solvers within the standard alternating block variable approach. Our new methods exploit structure and reformulate the optimization problem as small independent subproblems. We employ bound-constrained Newton and quasi-Newton methods. We compare our algorithms against other codes, demonstrating superior speed for high accuracy results and the ability to quickly find sparse solutions.

Probabilistic inference with noisy-threshold models based on a CP tensor decomposition

The specification of conditional probability tables (CPTs) is a difficult task in the construction of probabilistic graphical models. Several types of canonical models have been proposed to ease that difficulty. Noisy-threshold models generalize the two most popular canonical models: the noisy-or and the noisy-and. When using the standard inference techniques the inference complexity is exponential with respect to the number of parents of a variable. More efficient inference techniques can be employed for CPTs that take a special form. CPTs can be viewed as tensors. Tensors can be decomposed into linear combinations of rank-one tensors, where a rank-one tensor is an outer product of vectors. Such decomposition is referred to as Canonical Polyadic (CP) or CANDECOMP-PARAFAC (CP) decomposition. The tensor decomposition offers a compact representation of CPTs which can be efficiently utilized in probabilistic inference. In this paper we propose a CP decomposition of tensors corresponding to CPTs of threshold functions, exactly ℓ-out-of-k functions, and their noisy counterparts. We prove results about the symmetric rank of these tensors in the real and complex domains. The proofs are constructive and provide methods for CP decomposition of these tensors. An analytical and experimental comparison with the parent-divorcing method (which also has a polynomial complexity) shows superiority of the CP decomposition-based method. The experiments were performed on subnetworks of the well-known QMRT-DT network generalized by replacing noisy-or by noisy-threshold models.