Matricized Tensor Times Khatri-Rao Product Research Articles

True Load Balancing for Matricized Tensor Times Khatri-Rao Product

MTTKRP is the bottleneck operation in algorithms used to compute the CP tensor decomposition. For sparse tensors, utilizing the compressed sparse fibers (CSF) storage format and the CSF-oriented MTTKRP algorithms is important for both memory and computational efficiency on distributed-memory architectures. Existing intelligent tensor partitioning models assume the computational cost of MTTKRP to be proportional to the total number of nonzeros in the tensor. However, this is not the case for the CSF-oriented MTTKRP on distributed-memory architectures. We outline two deficiencies of nonzero-based intelligent partitioning models when CSF-oriented MTTKRP operations are performed locally: failure to encode processors' computational loads and increase in total computation due to fiber fragmentation. We focus on existing fine-grain hypergraph model and propose a novel vertex weighting scheme that enables this model encode correct computational loads of processors. We also propose to augment the fine-grain model by fiber nets for reducing the increase in total computational load via minimizing fiber fragmentation. In this way, the proposed model encodes minimizing the load of the bottleneck processor. Parallel experiments with real-world sparse tensors on up to 1024 processors prove the validity of the outlined deficiencies and demonstrate the merit of our proposed improvements in terms of parallel runtimes.

Towards efficient canonical polyadic decomposition on sunway many-core processor

Canonical Polyadic Decomposition (CPD) is one of the most popular tensor decomposition methods and plays an important role in big data analysis. For sparse tensor, the major computation procedure in CPD, which is known as matricized tensor times Khatri-Rao product (MTTKRP), exhibits discontinuous memory access and turns to be the performance bottleneck from achieving high performance on emerging processor architectures. In this paper, we propose swCPD, an efficient CPD implementation on the many-core Sunway processor. The swCPD accelerates the optimization algorithms dominating the performance of MTTKRP, including Alternating Least Squares (ALS), Gradient Descent (GD) and Randomized Block Sampling (RBS), as well as the latest fast Levenberg–Marquardt (fLM++) and Generalized Canonical Polyadic Decomposition with Stochastic Gradient Descent (GCP-SGD). The main idea adopted in swCPD is a hierarchical partitioning mechanism. From the computation perspective, the 64 Computation Processing Elements (CPEs) in a Sunway processor are divided into eight groups, with each group containing seven workers and one controller. From the data perspective, we partition the sparse tensor into different granularities, which are blocks, bands and tiles. Moreover, we develop a communication mechanism through register communication for cooperation between CPEs. We evaluate the implementation of swCPD with both synthesized and real-world datasets. The experiment results show that each optimized algorithm in swCPD achieves better performance than corresponding algorithms adopted in cutting-edge CPD implementations.

An online and generalized non-negativity constrained model for large-scale sparse tensor estimation on multi-GPU

Non-negative Tensor Factorization (NTF) models are effective and efficient in extracting useful knowledge from various types of probabilistic distribution with multi-way information. Current NTF models are mostly designed for problems in computer vision which involve the whole Matricized Tensor TimesKhatri−RaoProduct (MTTKRP). Meanwhile, a Sparse NTF (SNTF) proposed to solve the problem of sparse Tensor Factorization (TF) can result in large-scale intermediate data. A Single-thread-based SNTF (SSNTF) model is proposed to solve the problem of non-linear computing and memory overhead caused by large-scale intermediate data. However, the SSNTF is not a generalized model. Furthermore, the above methods cannot describe the stream-like data from industrial applications in mainstream processors, e.g, Graphics Processing Unit (GPU) and multi-GPU in an online way. To address these two issues, a Generalized SSNTF (GSSNTF) is proposed, which extends the works of SSNTF to the Euclidean distance, KullbackLeibler (KL)-divergence, and ItakuraSaito (IS)-divergence. The GSSNTF only involves the feature elements instead of the entire factor matrices during its update process, which can avoid the formation of large-scale intermediate matrices with convergence and accuracy promises. Furthermore, GSSNTF can merge the new data into the state-of-the-art built tree dataset for sparse tensor, and then online learning has the promise of the correct data format. At last, a model of Compute Unified Device Architecture (CUDA) parallelizing GSSNTF (CUGSSNTF) is proposed on GPU and Multi-GPU (MCUGSSNTF). Thus, CUGSSNTF has linear computing complexity and space requirement, and linear communication overhead on multi-GPU. CUGSSNTF and MCUGSSNTF are implemented on 8 P100 GPUs in this work, and the experimental results from real-world industrial data sets indicate the linear scalability and 40X speedup performances of CUGSSNTF than the state-of-the-art parallelized approachs.

Software for Sparse Tensor Decomposition on Emerging Computing Architectures

In this paper, we develop software for decomposing sparse tensors that is portable to and performant on a variety of multicore, manycore, and GPU computing architectures. The result is a single code whose performance matches optimized architecture-specific implementations. The key to a portable approach is to determine multiple levels of parallelism that can be mapped in different ways to different architectures, and we explain how to do this for the matricized tensor times Khatri--Rao product (MTTKRP), which is the key kernel in canonical polyadic tensor decomposition. Our implementation leverages the Kokkos framework, which enables a single code to achieve high performance across multiple architectures that differ in how they approach fine-grained parallelism. We also introduce a new construct for portable thread-local arrays, which we call compile-time polymorphic arrays. Not only are the specifics of our approaches and implementation interesting for tuning tensor computations, but they also provide a roadmap for developing other portable high-performance codes. As a last step in optimizing performance, we modify the MTTKRP algorithm itself to do a permuted traversal of tensor nonzeros to reduce atomic-write contention. We test the performance of our implementation on 16- and 68-core Intel CPUs and the K80 and P100 NVIDIA GPUs, showing that we are competitive with state-of-the-art architecture-specific codes while having the advantage of being able to run on a variety of architectures.

Shared-memory parallelization of MTTKRP for dense tensors

The matricized-tensor times Khatri-Rao product (MTTKRP) is the computational bottleneck for algorithms computing CP decompositions of tensors. In this work, we develop shared-memory parallel algorithms for MTTKRP involving dense tensors. The algorithms cast nearly all of the computation as matrix operations in order to use optimized BLAS subroutines, and they avoid reordering tensor entries in memory. We use our parallel implementation to compute a CP decomposition of a neuroimaging data set and achieve a speedup of up to 7.4X over existing parallel software.