Tensor Decomposition for Signal Processing and Machine Learning

Several combined signal processing applications such as the joint processing of EEG and MEG data can benefit from coupled tensor decompositions, for instance, the coupled CP (Canonical Polyadic) decomposition. The coupled CP decomposition jointly decomposes tensors that have at least one factor matrix in common. The SECSI (Semi-Algebraic framework for approximate CP decomposition via SImultaneaous matrix diagonalization) framework is an efficient tool for the calculation of the CP decomposition based on matrix diagonalizations. It provides a semi-algebraic solution for the CP decomposition even in ill-posed scenarios, e.g., if the columns of a factor matrix are highly correlated. Moreover, the SECSI framework provides an adjustable complexity-accuracy trade-off. In this paper, we present an extension of the SECSI framework to the efficient calculation of coupled CP decompositions and show its advantages compared to the traditional solution via alternating least squares (ALS) and other state of the art algorithms.

Tensor decompositions play an important role in a variety of applications, such as signal processing and machine learning. In practice, the tensor can be incomplete or very large, making it difficult to analyze and process using conventional tensor techniques. In this paper we focus on the basic canonical polyadic decomposition (CPD). We propose an algebraic framework for finding the CPD of tensors that have missing fibers. This includes extensions of multilinear algebraic as well as generic uniqueness conditions originally developed for the CPD of fully observed tensors. Computationally, we reduce the CPD of a tensor with missing fibers to relatively simple matrix completion problems via a matrix eigenvalue decomposition (EVD). Under the given conditions, the EVD-based algorithm is guaranteed to return the exact CPD. The derivation establishes connections with so-called coupled CPDs, an emerging concept that has proven to be of great interest in a range of array processing and signal processing applications. It will become clear that relatively few fibers are needed in order to compute the CPD. This makes fiber sampling interesting for large scale tensor decompositions. Numerical experiments show that the algebraic framework may significantly speed up more common optimization-based computation schemes for the estimation of the CPD of incomplete noisy data tensors.

Canonical polyadic (CP) decomposition of a tensor is a basic operation in a lot of applications such as data mining and video foreground/background separation. However, existing algorithms for CP decomposition require users to provide a rank of the target tensor data as part of the input and finding the rank of a tensor is an NP-hard problem. Currently, to perform CP decomposition, users are required to make an informed guess of a proper tensor rank based on the data at hand, and the result may still be suboptimal. In this paper, we propose to conduct CP decomposition and tensor rank approximation together, so that users do not have to provide the proper rank beforehand, and the decomposition algorithm will find the proper rank and return a high-quality result. We formulate an optimization problem with an objective function consisting of a least-squares Tikhonov regularization and a sparse l 1 -regularization term. We also test its effectiveness over real applications with moving object videos.

The goal of machine learning is to program computers to use example data or past experience to solve a given problem. Many successful applications of machine learning exist already, including systems that analyze past sales data to predict customer behavior, optimize robot behavior so that a task can be completed using minimum resources, and extract knowledge from bioinformatics data. Introduction to Machine Learning is a comprehensive textbook on the subject, covering a broad array of topics not usually included in introductory machine learning texts. In order to present a unified treatment of machine learning problems and solutions, it discusses many methods from different fields, including statistics, pattern recognition, neural networks, artificial intelligence, signal processing, control, and data mining. All learning algorithms are explained so that the student can easily move from the equations in the book to a computer program. The text covers such topics as supervised learning, Bayesian decision theory, parametric methods, multivariate methods, multilayer perceptrons, local models, hidden Markov models, assessing and comparing classification algorithms, and reinforcement learning. New to the second edition are chapters on kernel machines, graphical models, and Bayesian estimation; expanded coverage of statistical tests in a chapter on design and analysis of machine learning experiments; case studies available on the Web (with downloadable results for instructors); and many additional exercises. All chapters have been revised and updated. Introduction to Machine Learning can be used by advanced undergraduates and graduate students who have completed courses in computer programming, probability, calculus, and linear algebra. It will also be of interest to engineers in the field who are concerned with the application of machine learning methods. Adaptive Computation and Machine Learning series

The Canonical Polyadic (CP) decomposition of R-way arrays is a powerful tool in multi-dimensional signal processing. There exists many methods to compute the CP decomposition. In particular, the Semi-Algebraic framework for the approximate Canonical Polyadic (CP) decomposition via SImultaneaous matrix diagonalization (SECSI) is an efficient and flexible framework for the computation of the CP decomposition. In this work, we perform a first-order performance analysis of the SECSI framework for the computation of the approximate CP decomposition of a noise corrupted low-rank 3-way tensor. We provide closed-form expressions of the relative Mean Square Factor Error (rMSFE) for each of the estimated factor matrices. The derived expressions are formulated in terms of the second-order moments of the noise, such that apart from a zero mean, no assumptions on the noise statistics are required. The numerical results depict the excellent match between the closed-form expressions and the empirical results.

A new algorithm of the canonical polyadic decomposition (CPD) presented here. It features lower computational complexity and memory usage than the available state of the art implementations. We begin with some examples of CPD applications to real world problems. A short summary of the main contributions in this work follows. In chapter 1 we review classical tensor algebra and geometry, with focus on the CPD. Chapter 2 focuses on tensor compression, which is considered (in this work) to be one of the most important parts of the CPD algorithm. In chapter 3 we talk about the Gauss-Newton method, which is a nonlinear least squares method used to minimize nonlinear functions. Chapter 4 is the longest one of this thesis. In this chapter we introduce the main character of this thesis: Tensor Fox. Basically it is a tensor package which includes a CPD solver. After introducing Tensor Fox we will conduct lots of computational experiments comparing this solver with several others. At the end of this chapter we introduce the Tensor Train decomposition and show how to use it to compute higher order CPDs. We also discuss some important details such as regularization, preconditioning, conditioning, parallelism, etc. In chapter 5 we consider the intersection between tensor decompositions and machine learning. A novel model is introduced, which works as a tensor version of neural networks. Finally, in chapter 6 we reach the final conclusions and introduce our expectations for future developments.

In signal processing several problems have been formulated as Simultaneous Generalized Schur Decomposition (SGSD) problems. Applications are found in blind source separation and multidimensional harmonic retrieval. Furthermore, SGSD methods for computing a third-order Canonical Polyadic (CP) decomposition have been proposed. The original SGSD method requires that all three matrix factors of the CP decomposition have full column rank. We first propose a new version of the SGSD method for computing a third-order CP decomposition. The proposed method mainly differs from the existing method in the way the triangular matrices are computed. Second, we propose an alternative SGSD method which only requires that two of the matrix factors of the CP decomposition have full column rank.

Contributors

Bilinear factorizations subject to monomial equality constraints via tensor decompositions

The Canonical Polyadic Decomposition (CPD) is one of the most basic tensor models used in signal processing and machine learning. Despite its wide applicability, identifiability conditions and algorithms for CPD in cases where the tensor is incomplete are lagging behind its practical use. We first present a tensor-based framework for bilinear factorizations subject to monomial constraints, called monomial factorizations. Next, we explain that the CPD of a tensor that has missing fibers can be interpreted as a monomial factorization problem. Finally, using the monomial factorization interpretation, we show that CPD recovery conditions can be obtained that only rely on the observed fibers of the tensor.

Canonical Polyadic Decomposition (CPD) is a powerful technique for uncovering multilinear relationships in tensors. Current research in scalable CPD has focused on designing efficient decomposition algorithms for large sparse tensors that arise in machine learning and data mining applications. This work addresses the complementary need for efficient decomposition algorithms for large dense tensors that arise in signal processing applications. Such tensors are often highly skewed, with one mode (e.g., time) orders of magnitude larger than the others. We present an algorithm appropriate for MapReduce settings that uses both regularization and sketching to efficiently operate on such tensors. We have open-sourced an Apache Spark implementation of the algorithm and evaluate it on synthetic and real datasets to characterize the trade-offs in runtime and accuracy when using different types and combinations of regularization and sketching. We observe that a combination of random entry sketching plus Tikhonov regularization works best independent of the type or level of noise in the tensor. Similarly, we find that random entry sketching plus proximal regularization works best for ill-conditioned tensors. Further experiments demonstrate that the runtime scales sublinearly with the tensor size and highly sublinearly with the tensor rank. The use of regularization and sketching results in runtimes that are factors of 42-112× faster than those of the previous state-of-the-art MapReduce CPD implementation for large dense, skewed tensors, while having a negligible impact on the accuracy of the decompositions.

This paper addresses symmetric tensor canonical polyadic decomposition (CPD) problem from a probabilistic inference perspective. Symmetric tensor CPD aims at decomposing a symmetric tensor into a succinct sum of rank-1 components and has found wide-spread applications in signal processing and machine learning. By exploiting its inherent symmetric structure, a variety of algorithms that leverage advances in algebras and optimizations have been proposed. However, the implementation of these algorithms needs the exact value of tensor rank, which however is unknown in practice and difficult to estimate. Fortunately, recent developments in probabilistic tensor CPD have successfully achieved automatic rank determination by recasting the rank estimation problem into a hyper-parameter inference problem. Nevertheless, the conjugacy property in exponential distribution family, which is essential to permit closed-form derivations of inference algorithms, does not hold for the probabilistic symmetric tensor CPD model. This calls for a novel design of inference algorithm that can achieve accurate and fast CPD with automatic tensor rank determination, and this paper provides such a solution. Numerical results are presented to validate the excellent performance of the proposed algorithm.

The uniqueness properties of the canonical polyadic decomposition (CPD) of higher-order tensors make it an attractive tool for signal separation. However, CPD uniqueness is not yet fully understood. In this paper, we first present a new uniqueness condition for a polyadic decomposition (PD) where one of the factor matrices is assumed to be known. We also show that this result can be used to obtain a new overall uniqueness condition for the CPD. In signal processing the CPD factor matrices are often constrained. Building on the preceding results, we provide a new uniqueness condition for a CPD with a columnwise orthonormal factor matrix, representing uncorrelated signals. We also obtain a new uniqueness condition for a CPD with a partial Hermitian symmetry, useful for tensors in which covariance matrices are stacked, which are common in statistical signal processing. We explain that such constraints can lead to more relaxed uniqueness conditions. Finally, we provide an inexpensive algorithm for computing a PD with a known factor matrix that is also useful for the computation of the full CPD.

PurposeThe purpose of this study is to explore to which extent data mining research would be associated with the library and information science (LIS) discipline. This study aims to identify data mining related subject terms and topics in representative LIS scholarly publications.Design/methodology/approachA large set of bibliographic records over 38,000 was collected from a scholarly database representing the fields of LIS and the data mining, respectively. A multitude of text mining techniques were applied to investigate prevailing subject terms and research topics, such as influential term analysis and Dirichlet multinomial regression topic modeling.FindingsThe findings of this study revealed the relationship between the LIS and data mining research domains. Various data mining method terms were observed in recent LIS publications, such as machine learning, artificial intelligence and neural networks. The topic modeling result identified prevailing data mining related research topics in LIS, such as machine learning, deep learning, big data and among others. In addition, this study investigated the trends of popular topics in LIS over time in the recent decade.Originality/valueThis investigation is one of a few studies that empirically investigated the relationships between the LIS and data mining research domains. Multiple text mining techniques were employed to delineate to which extent the two research domains would be associated with each other based on both at the term-level and topic-level analysis. Methodologically, the study identified influential terms in each domain using multiple feature selection indices. In addition, Dirichlet multinomial regression was applied to explore LIS topics in relation to data mining.

The decomposition of tensors into simple rank-1 terms is key in a variety of applications in signal processing, data analysis and machine learning. While this canonical polyadic decomposition (CPD) is unique under mild conditions, including prior knowledge such as nonnegativity can facilitate interpretation of the components. Inspired by the effectiveness and efficiency of Gauss-Newton (GN) for unconstrained CPD, we derive a proximal, semismooth GN type algorithm for nonnegative tensor factorization. If the algorithm converges to the global optimum, we show that $Q$-quadratic convergence can be obtained in the exact case. Global convergence is achieved via backtracking on the forward-backward envelope function. The $Q$-quadratic convergence is verified experimentally, and we illustrate that using the GN step significantly reduces number of (expensive) gradient computations compared to proximal gradient descent.