Joint Diagonalization Problem Research Articles

Leveraging Joint-Diagonalization in Transform-Learning NMF

Non-negative matrix factorization with transform learning (TL-NMF) is a recent idea that aims at learning data representations suited to NMF. In this work, we relate TL-NMF to the classical matrix joint-diagonalization (JD) problem. We show that, when the number of data realizations is sufficiently large, TL-NMF can be replaced by a two-step approach -- termed as JD+NMF -- that estimates the transform through JD, prior to NMF computation. In contrast, we found that when the number of data realizations is limited, not only is JD+NMF no longer equivalent to TL-NMF, but the inherent low-rank constraint of TL-NMF turns out to be an essential ingredient to learn meaningful transforms for NMF.

Generalized Givens Rotations Applied to Complex Joint Eigenvalue Decomposition

This paper shows the different ways of using generalized Givens rotations in complex joint eigenvaluedecomposition (JEVD) problem. It presents the different schemes of generalized Givens rotation, justifies the introducedapproximations and focuses on the process of extending an algorithm developed for real JEVD to the complex JEVD.Several Joint Diagonalization problem use generalized Givens rotations to achieve the solution, many algorithmsdeveloped in the real case exist in the literature and are not generalized to the complex case. Hence, we show herein asimple and not trivial way to get the complex case from the real one. Simulation results are provided to highlight theeffectiveness and behaviour of the proposed techniques for different scenarios.

Blind Direction-of-Arrival Estimation in Acoustic Vector-Sensor Arrays via Tensor Decomposition and Kullback-Leibler Divergence Covariance Fitting

A blind Direction-of-Arrivals (DOAs) estimate of narrowband signals for Acoustic Vector-Sensor (AVS) arrays is proposed. Building upon the special structure of the signal measured by an AVS, we show that the covariance matrix of all the received signals from the array admits a natural low-rank 4-way tensor representation. Thus, rather than estimating the DOAs directly from the raw data, our estimate arises from the unique parametric Canonical Polyadic Decomposition (CPD) of the observations' Second-Order Statistics (SOSs) tensor. By exploiting results from fundamental statistics and the recently re-emerging tensor theory, we derive a consistent blind CPD-based DOAs estimate without prior assumptions on the array configuration. We show that this estimate is a solution to an equivalent approximate joint diagonalization problem, and propose an ad-hoc iterative solution. Additionally, we derive the Cram\'er-Rao lower bound for Gaussian signals, and use it to derive the iterative Fisher scoring algorithm for the computation of the Maximum Likelihood Estimate (MLE) in this particular signal model. We then show that the MLE for the Gaussian model can in fact be used to obtain improved DOAs estimates for non-Gaussian signals as well (under mild conditions), which are optimal under the Kullback-Leibler divergence covariance fitting criterion, harnessing additional information encapsulated in the SOSs. Our analytical results are corroborated by simulation experiments in various scenarios, which also demonstrate the considerable improved accuracy w.r.t. a competing state-of-the-art blind estimate for AVS arrays, reducing the resulting root mean squared error by up to more than an order of magnitude.

A two-step algorithm for joint EigenValue decomposition - Application to canonical polyadic decomposition of fluorescence spectra

In this paper, we propose a new Joint EigenValue Decomposition (JEVD) algorithm. JEVD problem belongs to the family of joint diagonalization problems. Hence, JEVD algorithms aim at estimating the common basis of eigenvectors of a matrix set. This problem occurs in many signal processing applications. It has notably allowed to develop efficient algorithms for the Canonical Polyadic Decomposition (CPD) of multiway arrays. The proposed JEVD algorithm is based on an original two-step approach. The first step consists in transforming the considered matrix set into a set of positive definite matrices. In this purpose, we introduce an ad hoc joint symmetrization algorithm. This first step allows us to transform the JEVD problem into a simpler orthogonal joint diagonalization problem. The second step is then performed using an efficient orthogonal joint diagonalization algorithm of the literature. Eventually, the performance of the proposed approach is deeply investigated in the CPD context of multidimensional fluorescence data. More particularly, we consider difficult scenarios such as the cases of an overestimated rank and highly correlated factors.

Riemannian conjugate gradient methods with inverse retraction

We propose a new class of Riemannian conjugate gradient (CG) methods, in which inverse retraction is used instead of vector transport for search direction construction. In existing methods, differentiated retraction is often used for vector transport to move the previous search direction to the current tangent space. However, a different perspective is adopted here, motivated by the fact that inverse retraction directly measures the displacement from the current to the previous points in terms of tangent vectors at the current point. The proposed algorithm is implemented with the Fletcher–Reeves and the Dai–Yuan formulae, respectively, and global convergence is established using modifications of the Riemannian Wolfe conditions. Computational details of the practical inverse retractions over the Stiefel and fixed-rank manifolds are discussed. Numerical results obtained for the Brockett cost function minimization problem, the joint diagonalization problem, and the low-rank matrix completion problem demonstrate the potential effectiveness of Riemannian CG with inverse retraction.

Approximate Joint Diagonalization with Riemannian Optimization on the General Linear Group

We consider the classical problem of approximate joint diagonalization of matrices, which can be cast as an optimization problem on the general linear group. We propose a versatile Riemannian optim...

Perturbation analysis for matrix joint block diagonalization

The matrix joint block-diagonalization problem (jbdp) of a given matrix set A={Ai}i=1m is about finding a nonsingular matrix W such that all WTAiW are block-diagonal. It includes the matrix joint diagonalization problem (jdp) as a special case for which all WTAiW are required diagonal. Generically, such a matrix W may not exist, but there are practical applications such as multidimensional independent component analysis (MICA) for which it does exist under the ideal situation, i.e., no noise is presented. However, in practice noises do get in and, as a consequence, the matrix set is only approximately block-diagonalizable, i.e., one can only make all W˜TAiW˜ nearly block-diagonal at best, where W˜ is an approximation to W, obtained usually by computation. The main goal of this paper is to develop a perturbation theory for jbdp to address, among others, the question: how accurate this W˜ is. Previously such a theory for jdp has been discussed, but no effort had been attempted for jbdp because, in large part, there is no quantitative way to describe solution uniqueness of jbdp until 2017 when Cai and Liu (2017) [9] successfully obtained a necessary and sufficient uniqueness condition. Based on the condition, in this article, we will establish a perturbation theory for jbdp. Our main contributions include an error bound for the approximate block-diagonalizer W˜ and a backward error analysis for jbdp. Numerical tests are presented to validate the theoretical results.

Blind separation of temporally correlated noncircular sources using complex matrix joint diagonalization

In this paper, we introduce a new blind source separation (BSS) method for temporal correlated noncircular sources that uses widely linear filter (WLF) model to efficiently describe the temporal structure. The algorithm consists of a WLF coefficients estimator followed by complex matrix joint diagonalization. In the derivation of the new BSS algorithm, an emerging matrix joint diagonalization issue needs to be addressed. Subsequently, a complex matrix joint diagonalization algorithm based on successive Shear and Givens rotations (SGR) is proposed to deal with the new joint diagonalization problem. Simulations of SGR show that it converges fast and converges to excellent value, and it shows a satisfactory anti-perturbation performance of matrix for the new BSS algorithm. We compare the performance of the new BSS algorithm with several competing algorithms, and the simulation results demonstrate the superior performance of the proposed BSS algorithm for both noncircular complex Gaussian and non-Gaussian sources.

Hybrid Joint Diagonalization Algorithms

This paper deals with a hybrid joint diagonalization (JD) problem considering both Hermitian and transpose congruences. Such problem can be encountered in certain non-circular signal analysis applications including blind source separation. We introduce new Jacobi-like algorithms using Givens or a combination of Givens and hyperbolic rotations. These algorithms are compared with state-of-the-art methods and their performance gain, especially in the high dimensional case, is assessed through simulation experiments including examples related to blind separation of non-circular sources.

LU-based Jacobi-like algorithms for non-orthogonal joint diagonalization

In this paper, based on the LU decomposition, we propose three non-orthogonal Jacobi-like alternating iterative algorithms with two strategies for solving the joint diagonalization problem of a set of Hermitian matrices. In this kind of algorithm, each transformation includes one upper triangular iterative step and one lower triangular iterative step, and each step involves one parameter. The optimal parameter of each step is derived analytically. The convergence of our proposed algorithms is proven. According to this convergence analysis, the existing GNJD algorithm is revisited. Finally, numerical simulations are presented to illustrate the effectiveness of the proposed algorithms in comparison with existing ones.

Riemannian Optimization and Approximate Joint Diagonalization for Blind Source Separation

We consider the blind source separation (BSS) problem and the closely related approximate joint diagonalization (AJD) problem of symmetric positive definite (SPD) matrices. These two problems can be reduced to an optimization problem with three key components: the criterion to minimize, the constraint on the solution, and the optimization algorithm to solve it. This paper contains two contributions that allow us to treat these issues independently. We build the first complete Riemannian optimization framework suited for BSS and AJD handling three classical constraints, and allowing to use a large panel of general optimization algorithms on manifolds. We also perform a thorough study of the AJD problem of SPD matrices from an information geometry point of view. We study AJD criteria based on several divergences of the set of SPD matrices, provide three optimization strategies to minimize them, and analyze their properties. Our numerical experiments on simulated and pseudoreal electroencephalographic data show the interest of the Riemannian optimization framework and of the different AJD criteria we consider.

Nonlinear Blind Source Separation Unifying Vanishing Component Analysis and Temporal Structure

Nonlinear blind source separation (BSS) is one of the unsolved problems in unsupervised learning, because the solutions are highly non-unique when there is no prior information for the mixing functions. In this paper, we present a novel approach to tackle the ill-posedness of the nonlinear BSS problem with a few assumptions. The derivation of our algorithm is inspired by the idea of an efficient layer-by-layer representation to approximate the nonlinearity. Once such a representation is built, a final output layer is constructed by solving a convex optimization problem. Relying on the multi-layer architecture, the algorithm transforms a time-invariant nonlinear BSS to the local linear problem with a tolerable computational cost. Then, the projected data can break the nonlinear problem down into the version of a generalized joint diagonalization problem in the feature space. Importantly, the parameters and forms of polynomials depend solely on the input data, which guarantee the robustness of the structure. We thus address the general problem without being restricted to any specific mixture or parametric model. Experiments show that the proposed algorithm has a higher estimation accuracy on audio data sets from the real world for separating various nonlinear mixtures.

Riemannian Newton-type methods for joint diagonalization on the Stiefel manifold with application to independent component analysis

The joint approximate diagonalization of non-commuting symmetric matrices is an important process in independent component analysis. This problem can be formulated as an optimization problem on the Stiefel manifold that can be solved using Riemannian optimization techniques. Among the available optimization techniques, this study utilizes the Riemannian Newton’s method for the joint diagonalization problem on the Stiefel manifold, which has quadratic convergence. In particular, the resultant Newton’s equation can be effectively solved by means of the Kronecker product and the vec and veck operators, which reduce the dimension of the equation to that of the Stiefel manifold. Numerical experiments are performed to show that the proposed method improves the accuracy of the approximate solution to this problem. The proposed method is also applied to independent component analysis for the image separation problem. The proposed Newton method further leads to a novel and fast Riemannian trust-region Newton method for the joint diagonalization problem.

Non-orthogonal joint diagonalization algorithm preventable ill conditioned solutions for blind source separation

The general joint diagonalization problem involves estimating the separating matrix only given mixing signals interrelated matrix set. For nonorthogonal joint diagonalization based on the weighted least-squares criterion, the algorithms may converge to trivial (zero) solution. Certainly, the trivial solution can simply be avoided by adopting some constraint on the diagonalizing matrix or penalty terms. However, free of zero solution is not enough, especially for the blind signal separation (BSS). Actually, ill-conditioned diagonalizer even though nonzero makes the objective function unstable or even divergence in the process optimization. Therefore, it is necessary to prevent the iterative solutions from degenerating ill-conditioned forms. To solve this problem, a novel nonleast-squares criterion for non-orthogonal joint diagonalization is proposed. It is imposed constrainted terms on diagonalizers, which are induced form the mathematic define of the ill condition matrix. Finally, Computer simulations indicate that the new algorithm yields diagonalizers which not only minimize the diagonalization error but also have as small condition numbers as possible, meanwhile, degenerate solutions are avoided strictly.

Diagonality measures of Hermitian positive-definite matrices with application to the approximate joint diagonalization problem

In this paper, we introduce properly-invariant diagonality measures of Hermitian positive-definite matrices. These diagonality measures are defined as distances or divergences between a given positive-definite matrix and its diagonal part. We then give closed-form expressions of these diagonality measures and discuss their invariance properties. The diagonality measure based on the log-determinant α-divergence is general enough as it includes a diagonality criterion used by the signal processing community as a special case. These diagonality measures are then used to formulate minimization problems for finding the approximate joint diagonalizer of a given set of Hermitian positive-definite matrices. Numerical computations based on a modified Newton method are presented and commented.

Estimating multivariate latent-structure models

A constructive proof of identification of multilinear decompositions of multiway arrays is presented. It can be applied to show identification in a variety of multivariate latent structures. Examples are finite-mixture models and hidden Markov models. The key step to show identification is the joint diagonalization of a set of matrices in the same nonorthogonal basis. An estimator of the latent-structure model may then be based on a sample version of this joint-diagonalization problem. Algorithms are available for computation and we derive distribution theory. We further develop asymptotic theory for orthogonal-series estimators of component densities in mixture models and emission densities in hidden Markov models.

Some perturbation results for a normalized Non-Orthogonal Joint Diagonalization problem

Non-Orthogonal Joint Diagonalization (NOJD) of a given real symmetric matrix set A={Aj}j=0p is to find a nonsingular matrix W such that W⊤AjW for j=0,1,…,p are all as diagonal as possible. If the columns of the solution W are all required to be unit length, we call such NOJD problem as the Normalized NOJD (NNOJD) problem. In this paper, we discuss the perturbation theory for NNOJD as an optimization problem. Based on the perturbation analysis of general constrained optimization problem given in [16], we obtain an upper bound for the distance between an approximated solution of the perturbed optimal problem and the set of exact joint diagonalizers. As corollaries, a perturbation bound and an error bound are also given. Numerical examples validate the bounds.

Relative gradient based algorithms for general joint diagonalization of complex matrices

This article deals with the problem of joint diagonalization of hermitian and/or complex symmetric matrices. Within the framework of gradient algorithms, we develop various algorithms which are based on different levels of approximation of the classical diagonalization criterion. The algorithms are based on a multiplicative update and on the derivation of an optimal step-size. One of the algorithms is a generalization of DOMUNG to the complex case. Finally, in the blind source separation context, computer simulations illustrate the relative performances of some proposed algorithms in comparison to the true gradient one.

A Decoupled Jacobi-Like Algorithm for Non-Unitary Joint Diagonalization of Complex-Valued Matrices

We consider the problem of non-orthogonal joint diagonalization of a set of complex matrices. This appears in many signal processing problems and is instrumental in source separation. We propose a new Jacobi-like algorithm based both on a special parameterization of the diagonalizing matrix and on an adapted local criterion. The optimization scheme is based on an alternate estimation of the useful parameters. Numerical simulations illustrate the overall very good performances of the proposed algorithm in comparison to two other Jacobi-like algorithms and to a global algorithm existing in the literature.

A Riemannian symmetric rank-one trust-region method

The well-known symmetric rank-one trust-region method--where the Hessian approximation is generated by the symmetric rank-one update--is generalized to the problem of minimizing a real-valued function over a $$d$$ d -dimensional Riemannian manifold. The generalization relies on basic differential-geometric concepts, such as tangent spaces, Riemannian metrics, and the Riemannian gradient, as well as on the more recent notions of (first-order) retraction and vector transport. The new method, called RTR-SR1, is shown to converge globally and $$d+1$$ d + 1 -step q-superlinearly to stationary points of the objective function. A limited-memory version, referred to as LRTR-SR1, is also introduced. In this context, novel efficient strategies are presented to construct a vector transport on a submanifold of a Euclidean space. Numerical experiments--Rayleigh quotient minimization on the sphere and a joint diagonalization problem on the Stiefel manifold--illustrate the value of the new methods.