Enhanced Line Search Research Articles

Tensor-Based Underdetermined Blind Identification of Instantaneous Mixtures

In order to explore the effective statistical information of observed signals to stack into a tensor, while reduce the computational complexity and speed up the convergence, a new tensor-based underdetermined blind identification method of instantaneous mixture is proposed in this brief. Firstly, the autocovariance matrix of each observed sub-block is calculated to construct the symmetric third-order tensor according to segmentation scheme. Secondly, the original tensor is reduced into a low-rank kernel tensor by truncated multi-linear singular value decomposition (MLSVD). Finally, an enhanced line search (ELS) technique is applied to speed up the convergence of alternating least squares (ALS) to identify the mixing matrix. The simulation results indicate that the presented approach provides a superior estimated performance compared to the existing methods in both real and complex scenarios.

Low-complexity tensor-based blind receivers for MIMO systems

Abstract In the tensor-based MIMO receivers, the multidimensional MIMO signals first are expressed as a third-order tensor model, wherein the factor matrices of tensor model are corresponding time/frequency, symbols, code/diversity of signals. A algorithm then is used for fitting this tensor mode, in which the symbols are estimated as a independent factor matrix. Although the performance of tensor-based receivers strongly depends on the initializations of the factor matrices. However, due to the absence of a priori on channels, these initializations are done randomly in alternating least squares (ALS), a basic algorithm for fitting the tensor models. In order to avoid these random initializations, this paper proposes two algorithms for fitting the tensor models. The first one, called delta bilinear ALS (DBALS) algorithm, where we exploit the increment values between two iterations of the factor matrices, refine these predictions by using the enhanced line search and use these refined values to initialize for two factor matrices. The second one, called orthogonal DBALS algorithm that takes into account the potential orthogonal in factor matrix for the DBALS algorithm, to provide the initialization for this factor matrix. By this way, we avoid random initializations for three factor matrices of tensor model. The performance of proposed receivers is illustrated by means of simulation results and a comparison is made with traditional ALS algorithm and other receivers. Beside a performance improving, our receivers give a lower complexity due to avoid random initializations.

CFO robust blind receivers for MIMO-OFDM systems based on PARALIND factorizations

Carrier frequency offsets (CFO) is a well known impairment that destroys the orthogonality amongst subcarriers in multi-input multi-output orthogonal frequency division multiplexing (MIMO-OFDM) systems, and the resulting inter-carrier interference significantly degrades the performance of such systems. This paper proposes the CFO robust blind receivers for MIMO-OFDM systems based on parallel profiles with linear dependencies (PARALIND) factorizations. First, we model MIMO-OFDM signal in the presence of CFO to satisfy the PARALIND model, where the symbols, the CFO, and the MIMO channel matrices can be viewed as three independent factor matrices. Such a hybrid tensorial modeling enables a blind and joint estimation of symbols and CFO. Then, in order to improve the fitting precision of the PARALIND model, we introduce two algorithms for fitting the PARALIND model. The first one, called delta alternating least squares (DALS) algorithm, where we predict coarsely by exploiting the increment values between two iterations of the factor matrices, refine these predictions by using the enhanced line search and use these refined values to initialize factor matrices, this procedure helps us avoid the random initialization found in the traditional ALS algorithm. The second one, called orthogonal DALS (ODALS) algorithm that uses the solution of orthogonal constraint for the DALS algorithm to provide the initialization for one more factor matrix. The performance of proposed receivers is illustrated by means of simulation results, and a comparison is made with other approaches that allow to estimate symbols and CFO in two separated steps. Beside a robust to CFO characteristics, our approach gives a better BER performance than other compared blind approaches.

Partitioned Alternating Least Squares Technique for Canonical Polyadic Tensor Decomposition

Canonical polyadic decomposition (CPD), also known as parallel factor analysis, is a representation of a given tensor as a sum of rank-one components. Traditional method for accomplishing CPD is the alternating least squares (ALS) algorithm. Convergence of ALS is known to be slow, especially when some factor matrices of the tensor contain nearly collinear columns. We propose a novel variant of this technique, in which the factor matrices are partitioned into blocks, and each iteration jointly updates blocks of different factor matrices. Each partial optimization is quadratic and can be done in closed form. The algorithm alternates between different random partitionings of the matrices. As a result, a faster convergence is achieved. Another improvement can be obtained when the method is combined with the enhanced line search of Rajih et al. Complexity per iteration is between those of the ALS and the Levenberg–Marquardt (damped Gauss–Newton) method. It is important, however, that the idea of alternating quadratic optimization with partitioned factor matrices is general and can be applied to other variants of the tensor decomposition problems, e.g., when non-Gaussian additive noise is considered.

Decomposition of 3-way arrays: A comparison of different PARAFAC algorithms

Abstract The PARAFAC method is an approach to decompose multidimensional arrays into component matrices for a given number of components. The most common way for calculating the decomposition is the alternating least squares method (ALS). Many other algorithms are modifications of ALS, including algorithms utilizing line search, enhanced line search or Tikhonov regularization. These algorithms and a new one, combining line search and Tikhonov regularization, are discussed in this paper. It is demonstrated that the new algorithm combines fast computation and successful handling of ill-conditioned problems, like given in the case of bottlenecks and swamps. Another point discussed is the application of compression for improving the algorithms considered.

A proximal ANLS algorithm for nonnegative tensor factorization with a periodic enhanced line search

The Alternating Nonnegative Least Squares (ANLS) method is commonly used for solving nonnegative tensor factorization problems. In this paper, we focus on algorithmic improvement of this method. We present a Proximal ANLS (PANLS) algorithm to enforce convergence. To speed up the PANLS method, we propose to combine it with a periodic enhanced line search strategy. The resulting algorithm, PANLS/PELS, converges to a critical point of the nonnegative tensor factorization problem under mild conditions. We also provide some numerical results comparing the ANLS and PANLS/PELS methods.

CONFAC Decomposition Approach to Blind Identification of Underdetermined Mixtures Based on Generating Function Derivatives

This work proposes a new tensor-based approach to solve the problem of blind identification of underdetermined mixtures of complex-valued sources exploiting the cumulant generating function (CGF) of the observations. We show that a collection of second-order derivatives of the CGF of the observations can be stored in a third-order tensor following a constrained factor (CONFAC) decomposition with known constrained structure. In order to increase the diversity, we combine three derivative types into an extended third-order CONFAC decomposition. A detailed uniqueness study of this decomposition is provided, from which easy-to-check sufficient conditions ensuring the essential uniqueness of the mixing matrix are obtained. From an algorithmic viewpoint, we develop a CONFAC-based enhanced line search (CONFAC-ELS) method to be used with an alternating least squares estimation procedure for accelerated convergence, and also analyze the numerical complexities of two CONFAC-based algorithms (namely, CONFAC-ALS and CONFAC-ELS) in comparison with the Levenberg-Marquardt (LM)-based algorithm recently derived to solve the same problem. Simulation results compare the proposed approach with some higher-order methods. Our results also corroborate the advantages of the CONFAC-based approach over the competing LM-based approach in terms of performance and computational complexity.

New ALS Methods With Extrapolating Search Directions and Optimal Step Size for Complex-Valued Tensor Decompositions

In signal processing, data analysis and scientific computing, one often encounters the problem of decomposing a tensor into a sum of contributions. To solve such problems, both the search direction and the step size are two crucial elements in numerical algorithms, such as alternating least squares algorithm (ALS). Owing to the nonlinearity of the problem, the often used linear search direction is not always powerful enough. In this paper, we propose two higher-order search directions. The first one, geometric search direction, is constructed via a combination of two successive linear directions. The second one, algebraic search direction, is constructed via a quadratic approximation of three successive iterates. Then, in an enhanced line search along these directions, the optimal complex step size contains two arguments: modulus and phase. A current strategy is ELSCS that finds these two arguments alternately. So it may suffer from a local optimum. We broach a direct method, which determines these two arguments simultaneously, so as to obtain the global optimum. Finally, numerical comparisons on various search direction and step size schemes are reported in the context of blind separation-equalization of convolutive DS-CDMA mixtures. The results show that the new search directions have greatly improve the efficiency of ALS and the new step size strategy is competitive.

Computing the polyadic decomposition of nonnegative third order tensors

Computing the minimal polyadic decomposition (also often referred to as canonical decomposition or sometimes Parafac) amounts to finding the global minimum of a coercive polynomial in many variables. In the case of arrays with nonnegative entries, the low-rank approximation problem is well posed. In addition, due to the large dimension of the problem, the decomposition can be rather efficiently calculated with the help of preconditioned nonlinear conjugate gradient algorithms, as subsequently shown, if equipped with an algebraic calculation of the globally optimal stepsize in low dimension. Other algorithms are also studied (gradient and quasi-Newton approaches) for comparisons. Two versions of each algorithm are considered: the enhanced line search version (ELS), and the backtracking version alternating with ELS. Computer simulations are provided and demonstrate the good behavior of these algorithms dedicated to nonnegative arrays, compared to others put forward in the literature. Finally, applications in the context of data analysis illustrate various algorithms. The main advantage of the suggested approach is to explicitly take into account the nonnegative nature of the loading matrices in the problem parameterization, instead of enforcing positive entries by projection. According to the experiments we have run, such an approach also happens to be more robust with respect to possible modeling errors.

Iterative methods for the canonical decomposition of multi-way arrays: Application to blind underdetermined mixture identification

Two main drawbacks can be stated in the alternating least square (ALS) algorithm used to fit the canonical decomposition (CAND) of multi-way arrays. First its slow convergence caused by the presence of collinearity between factors in the multi-way array it decomposes. Second its blindness to Hermitian symmetries of the considered arrays. Enhanced line search (ELS) scheme was found to be a good way to cope with the slow convergence of the ALS algorithm together with a partial use of the Hermitian symmetry. However, to our knowledge, required equations to perform the latter scheme are only given in the case of third and fifth order arrays. Therefore, our first contribution consists in generalizing the ELS procedure to the case of complex arrays of any order greater than three. Our second contribution is another improvement of the ALS scheme, able to profit from Hermitianity and positive semi-definiteness of the considered arrays. It consists in resorting to the CAND first of a third order array having one unitary loading matrix and second of several rank-1 arrays. An iterative algorithm is then proposed alternating between Procrustes problem solving and the computation of rank-one matrix approximations in order to achieve the CAND of the third order array.

Tensor decompositions, alternating least squares and other tales

AbstractThis work was originally motivated by a classification of tensors proposed by Richard Harshman. In particular, we focus on simple and multiple ‘bottlenecks’, and on ‘swamps’. Existing theoretical results are surveyed, some numerical algorithms are described in details, and their numerical complexity is calculated. In particular, the interest in using the enhanced line search (ELS) enhancement in these algorithms is discussed. Computer simulations feed this discussion. Copyright © 2009 John Wiley & Sons, Ltd.

Enhanced Line Search: A Novel Method to Accelerate PARAFAC

Several modifications have been proposed to speed up the alternating least squares (ALS) method of fitting the PARAFAC model. The most widely used is line search, which extrapolates from linear trends in the parameter changes over prior iterations to estimate the parameter values that would be obtained after many additional ALS iterations. We propose some extensions of this approach that incorporate a more sophisticated extrapolation, using information on nonlinear trends in the parameters and changing all the parameter sets simultaneously. The new method, called “enhanced line search (ELS),” can be implemented at different levels of complexity, depending on how many different extrapolation parameters (for different modes) are jointly optimized during each iteration. We report some tests of the simplest parameter version, using simulated data. The performance of this lowest-level of ELS depends on the nature of the convergence difficulty. It significantly outperforms standard LS when there is a “convergence bottleneck,” a situation where some modes have almost collinear factors but others do not, but is somewhat less effective in classic “swamp” situations where factors are highly collinear in all modes. This is illustrated by examples. To demonstrate how ELS can be adapted to different N-way decompositions, we also apply it to a four-way array to perform a blind identification of an under-determined mixture (UDM). Since analysis of this dataset happens to involve a serious convergence “bottleneck” (collinear factors in two of the four modes), it provides another example of a situation in which ELS dramatically outperforms standard line search.

An enhanced line search scheme for complex-valued tensor decompositions. Application in DS-CDMA

In this paper, we introduce an enhanced line search algorithm to accelerate the convergence of the alternating least squares (ALS) algorithm, which is often used to decompose a tensor in a sum of contributions. This scheme can be used for the computation in the complex case of the Parallel Factor model or the more general block component model. We then illustrate the performance of the algorithm in the context of blind separation-equalization of convolutive DS-CDMA mixtures.