Alternating Minimization Algorithm Research Articles

Incremental CP Tensor Decomposition by Alternating Minimization Method

In practical applications, incremental tensors are very common: only a portion of tensor data is available, and new data are arriving in the next time step or continuously over time. To handle this type of tensors time-saving algorithms are required for online computation. In this paper, we consider incremental CP (CANDECOMP/PARAFAC) decomposition, which requires one to update the CP decomposition after new tensor data, together with the exiting tensor, are ready for analysis. There exist several incremental CP decomposition algorithms, but almost all of these algorithms assume that the number of CP decomposition components remains fixed in the incremental process. The main contribution of this paper is the study of how to add components in the incremental CP decomposition. We derive the coordinate representation of the incremental CP decomposition with respect to a special basis and show related properties of tensor rank and the uniqueness of such incremental CP decomposition. Under the framework of this representation, the proposed method can be solved by using an alternating minimization algorithm. Numerical examples are presented to show the good performance of the proposed algorithms in terms of computational time and data fitting compared with existing methods.

T-Hy-Demosaicing: Hyperspectral Reconstruction Via Tensor Subspace Representation Under Orthogonal Transformation

This article aims to solve the problem of the hyperspectral imagery (HSI) demosaicing under a novel subsampling hyperspectral sensing strategy. The existing method utilizes the periodic structure of subsampling to estimate a fixed subspace in matrix form from the measurement result, which reduces the representation ability of the subspace in iterations and destroys the intrinsic structure of the tensor. To overcome these drawbacks, we propose a tensor-based HSI demosaicing (T-Hy-demosaicing) model with tensor subspace representation, which takes the low-tubal-rankness and the nonlocal self-similarity into account. In particular, we suggest a tensor singular value decomposition based on orthogonal transformation (Tran-based t-SVD) to learn the tensor subspace that possesses a more powerful representation ability. In addition, we develop an effective algorithm to solve the proposed nonconvex model under the framework of the proximal alternating minimization algorithm. Experiments conducted on simulated datasets illustrate that the proposed method outperforms other comparative methods in both visual and quantitative terms.

A new approach for Cauchy noise removal

<abstract> In this paper, a new total generalized variational (TGV) model for restoring images with Cauchy noise is proposed, which contains a non-convex fidelity term and a TGV regularization term. In order to obtain a strictly convex model, we add an appropriate proximal term to the non-convex fidelity term. We prove that the solution of the proposed model exists and is unique. Due to the convexity of the proposed model and in order to get a convergent algorithm, we employ an alternating minimization algorithm to solve the proposed model. Finally, we demonstrate the performance of our scheme by numerical examples. Numerical results demonstrate that the proposed algorithm significantly outperforms some previous methods for Cauchy noise removal. </abstract>

An Efficient Interference-Aware Constrained Massive MIMO Beamforming for mm-Wave JSDM

Low-complexity beamformer design with practical constraints is an attractive research area for hybrid analog/digital systems in mm-wave massive multiple-input multiple-output (MIMO). This paper investigates interference-aware pre-beamformer (analog beamformer) design for joint spatial division and multiplexing (JSDM) which is a user-grouping based two-stage beamforming method. Single-carrier frequency domain equalization (SC-FDE) is employed in uplink frequency-selective channels. First, unconstrained slowly changing statistical analog beamformer of each group, namely, generalized eigenbeamformer (GEB) which has strong interference suppression capability is designed by maximizing the mutual information in reduced dimension. Then, constant-modulus constrained approximations of unconstrained beamformer are obtained by utilizing alternating minimization algorithms for fully connected arrays and fixed subarrays. In addition, a dynamic subarray algorithm is proposed where the connections between radio frequency (RF) chains and antennas are changed with changing channel statistics. Convergence of the proposed alternating minimization-based algorithms is provided along with their complexity analysis. It is observed that the additional complexity of proposed algorithms is insignificant for the overall system design. Although most of the interference is suppressed with the help of proposed constrained beamformers, there may be some residual interference after analog beamforming stage. Thus, minimum mean square error (MMSE) criterion based iterative block decision feedback equalization (IB-DFE) method, which takes the residual interference in reduced dimension into account, is promoted for digital beamforming stage. Simulation results verify the superiority of the proposed interference-aware constrained design over existing approaches in terms of beampattern, spectral efficiency, outage capacity, bit-error rate (BER), and channel estimation accuracy.

Dynamic Point Cloud Inpainting via Spatial-Temporal Graph Learning

The maturity of depth sensors and laser scanning techniques has enabled the convenient acquisition of 3D dynamic point clouds—one natural representation of 3D objects/scenes in motion, leading to a wide range of applications such as immersive tele-presence, autonomous driving, augmented and virtual reality. Nevertheless, dynamic point clouds usually exhibit holes of missing data, thus inpainting is crucial to the subsequent rendering or downstream understanding tasks. Dynamic point cloud inpainting has been largely overlooked so far, which is also quite challenging due to the irregular sampling patterns both in the spatial domain and temporal domain. To this end, we propose an efficient dynamic point cloud inpainting method based on a learnable spatial-temporal graph representation, exploiting both the second-order inter-frame coherence and the intra-frame self-similarity. The key is the second-order inter-frame coherence that enforces the consistent flow in 3D motion over time, for which we search the temporal correspondence in consecutive frames for the same underlying surface by the point-to-plane distance and represent the correlation between them via temporal edge weights in the graph. Based on the second-order inter-frame coherence and intra-frame self-similarity, we formulate dynamic point cloud inpainting as a joint optimization problem of the desired point cloud and underlying spatial-temporal graph, which is regularized by consistency in the temporal edge weights and smoothness in the spatial domain. We analyze and reformulate the optimization, leading to an efficient alternating minimization algorithm. Experimental results show that the proposed approach outperforms several competing methods significantly, both on synthetic holes and real holes.

A nonlocal low rank model for poisson noise removal

<p style='text-indent:20px;'>Patch-based methods, which take the advantage of the redundancy and similarity among image patches, have attracted much attention in recent years. However, these methods are mainly limited to Gaussian noise removal. In this paper, the Poisson noise removal problem is considered. Unlike Gaussian noise which has an identical and independent distribution, Poisson noise is signal dependent, which makes the problem more challenging. By incorporating the prior that a group of similar patches should possess a low-rank structure, and applying the maximum a posterior (MAP) estimation, the Poisson noise removal problem is formulated as an optimization one. Then, an alternating minimization algorithm is developed to find the minimizer of the objective function efficiently. Convergence of the minimizing sequence will be established, and the efficiency and effectiveness of the proposed algorithm will be demonstrated by numerical experiments.

Simultaneous spatial smoothing and outlier detection using penalized regression, with application to childhood obesity surveillance from electronic health records.

Electronic health records (EHRs) have become a platform for data-driven granular-level surveillance in recent years. In this paper, we make use of EHRs for early prevention of childhood obesity. The proposed method simultaneously provides smooth disease mapping and outlier information for obesity prevalence that are useful for raising public awareness and facilitating targeted intervention. More precisely, we consider a penalized multilevel generalized linear model. We decompose regional contribution into smooth and sparse signals, which are automatically identified by a combination of fusion and sparse penalties imposed on the likelihood function. In addition, we weigh the proposed likelihood to account for the missingness and potential nonrepresentativeness arising from the EHR data. We develop a novel alternating minimization algorithm, which is computationally efficient, easy to implement, and guarantees convergence. Simulation studies demonstrate superior performance of the proposed method. Finally, we apply our method to the University of Wisconsin Population Health Information Exchangedatabase.

Blind Deconvolution for Poissonian Blurred Image With Total Variation and L 0-Norm Gradient Regularizations

This paper proposes a regularized blind deconvolution method for restoring Poissonian blurred image. The problem is formulated by utilizing the L0 -norm of image gradients and total variation (TV) to regularize the latent image and point spread function (PSF), respectively, and combining them with the negative logarithmic Poisson log-likelihood. To solve the problem, we propose an approach which combines the methods of variable splitting and Lagrange multiplier to convert the original problem into three sub-problems, and then design an alternating minimization algorithm which incorporates the estimation of PSF and latent image as well as the updation of Lagrange multiplier into account. We also design a non-blind deconvolution method based on TV regularization to further improve the quality of the restored image. Experimental results on both synthetic and real-world Poissonian blurred images show that the proposed method can achieve restored images of very high quality, which is competitive with or even better than some state of the art methods.

A Dynamical Approach to Two-Block Separable Convex Optimization Problems with Linear Constraints

The aim of this manuscript is to approach by means of first order differential equations/inclusions convex programming problems with two-block separable linear constraints and objectives, whereby (at least) one of the components of the latter is assumed to be strongly convex. Each block of the objective contains a further smooth convex function. We investigate the dynamical system proposed and prove that its trajectories converge weakly to a saddle point of the Lagrangian of the convex optimization problem. The dynamical system provides through time discretization the alternating minimization algorithm AMA and also its proximal variant recently introduced in the literature.

Structured Alternating Minimization for Union of Nested Low-Rank Subspaces Data Completion

In this article, we consider a particular data structure consisting of a union of several nested low-rank subspaces with missing data entries. Given the rank of each subspace, we treat the data completion problem, i.e., to estimate the missing entries. Starting from the case of two-dimensional data, i.e., matrices, we show that the union of nested subspaces data structure leads to a structured decomposition U = XY where the factor Y has blocks of zeros that are determined by the rank values. Moreover, for high-dimensional data, i.e., tensors, we show that a similar structured CP decomposition also exists, U = Σ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l=1</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</sup> a <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</sup> ⊗ a <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</sup> ⊗ ··· ⊗ a <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</sup> , where A <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sub> = [a <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> ···a <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</sup> ] contains blocks of zeros determined by the rank values. Based on such structured decompositions, we develop efficient alternating minimization algorithms for both matrix and tensor completions, by enforcing the above structures in each iteration including the initialization. Compared with naive approaches where either the additional rank constraints are ignored, or data completion is performed part by part, the proposed structured alternating minimization approaches exhibit faster convergence and higher recovery accuracy.

Robust multi-view k-means clustering with outlier removal

Contemporary datasets are often comprised of multiple views of data, which provide complete and complementary information in different views, and multi-view clustering is one of the most crucial techniques in multi-view data analysis. However, traditional multi-view clustering methods are sensitive to noises and outliers, suffering from severe performance degradation when the dataset contains many outliers. Moreover, the commonly used multi-view clustering methods are restricted by high time complexity. To address these problems, we propose a robust multi-view k-means algorithm with outlier detection, i.e., Multi-View Clustering with Outlier Removal (MVCOR). This method is designed to remove the outliers and thus boosts the clustering performance on multi-view data with low time complexity. By defining two types of outliers, MVCOR uses the well-defined outlier removal strategy to categorize all the outliers into two specific clusters and performs robust clustering on the clean data at the same time. This strategy significantly improves the clustering performance as well as the model robustness, making MVCOR a more practical approach for real-world scenarios. Besides, the proposed model is efficiently optimized by a well-designed alternating minimization algorithm which is strictly proved to be convergent. Extensive experiments on both synthetic and real-world datasets demonstrate that MVCOR consistently outperforms the related clustering methods on clustering performance as well as robustness to outliers, and achieves comparable performance to the state-of-the-art multi-view outlier detection methods.

EISRP: Efficient infrared signal restoration processing for object tracking in human-robot interaction

Infrared (IR) imaging spectral signal can accurately distinguish different industrial metal materials in the robot vision sensing. The novel IR imaging system can assist the robot limb to gasp the object in the human-robot interaction processing. However, the low-resolution problem is often existed in the IR spectral signals. To solve this issue, we present a novel IR spectral deconvolution method via total variation regularization for robust spectral signal recognition. To reveal the properties of high-resolution IR spectrum, the histogram of spectral gradients is computed and fitted by the logarithmic function. Then, the low-resolution and high-resolution IR spectra can be distinguished by total variation-regularized prior property. To optimize the non-convex deconvolution model, alternating minimization algorithm is introduced to ensure that the instrument function and high-resolution spectral signals converge quickly. The performance of the proposed method is evaluated on the simulated IR spectrum and real spectrum. A large number of quantitative and qualitative experimental results verify that the proposed method can obtain robust results and significantly outperform the existing state-of-the-art spectral deconvolution method. The high-resolution IR spectrum promotes the development of IR spectral recognition in human-robot interaction.

Hyperspectral image denoising based on low-rank coefficients and orthonormal dictionary

Hyperspectral images (HSIs) are unavoidably contaminated by noise during data acquisition and transmission. A variety of noise reduction approaches have been developed for HSIs, in which the low-rank based methods have emerged as a powerful tool. However, most of low-rank based HSI denoising methods are designed to explore the low-rankness in the pixel space under a fixed dictionary, without seeking the optimal dictionary and exploiting the low-rankness of representation coefficients. In this work, a novel HSI denoising model based on Low-Rank Coefficients and Orthonormal Dictionary (LRCOD) is proposed. The novelty of this work lies in LRCOD’s exploiting the low-rankness of representation coefficients and learning an orthonormal dictionary for HSI denoising. To solve the proposed model, an efficient alternating minimization algorithm is developed, since one sub-problem has a closed-form solution and the other can be efficiently solved by the curvilinear search algorithm. Compared with gradient-type algorithms, our algorithm is easy to implement as there is no need to tune optimization parameters like step sizes. The experimental results over both simulated and real datasets verified the superiority of the proposed method. Specifically, under various levels of noise in different scenes, LRCOD yields the best MPSNR in 30 of 35 cases and runs faster than other methods except one non-iterative algorithm.

Deep Residual Autoencoders for Expectation Maximization-Inspired Dictionary Learning.

We introduce a neural-network architecture, termed the constrained recurrent sparse autoencoder (CRsAE), that solves convolutional dictionary learning problems, thus establishing a link between dictionary learning and neural networks. Specifically, we leverage the interpretation of the alternating-minimization algorithm for dictionary learning as an approximate expectation-maximization algorithm to develop autoencoders that enable the simultaneous training of the dictionary and regularization parameter (ReLU bias). The forward pass of the encoder approximates the sufficient statistics of the E-step as the solution to a sparse coding problem, using an iterative proximal gradient algorithm called FISTA. The encoder can be interpreted either as a recurrent neural network or as a deep residual network, with two-sided ReLU nonlinearities in both cases. The M-step is implemented via a two-stage backpropagation. The first stage relies on a linear decoder applied to the encoder and a norm-squared loss. It parallels the dictionary update step in dictionary learning. The second stage updates the regularization parameter by applying a loss function to the encoder that includes a prior on the parameter motivated by Bayesian statistics. We demonstrate in an image-denoising task that CRsAE learns Gabor-like filters and that the EM-inspired approach for learning biases is superior to the conventional approach. In an application to recordings of electrical activity from the brain, we demonstrate that CRsAE learns realistic spike templates and speeds up the process of identifying spike times by 900× compared with algorithms based on convex optimization.

Irreversibility and alternate minimization in phase field fracture: a viscosity approach

This work is devoted to the analysis of convergence of an alternate (staggered) minimization algorithm in the framework of phase field models of fracture. The energy of the system is characterized by a nonlinear splitting of tensile and compressive strains, featuring non-interpenetration of the fracture lips. The alternating scheme is coupled with an L^{2}-penalization in the phase field variable, driven by a viscous parameter delta >0, and with an irreversibility constraint, forcing the monotonicity of the phase field only w.r.t. time, but not along the whole iterative minimization. We show first the convergence of such a scheme to a viscous evolution for delta >0 and then consider the vanishing viscosity limit delta rightarrow 0.

Further Development of Subspace Imaging to Magnetic Resonance Fingerprinting: A Low-rank Tensor Approach.

Magnetic resonance fingerprinting is a recent quantitative MRI technique that simultaneously acquires multiple tissue parameter maps (e.g., T1, T2, and spin density) in a single imaging experiment. In our early work, we demonstrated that the low-rank/subspace reconstruction significantly improves the accuracy of tissue parameter maps over the conventional MR fingerprinting reconstruction that utilizes simple pattern matching. In this paper, we generalize the low-rank/subspace reconstruction by introducing a multilinear low-dimensional image model (i.e., a low-rank tensor model). With this model, we further estimate the subspace associated with magnetization evolutions to simplify the image reconstruction problem. The proposed formulation results in a nonconvex optimization problem which we solve by an alternating minimization algorithm. We evaluate the performance of the proposed method with numerical experiments, and demonstrate that the proposed method improves the conventional reconstruction method and the state-of-the-art low-rank reconstruction method.

Feature and Nuclear Norm Minimization for Matrix Completion

Matrix completion, whose goal is to recover a matrix from a few entries observed, is a fundamental model behind many applications. Our study shows that, in many applications, the to-be-complete matrix can be represented as the sum of a low-rank matrix and a sparse matrix associating with side information matrices. The low-rank matrix depicts the global patterns while the sparse matrix characterizes the local patterns, which are often described by the side information. Accordingly, to achieve high-quality matrix completion, we propose a Feature and Nuclear Norm Minimization (FNNM) model. The rationale of FNNM is to employ transductive completion to generalize the global pattern and inductive completion to recover the local pattern. Alternative minimization algorithm based on fixed-point iteration is developed to numerically solve the FNNM model. FNNM has demonstrated promising results on a variety of applications, including movie recommendation, drug-target interaction prediction, and multi-label learning, consistently outperforming the state-of-the-art matrix completion algorithms.

Semi-Supervised Learning with Auto-Weighting Feature and Adaptive Graph

Traditional graph-based Semi-Supervised Learning (SSL) methods usually contain two separate steps. First, constructing an affinity matrix. Second, inferring the unknown labels. While such a two-step method has been successful, it cannot take full advantage of the correlation between affinity matrix and label information. In order to address the above problem, we propose a novel graph-based SSL method. It can learn the affinity matrix and infer the unknown labels simultaneously. Moreover, feature selection with auto-weighting is introduced to extract the effective and robust features. Further, the proposed method learns the data similarity matrix by assigning the adaptive neighbors for each data point based on the local distance. We solve the unified problem via an alternative minimization algorithm. Extensive experimental results on synthetic data and benchmark data show that the proposed method consistently outperforms the state-of-the-art approaches.

A symmetric alternating minimization algorithm for total variation minimization

In this paper, we propose a novel symmetric alternating minimization algorithm to solve a broad class of total variation (TV) regularization problems. Unlike the usual zk → xk Gauss-Seidel cycle, the proposed algorithm performs the special x¯k→zk→xk cycle. The main idea for our setting is the recent symmetric Gauss-Seidel (sGS) technique which is developed for solving the multi-block convex composite problem. This idea also enables us to build the equivalence between the proposed method and the well-known accelerated proximal gradient (APG) method. The faster convergence rate of the proposed algorithm can be directly obtained from the APG framework and numerical results including image denoising, image deblurring, and analysis sparse recovery problem demonstrate the effectiveness of the new algorithm.

Semi-Supervised Multi-view clustering based on orthonormality-constrained nonnegative matrix factorization

Multi-view clustering aims at integrating the complementary information between different views so as to obtain an accurate clustering result. In addition, the traditional clustering is a kind of unsupervised learning method, which does not take the label information into learning. In this paper, we propose a novel model, called semi-supervised multi-view clustering based on orthonormality-constrained nonnegative matrix factorization (MVOCNMF), to cluster the multi-view data into a number of categories. In the proposed model, based on the label information, we first learn the low-dimensional representations of data by the constrained NMF technique, and simultaneously cluster the samples with the same label into the clustering prototypes for each view. After that, we put forward a novel orthonormality constraint term to obtain the desirable representations for each view, and use the co-regularization to integrate the complementary information from different views. We further develop an alternating minimization algorithm to solve the proposed model, and present the convergence analysis and computational complexity of the proposed method. Extensive experimental results on several multi-view datasets have shown that the proposed MVOCNMF method outperforms the existing multi-view clustering methods.