Alternating Minimization Algorithm Research Articles

Robust enhanced indexation optimization with sparse industry layout constraint

In this paper, we investigate an enhanced indexation methodology using robust Conditional Value-at-Risk (CVaR) and group-sparse optimization. A featured difference from the existing literatures is to describe the tail risk using the worst-case CVaR of excess returns, and the process of industry selection using a weighted ℓ∞,1-norm constraint. We develop an accelerated alternating minimization algorithm for solving this problem. At each iteration, this method usually alternately solves a convex cone program, which admits a closed-form solution via convex duality theory, and a projection problem onto a weighted infinity-to-one-ball, where a fixed-point iteration projection method is developed, terminating in finite number of iterations. The global convergence rates in terms of the primal and dual residuals are also provided. Empirical tests on actual data sets are presented to demonstrate the superior out-of-sample performance of our proposed strategy.

Near-optimal bounds for generalized orthogonal Procrustes problem via generalized power method

Given multiple point clouds, how to find the rigid transform (rotation, reflection, and shifting) such that these point clouds are well aligned? This problem, known as the generalized orthogonal Procrustes problem (GOPP), has found numerous applications in statistics, computer vision, and imaging science. While one commonly-used method is finding the least squares estimator, it is generally an NP-hard problem to obtain the least squares estimator exactly due to the notorious nonconvexity. In this work, we apply the semidefinite programming (SDP) relaxation and the generalized power method to solve this generalized orthogonal Procrustes problem. In particular, we assume the data are generated from a signal-plus-noise model: each observed point cloud is a noisy copy of the same unknown point cloud transformed by an unknown orthogonal matrix and also corrupted by additive Gaussian noise. We show that the generalized power method (equivalently alternating minimization algorithm) with spectral initialization converges to the unique global optimum to the SDP relaxation, provided that the signal-to-noise ratio is high. Moreover, this limiting point is exactly the least squares estimator and also the maximum likelihood estimator. Our theoretical bound is near-optimal in terms of the information-theoretic limit (only loose by a factor of the dimension and a log factor). Our results significantly improve the state-of-the-art results on the tightness of the SDP relaxation for the generalized orthogonal Procrustes problem, an open problem posed by Bandeira et al. (2014) [8].

Metalearning-Based Alternating Minimization Algorithm for Nonconvex Optimization.

In this article, we propose a novel solution for nonconvex problems of multiple variables, especially for those typically solved by an alternating minimization (AM) strategy that splits the original optimization problem into a set of subproblems corresponding to each variable and then iteratively optimizes each subproblem using a fixed updating rule. However, due to the intrinsic nonconvexity of the original optimization problem, the optimization can be trapped into a spurious local minimum even when each subproblem can be optimally solved at each iteration. Meanwhile, learning-based approaches, such as deep unfolding algorithms, have gained popularity for nonconvex optimization; however, they are highly limited by the availability of labeled data and insufficient explainability. To tackle these issues, we propose a meta-learning based alternating minimization (MLAM) method that aims to minimize a part of the global losses over iterations instead of carrying minimization on each subproblem, and it tends to learn an adaptive strategy to replace the handcrafted counterpart resulting in advance on superior performance. The proposed MLAM maintains the original algorithmic principle, providing certain interpretability. We evaluate the proposed method on two representative problems, namely, bilinear inverse problem: matrix completion and nonlinear problem: Gaussian mixture models. The experimental results validate the proposed approach outperforms AM-based methods.

Distributed online multi‐task sparse identification for multiple systems with asynchronous updates

AbstractSimultaneous online identification of the structure and parameters for multiple systems can use the multi‐task learning to share parameter sparse patterns across different identification tasks to improve the estimation accuracy. To reduce the communication and computational burden, improve the convergence rate and increase the reliability of the identification task, a distributed asynchronous multi‐task sparse identification algorithm is proposed. The alternating minimization algorithm is applied to design a distributed multi‐task loss function. Then by constructing the auxiliary matrix and transforming the sparse constraint on the parameter matrix to the constructed auxiliary matrix, the asynchronous parameter update and sparse model update can be realized. The algorithm uses a recursive form to complete online identification which is feasible to analyze. After proving the convergence of the identification algorithm, we present an example to support the theoretical analysis and show the advantages of the proposed algorithm.

Sparse kernel k-means for high-dimensional data

The kernel k-means method usually loses its power when clustering high-dimensional data, due to a large number of irrelevant features. We propose a novel sparse kernel k-means clustering (SKKM) to extend the advantages of kernel k-means to the high-dimensional cases. We assign each feature a 0-1 indicator and optimize an equivalent kernel k-means loss function while penalizing the sum of the indicators. An alternating minimization algorithm is proposed to estimate both the class labels and the feature indicators. We prove the consistency of both clustering and feature selection of the proposed method. In addition, we apply the proposed framework to the normalized cut. In the numerical experiments, we demonstrate that the proposed method provides better/comparable performance compared to the existing high-dimensional clustering methods.

Tensor completion with noisy side information

We develop a new model for tensor completion which incorporates noisy side information available on the rows and columns of a 3-dimensional tensor. This method learns a low rank representation of the data along with regression coefficients for the observed noisy features. Given this model, we propose an efficient alternating minimization algorithm to find high-quality solutions that scales to large data sets. Through extensive computational experiments, we demonstrate that this method leads to significant gains in out-of-sample accuracy filling in missing values in both simulated and real-world data. We consider the problem of imputing drug response in three large-scale anti-cancer drug screening data sets: the Genomics of Drug Sensitivity in Cancer (GDSC), the Cancer Cell Line Encyclopedia (CCLE), and the Genentech Cell Line Screening Initiative (GCSI). On imputation tasks with 20% to 80% missing data, we show that the proposed method TensorGenomic matches or outperforms state-of-the-art methods including the original tensor model and a multilevel mixed effects model. With 80% missing data, TensorGenomic improves the R^2 from 0.404 to 0.552 in the GDSC data set, 0.407 to 0.524 in the CCLE data set, and 0.331 to 0.453 in the GCSI data set compared to the tensor model which does not take into account genomic side information.

A non–convex total generalized variation model for image denoising

We propose an effective regularization model based on second–order total generalized variation for image restoration with mixed Poisson–Gaussian noise. An efficient alternating minimization algorithm is employed to solve the considered model. Finally, provided numerical results show that our proposed model can preserve more details and get higher image visual quality than recent state-of-the-art methods.

Using Alternating Minimization and Convexified Carleman Weighted Objective Functional for a Time-Domain Inverse Scattering Problem

This paper considers a 1D time-domain inverse scattering problem for the Helmholtz equation in which penetrable scatterers are to be determined from boundary measurements of the scattering data. It is formulated as a coefficient identification problem for a wave equation. Using the Laplace transform, the inverse problem is converted into an overdetermined nonlinear system of partial differential equations. To solve this system, a Carleman weighted objective functional, which is proved to be strictly convex in an arbitrary set in a Hilbert space, is constructed. An alternating minimization algorithm is used to minimize the Carleman weighted objective functional. Numerical results are presented to illustrate the performance of the proposed algorithm.

Spatiotemporal traffic data imputation by synergizing low tensor ring rank and nonlocal subspace regularization

AbstractSpatiotemporal traffic data usually suffers from missing entries in the data acquisition and transmission process. Existing imputation methods only consider the global/local structure of spatiotemporal traffic data, resulting in insufficient estimation performance. Fortunately, it is found that traffic data admits the nonlocal self‐similarity (NSS) prior. This paper incorporates the global and nonlocal low‐rank priors of traffic data and proposes a tensor completion model for spatiotemporal traffic data imputation. To be specific, the proposed method uses tensor ring (TR) decomposition with an enhanced representation capability to characterize the global low‐TR‐rank prior of traffic data, e.g. the correlation of sensor and time modes of the tensor (i.e. traffic data). An implicit plug‐and‐play (PnP)‐based regularization is further utilized to exploit the NSS prior, which depicts the nonlocal similar traffic data patterns. Furthermore, the proximal alternating minimization algorithm under the PnP framework is derived to solve this model. The experiment results on various datasets and missing scenarios show the superiority of the proposed model.

Non-convex TGV regularized [formula omitted]-norm fidelity model for impulse noise removal

Aiming at preserving the structural features of images from degradations contaminated by impulse noise, this article proposes a novel strategy that inherits the characteristics of ℓ0-norm data fidelity and non-convex total generalized variation regularizer. The ℓ0-norm fidelity is more conducive to removing impulse noise, while the non-convex total generalized variation penalty term is both good at eliminating the staircase artifacts and maintaining neat contours. In addition, a modified alternating minimization algorithm is constructed to optimize the solution of the developed model, which merges the popular iteratively reweighted ℓ1 algorithm and primal-dual approach. Meanwhile, the convergence property of the designed algorithm is briefly presented. Finally, in the experimental part, exhaustive simulation experiments are carried out to demonstrate the effectiveness of the introduced scheme for removing impulse noise. As observed in the numerical results, our scheme possesses obvious superiorities in both visual effects and quantitative comparisons, in contrast to some existing popular image restoration methods.

Hybrid Low-Rank Tensor CP and Tucker Decomposition with Total Variation Regularization for HSI Noise Removal

The acquired hyperspectral images (HSIs) are affected by a mixture of several types of noise, which often suffer from information missing. Corrupted HSIs limit the precision of the subsequent processing. In this paper, the weighted Total Variation-regularized Hybrid Model of CP and Tucker (WTV-HMCT) is proposed to accurately identify the intrinsic structures of the clean HSIs. By jointly minimizing CP rank and Tucker rank in the low-rank tensor approximation, WTV-HMCT fully exploits the high-dimensional structure correlations of HSI. To ensure the piecewise smoothness of the recovered image, the hybrid low-rank tensor decomposition approach integrates the weighted spatial spectral total variation regularization for the separation of the noise-free HSI and mixed noise. By the Alternating Direction Method of Multipliers (ADMM), the optimization model is transformed into two subproblems. Finally, an efficient proximal alternating minimization algorithm is developed to optimize the proposed hybrid low-rank tensor decomposition efficiently. The experimental results show that the proposed model effectively handles Gauss noise, striping noise, and mixed noise and that it outperforms the most advanced methods in terms of evaluation metrics and visual evaluation.

Hyperspectral Image Denoising via Weighted Multidirectional Low-Rank Tensor Recovery.

Recently, low-rank tensor recovery methods based on subspace representation have received increased attention in the field of hyperspectral image (HSI) denoising. Unfortunately, those methods usually analyze the prior structural information within different dimensions indiscriminately, ignoring the differences between modes, leaving substantial room for improvement. In this article, we first consider the low-rank properties in the subspace and prove that the structure correlation across the nonlocal self-similarity mode is much stronger than in the spatial sparsity and spectral correlation modes. On that basis, we introduce a new multidirectional low-rank regularization, in which each mode is assigned a different weight to characterize its contribution to estimating the tensor rank. After that, integrating the proposed regularization with the subspace-based tensor recovery framework, an optimization model for HSI mixed noise removal is developed. The proposed model can be addressed efficiently via the alternating minimization algorithm. Extensive experiments implemented with synthetic and real data demonstrate that the proposed method significantly outperforms other state-of-the-art HSI denoising methods, which clearly indicates the effectiveness of the proposed approach in HSI denoising.

Joint mmWave Beamforming and Resource Allocation in NOMA-MEC Network for Internet of Things

Mobile edge computing (MEC) technology can reduce the computational burden of Internet of Things (IoT) devices by offloading computationally intensive tasks to physically proximate edge servers. The combination of millimeter-wave (mmWave) massive multiple input multiple output (MIMO) and non-orthogonal multiple access (NOMA) technique has draw significant attention for its high spectral efficiency. This paper merges the computation offloading and mmWave massive MIMO-NOMA communication to seek energy efficient performance for IoT devices in MEC network. First, we propose an IoT device clustering algorithm to schedule all IoT devices into limited beams. Following this, we minimize the total energy consumption of computing and offloading in the NOMA-MEC network by jointly optimizing hybrid beamforming, offloading task assignment (OTA) ratio and power allocation. Then, the non-convex joint optimization problem is solved by proposing a suboptimal solution with two stages. In the first stage, a cyclic block-coordinate update algorithm is proposed for analog beamforming design, and then the digital beamforming vectors are obtained by zero-forcing method. In the second stage, an alternating minimization algorithm is proposed to optimize the OTA ratio and power allocation. Finally, simulation results are provided to validate the convergence of the proposed algorithms and the superiority of our proposed schemes over other benchmark schemes, which demonstrate the outstanding performance leveraging the massive MIMO-NOMA technique during computational offloading.

Learning Sparse Kernel CCA With Graph Priors for Nonlinear Process Monitoring

Process monitoring (PM) is important for improving product quality and ensuring plant safety in industrial systems. Recently, canonical correlation analysis (CCA)-based PM has shown excellent performance. The core idea is to first seek the relationship between two sets of process variables via CCA, then construct a residual generator and determine test statistics, and finally achieve the best monitoring. However, the global and local structure of process variables is not fully exploited. In this article, we propose a new nonlinear PM approach called joint sparse constrained kernel CCA and graph learning (JSKCCA-GL) by incorporating joint sparse kernel CCA (JSKCCA) and graph priors. Technically, imposing the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell _{{2},{0}}$ </tex-math></inline-formula> -norm joint sparse constraints can capture the global row-wise structure, and embedding the graph Laplacian priors can preserve the local structured variable correlation information. Furthermore, an efficient alternating minimization algorithm is developed by integrating least squares and iterative hard thresholding (IHT). The superiority of the proposed JSKCCA-GL is extensively verified on the benchmark Tennessee Eastman process (TEP) and a practical cylinder-piston assembly (CPA) of marine diesel engines. In particular, the fault detection rate (FDR) values for unknown-type faults in the TEP have been improved by around 16.10% on average.

Revisiting Convolutional Neural Networks from the Viewpoint of Kernel-Based Methods

Convolutional neural networks, as most artificial neural networks, are frequently viewed as methods different in essence from kernel-based methods. In this work we translate several classical convolutional neural networks into kernel-based counterparts. Each kernel-based counterpart is a statistical model called a convolutional kernel network with parameters that can be learned from data. We provide an alternating minimization algorithm with mini-batch sampling and implicit partial differentiation to learn from data the parameters of each convolutional kernel network. We also show how to obtain inexact derivatives with respect to the parameters using an algorithm based on two inter-twined Newton iterations. The models and the algorithms are illustrated on benchmark datasets in image classification. We find that the convolutional neural networks and their kernel counterparts often perform similarly. Supplemental appendices and code for the article are available online.

Self-Calibration Method of Sensors Array Errors Based on Rotation Measurement

Array calibration is the premise of successful array signal processing. The active calibration needs to know the direction of arrival (DOA) of calibration source signals accurately; otherwise, the performance will deteriorate. Although the conventional self-calibration does not need accurate DOA of calibration sources, it may suffer from falling into the local extremum since there are too many parameters to be estimated. Thus, we propose a novel self-calibration method based on the rotation measurement. The rotation measurement is that the array receives signals from a fixed source with different array rotation angles. The proposed method jointly estimates array errors and DOAs of the sources by an alternative minimization (AM) algorithm. Through the derivation of a suitable initial value in the AM algorithm, we could improve the robustness and reduce the computational complexity. The proposed method can achieve as high calibration accuracy as the active method even if there is an error in the source DOA information and can apply to arbitrary planar arrays. Both simulation and field experiment results demonstrate the effectiveness of the proposed method.

Beam Shaping for Massive MIMO Systems With One-Bit Spatial $\Sigma -\Delta$ ADCs

One-bit quantization opens up opportunities for massive MIMO implementation using cheap and power-efficient radio-frequency front-ends, but unfortunately brings severe amplitude distortion. The spatial <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\Sigma -\Delta$</tex-math></inline-formula> structure is one important manner that can recover the amplitude information from one-bit quantized signals. In this paper, we demonstrate that the upper bound of the spatial degree of quantization noise reduction (DQNR) is 3 for the considered spatial <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\Sigma -\Delta$</tex-math></inline-formula> structure. However, the upper bound is hard to be achieved via the zero forcing (ZF) or the maximum ratio combination (MRC) schemes, and the achieved spatial DQNR is even not greater than 1 in the worst case. Thus the distortion suppression is inefficient. To address this issue, we propose a random beam shaping (RBS) scheme and prove it can achieve the spatial DQNR of 3. Through the random beam shaping, the beams are smeared into the whole beamspace. Meanwhile, the random beam shaping results in the power leakage of the beams and it may limit the distortion suppression when considering the additive noise. To reduce the power leakage, we further propose an accurate beam shaping (ABS) scheme, in which the vector for the beam shaping is optimized through the alternative minimization algorithm and the beams are concentrated around the region with low power of quantization noise. The simulation results illustrate that the proposed beam shaping schemes can more efficiently exploit the large-scale antenna array to recover the amplitude information.

Dictionary Learning With Low-Rank Coding Coefficients for Tensor Completion.

In this article, we propose a novel tensor learning and coding model for third-order data completion. The aim of our model is to learn a data-adaptive dictionary from given observations and determine the coding coefficients of third-order tensor tubes. In the completion process, we minimize the low-rankness of each tensor slice containing the coding coefficients. By comparison with the traditional predefined transform basis, the advantages of the proposed model are that: 1) the dictionary can be learned based on the given data observations so that the basis can be more adaptively and accurately constructed and 2) the low-rankness of the coding coefficients can allow the linear combination of dictionary features more effectively. Also we develop a multiblock proximal alternating minimization algorithm for solving such tensor learning and coding model and show that the sequence generated by the algorithm can globally converge to a critical point. Extensive experimental results for real datasets such as videos, hyperspectral images, and traffic data are reported to demonstrate these advantages and show that the performance of the proposed tensor learning and coding method is significantly better than the other tensor completion methods in terms of several evaluation metrics.

A hybrid alternating minimization algorithm for structured convex optimization problems with application in Poissonian image processing

&lt;p style='text-indent:20px;'&gt;Motivated by applications in image processing, we consider a class of structured convex optimization problems in which the objective function is the sum of two component functions with favorable structures, and, furthermore, both of which are composed with linear operators. In particular, to recover images degraded by Poisson noise, the famous Kullback-Leibler divergence can be adopted in data fitting, in which case one component function is separable, strongly convex and easy to minimize. The other component function usually allows an easily computable proximal operator. We then propose, analyze and test a hybrid algorithm particularly tailored for the underlying model structures, which can be viewed as a combination of the famous alternating minimization algorithm of Tseng and the alternating direction method of multipliers. Under mild conditions, global iterate convergence as well as ergodic &lt;inline-formula&gt;&lt;tex-math id="M1"&gt;\begin{document}$ \mathcal{O}(1/N) $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; sublinear convergence rate results measured by the function value residual and constraint violation of a reformulated constrained optimization problem are established, where &lt;inline-formula&gt;&lt;tex-math id="M2"&gt;\begin{document}$ N $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; denotes the iteration counter. The proposed algorithm is applied to fractional-order total variation regularized image denoising and deblurring problems, where the noise obeys Poisson distribution. Our experimental results demonstrate the effectiveness of the proposed algorithm.&lt;/p&gt;

Three-Dimensional Image Reconstruction With Enhanced Spatial Accuracy for Pulsed-Laser Imaging Based on Compressive Sensing

Compressive sensing (CS) has been intensively studied as a superresolution technique in image acquisition and reconstruction. It has also been introduced into three-dimensional (3-D) laser imaging to enhance the spatial resolution of images obtained from existing few-pixel array sensors. These approaches, however, either suffer from low range resolution or lack long-range detecting abilities. To solve the CS reconstruction problem under the 3-D pulsed-laser imaging framework, we build a sampling model to combine the CS sampling process with the time-of-flight-based range measurement procedure in pulsed-laser imaging. To explore range information from the overlapping echo signals, we develop a gradual alternating minimization algorithm with a designed range equivalent matrix, in which the feasible region of the solution gradually shrinks to make the reconstruction procedure tractable. To further enhance the quality of the reconstructed intensity image, we introduce rank minimization and a fallback mechanism, in which the sparsity and stability of the reconstructed intensity image is refined. In this manner, the accuracy of reconstructed 3-D images can be improved in both the pixel direction and range direction. The simulation results based on real datasets demonstrate that our proposed scheme can yield superior accurate reconstruction and leads to significant improvements compared with recent approaches.