Non-convex Optimization Algorithm Research Articles

Robust Gain-Scheduled PID Control: A Parameter Dependent BMI Solution

Abstract In control practices, problems of parametric or time-varying uncertainties must be dealt with. Robust control based on norm theory and convex and non-convex optimization algorithms is a powerful tool to solve these problems in theory, but it is employed rarely in applications. In most engineering cases, Proportional-Integration-Derivative (PID) control is still the most popular method for its easy-to-tune and controllable properties. The control method proposed in this paper integrates the PID control into robust control formulation as a robust Structured Static Output Feedback (SSOF) problem of Linear-Parameter-Varying (LPV) systems, which can be converted into a Parameter Dependent Bilinear-Matrix-Inequality (PDBMI) optimization problem. A convex-concave decomposition based method is given to solve the proposed PDBMI problem. The proposed solution has a simple structure in PID form and can guarantee stability and robustness of the system being controlled in the whole operation range with less conservativeness than existing solution.

A Convex Programming Method for Rocket Powered Landing With Angle of Attack Constraint

Real time trajectory planning is vital in rocket precision powered landing guidance. Due to the nonconvex angle of attack (AOA) constraint and other constraints, including nonlinear dynamics and thrust constraint, the powered landing trajectory planning problem is highly nonconvex, which makes it difficult to be solved in real time via existing nonconvex optimization algorithms. As the main contribution of this paper, the AOA constraint is taken into account in the problem. A convex feasible set method is presented to handle it, in which a quadratic concave function is introduced to find a convex feasible subset in the original AOA constraint and the second order term of this function is estimated by Gersgorin disc theorem. For the remaining nonconvexities, the lossless convexification is employed to address the thrust constraint, and the successive linearization is performed to handle the nonlinear dynamics. Thus, a convex optimization problem with AOA constraint for landing trajectory planning is built. The optimal solution to the original problem is obtained by iteratively solving convex problems. Numerical simulations show that the proposed method can find a feasible landing trajectory where the AOA constraint is satisfied and has a better convergence performance compared with the traditional linearization method.

Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview

Substantial progress has been made recently on developing provably accurate and efficient algorithms for low-rank matrix factorization via nonconvex optimization. While conventional wisdom often takes a dim view of nonconvex optimization algorithms due to their susceptibility to spurious local minima, simple iterative methods such as gradient descent have been remarkably successful in practice. The theoretical footings, however, had been largely lacking until recently. In this tutorial-style overview, we highlight the important role of statistical models in enabling efficient nonconvex optimization with performance guarantees. We review two contrasting approaches: (1) two-stage algorithms, which consist of a tailored initialization step followed by successive refinement; and (2) global landscape analysis and initialization-free algorithms. Several canonical matrix factorization problems are discussed, including but not limited to matrix sensing, phase retrieval, matrix completion, blind deconvolution, robust principal component analysis, phase synchronization, and joint alignment. Special care is taken to illustrate the key technical insights underlying their analyses. This article serves as a testament that the integrated consideration of optimization and statistics leads to fruitful research findings.

A Compressed Sensing-Based Element Failure Diagnosis Method for Phased Array Antenna During Beam Steering

A novel diagnosis method for phased array antenna element failure using near-field measurements based on compressed sensing is proposed. In consideration of the beam steering property of phased arrays, sampling of the radiated field is performed with fixed probes. The independence of received field data is attained in the phase domain instead of the conventional spatial domain. It yields a new way to construct the sensing matrix and, hence, the measurement vector. This strategy enables onsite diagnosis while the array under test is still scanning regularly. The reconstruction of antenna element excitations is achieved via sparse approximation methods. A nonconvex optimization algorithm is utilized to reconstruct the unknown excitation amplitudes in consideration of noise. The detection area of a single probe is limited. Arrays consisting of isotropic antennas and Hertzian dipoles are tested to verify the effectiveness of the proposed method.

Faster First-Order Methods for Stochastic Non-Convex Optimization on Riemannian Manifolds.

First-order non-convex Riemannian optimization algorithms have gained recent popularity in structured machine learning problems including principal component analysis and low-rank matrix completion. The current paper presents an efficient Riemannian Stochastic Path Integrated Differential EstimatoR (R-SPIDER) algorithm to solve the finite-sum and online Riemannian non-convex minimization problems. At the core of R-SPIDER is a recursive semi-stochastic gradient estimator that can accurately estimate Riemannian gradient under not only exponential mapping and parallel transport, but also general retraction and vector transport operations. Compared with prior Riemannian algorithms, such a recursive gradient estimation mechanism endows R-SPIDER with lower computational cost in first-order oracle complexity. Specifically, for finite-sum problems with n components, R-SPIDER is proved to converge to an ϵ-approximate stationary point within [Formula: see text] stochastic gradient evaluations, beating the best-known complexity [Formula: see text]; for online optimization, R-SPIDER is shown to converge with [Formula: see text] complexity which is, to the best of our knowledge, the first non-asymptotic result for online Riemannian optimization. For the special case of gradient dominated functions, we further develop a variant of R-SPIDER with improved linear rate of convergence. Extensive experimental results demonstrate the advantage of the proposed algorithms over the state-of-the-art Riemannian non-convex optimization methods.

Nonconvex optimization-based inverse spectral decomposition

Abstract As a time–frequency analysis tool, inverse spectral decomposition (ISD) could be utilized to obtain a high-resolution time–frequency map via the inversion strategy. In the established inversion function, an analytic signal is disintegrated as a coefficient matrix whose elements represent the weights of the wavelet components with the different dominant frequencies and the time location in the complex wavelet library. By using a sparse constraint, a high-quality inverse decomposition result could be generated. In this paper, a modified ISD technique based on a nonconvex optimization algorithm is proposed to pursue a sparser coefficient solution found in the inversion processing by decreasing the redundant information. This new approach applies the lp(0 < p < 1) penalty term to build an accurate mapping relationship between the original signal and its time–frequency spectrum. This adequate regularization in reconstructed function serves as a better alternative to the l1 norm one in conventional ISD. Lower signal-to-noise ratio (SNR) and much weaker incoherence of wavelet library cannot impact on the accuracy of output with the ISD based on the lp norm (0 < p < 1) constraint. Synthetic data tests with the Gabor transform, the conventional ISD method and the nonconvex optimization-based ISD are applied to demonstrate the performance gaps. In the real application area, a blended image with instantaneous spectral attribute volumes produced by the new approach assists the identification of the geological anomalous body, which is verified via a physical model data and field data tests.

On a Monotone Scheme for Nonconvex Nonsmooth Optimization with Applications to Fracture Mechanics

A general class of nonconvex optimization problems is considered, where the penalty is the composition of a linear operator with a nonsmooth nonconvex mapping, which is concave on the positive real line. The necessary optimality condition of a regularized version of the original problem is solved by means of a monotonically convergent scheme. Such problems arise in continuum mechanics, as for instance cohesive fractures, where singular behaviour is usually modelled by nonsmooth nonconvex energies. The proposed algorithm is successfully tested for fracture mechanics problems. Its performance is also compared to two alternative algorithms for nonsmooth nonconvex optimization arising in optimal control and mathematical imaging.

Reusing the Past Difference Vectors in Differential Evolution-A Simple But Significant Improvement.

Differential evolution (DE) has established itself as a simple but efficient population-based, nonconvex optimization algorithm for continuous search spaces. Unlike the conventional real-coded genetic algorithms (GAs) and evolution strategies (ESs), DE uses a mandatory self-referential mutation for its population members, each of which are perturbed with the scaled difference(s) of the individuals from the current generation (iteration). These difference vectors determine the direction of the search moves for the individuals. However, unlike the better individuals, they are not retained in the elitist evolution cycle of DE. In this paper, we show that by archiving the most promising difference vectors from past generations and then by reusing them for generating offspring in the subsequent generations, we can strikingly improve the performance of DE. This strategy can be integrated with any classical or advanced DE variant with no serious overhead in time or space complexity. We demonstrate that when combined with the DE-based winners of the IEEE Congress on Evolutionary Computation (CEC) 2013, 2014, and 2017 competitions on real parameter optimization, the simple reuse strategy leads to a statistically significant performance improvement in the majority of test cases. We further showcase the efficacy of our proposal on a practical optimization problem concerning the design of circular antenna arrays with a prespecified radiation pattern.

Exactly Robust Kernel Principal Component Analysis.

Robust principal component analysis (RPCA) can recover low-rank matrices when they are corrupted by sparse noises. In practice, many matrices are, however, of high rank and, hence, cannot be recovered by RPCA. We propose a novel method called robust kernel principal component analysis (RKPCA) to decompose a partially corrupted matrix as a sparse matrix plus a high- or full-rank matrix with low latent dimensionality. RKPCA can be applied to many problems such as noise removal and subspace clustering and is still the only unsupervised nonlinear method robust to sparse noises. Our theoretical analysis shows that, with high probability, RKPCA can provide high recovery accuracy. The optimization of RKPCA involves nonconvex and indifferentiable problems. We propose two nonconvex optimization algorithms for RKPCA. They are alternating direction method of multipliers with backtracking line search and proximal linearized minimization with adaptive step size (AdSS). Comparative studies in noise removal and robust subspace clustering corroborate the effectiveness and the superiority of RKPCA.

Two-Timescale Hybrid Compression and Forward for Massive MIMO Aided C-RAN

We consider the uplink of a cloud radio access network (C-RAN), where massive MIMO remote radio heads (RRHs) serve as relays between users and a centralized baseband unit (BBU). Although employing massive MIMO at RRHs can improve the spectral efficiency, it also significantly increases the amount of data transported over the fronthaul links between RRHs and BBU, which becomes a performance bottleneck. Existing fronthaul compression methods for conventional C-RAN are not suitable for the massive MIMO regime because they require fully-digital processing and/or real-time full channel state information (CSI), incurring high implementation cost for massive MIMO RRHs. To overcome this challenge, we propose to perform a two-timescale hybrid analog-and-digital spatial filtering at each RRH to reduce the fronthaul consumption. Specifically, the analog filter is adaptive to the channel statistics to achieve massive MIMO array gain, and the digital filter is adaptive to the instantaneous effective CSI to achieve spatial multiplexing gain. Such a design can alleviate the performance bottleneck of limited fronthaul with reduced hardware cost and power consumption, and is more robust to the CSI delay. We propose an online algorithm for the two-timescale non-convex optimization of analog and digital filters, and establish its convergence to stationary solutions. Finally, simulations verify the advantages of the proposed scheme.

Improving the Signal-to-Noise Ratio of Superresolution Imaging Based on Single-Pixel Camera

Based on the theories of single-pixel camera and compressed sensing image reconstruction, the sparse basis, the projection method of measurement matrix, and the signal reconstruction algorithm are optimized. First, for the sparse representation of image, the restraint matrix is designed combining the characteristics of wavelet sparse coefficients to enhance the sparse representation ability of the discrete wavelet base, which improves the quality of image reconstruction and imaging by single-pixel camera. Then, in the single-pixel camera system, the projection method is improved according to the characteristics of the digital mirror device micromirror, and a new bilateral projection method based on the block diagonal measurement matrix is proposed to realize the superresolution imaging of the outdoor scene, reducing the number of measurement and improving the imaging quality. Finally, for the algorithm of image reconstruction, combining the characteristics of convex optimization and nonconvex optimization algorithms, a new algorithm from convex optimization approximately to nonconvex optimization algorithm is proposed, and compared with the traditional image reconstruction algorithm, such as greedy pursuit algorithm, minimum norm algorithm, and image interpolation, the peak signal-to-noise ratio and imaging quality of reconstructed images are effectively improved. The feasibility of the proposed methods is demonstrated by simulation experiments and imaging experiments of single-pixel camera.

A Flexible Framework for Cubic Regularization Algorithms for Nonconvex Optimization in Function Space

We propose a cubic regularization algorithm that is constructed to deal with nonconvex minimization problems in function space. It allows for a flexible choice of the regularization term and thus accounts for the fact that in such problems one often has to deal with more than one norm. Global and local convergence results are established in a general framework.

Multichannel impedance inversion for nonstationary seismic data based on the modified alternating direction method of multipliers

We have upgraded the conventional nonconvex optimization algorithm and addressed a new technique to estimate the acoustic impedance (AI) for attenuated seismic data based on the modified alternating direction method of multipliers (ADMM). To eliminate the discontinuity in the inverted AI profiles with the existing single-trace processing strategy, we construct a multichannel framework and a dimensional reduction operation is used with a brief matrix manipulation in the designed forward-simulation model. The [Formula: see text] norm constraint in the inversion function can assist to search a global optimal AI solution. In the promoted ADMM solving procedure, we invoke the limited-memory Broyden-Fletcher-Goldfarb-Shanno algorithm and the generalized iterated-shrinkage algorithm to solve the relative suboptimization problems. Our new technique could recover an absolute AI from the nonstationary seismic data directly, even though this inversion problem is seriously ill-posed and nonlinear. The inverted AI profile will be satisfied with the assumption that the calculated reflectivity is comparatively sparse corresponding to its minimum [Formula: see text]-norm. Appending this adequate constraint term to the objective function is crucially significant in reducing the number of estimated AI. Due to the consideration of seismic attenuation in the equation, the inversion approach is deployed into the nonstationary reflection data avoiding the drawbacks brought by the energy-compensation processing (e.g., inverse [Formula: see text]-filtering). Using synthetic and field data, we determine the performance of our method.

On the Landscape of Synchronization Networks: A Perspective from Nonconvex Optimization

Studying the landscape of nonconvex cost function is key towards a better understanding of optimization algorithms widely used in signal processing, statistics, and machine learning. Meanwhile, the famous Kuramoto model has been an important mathematical model to study the synchronization phenomena of coupled oscillators over various network topologies. In this paper, we bring together these two seemingly unrelated objects by investigating the optimization landscape of a nonlinear function $E(\boldsymbol{\theta}) = \frac{1}{2}\sum_{1\leq i,j\leq n} a_{ij}(1-\cos(\theta_i - \theta_j))$ associated to an underlying network and exploring the relationship between the existence of local minima and network topology. This function arises naturally in Burer-Monteiro method applied to $\mathbb{Z}_2$ synchronization as well as matrix completion on the torus. Moreover, it corresponds to the energy function of the homogeneous Kuramoto model on complex networks for coupled oscillators. We prove the minimizer of the energy function is unique up to a global translation under deterministic dense graphs and Erd\H{o}s-R\'enyi random graphs with tools from optimization and random matrix theory. Consequently, the stable equilibrium of the corresponding homogeneous Kuramoto model is unique and the basin of attraction for the synchronous state of these coupled oscillators is the whole phase space minus a set of measure zero. In addition, our results address when the Burer-Monteiro method recovers the ground truth exactly from highly incomplete observations in $\mathbb{Z}_2$ synchronization and shed light on the robustness of nonconvex optimization algorithms against certain types of so-called monotone adversaries. Numerical simulations are performed to illustrate our results.

Large-scale Cost-aware Classification Using Feature Computational Dependency Graph

With the rapid growth of real-time machine learning applications, the process of feature selection and model optimization requires to integrate with the constraints on computational budgets. A specific computational resource in this regard is the time needed for evaluating predictions on test instances. The joint optimization problem of prediction accuracy and prediction-time efficiency draws more and more attention in the data mining and machine learning communities. The runtime cost is dominated by the feature generation process that contains significantly redundant computations across different features that sharing the same computational component in practice. Eliminating such redundancies would obviously reduce the time costs in the feature generation process. Our previous Cost-aware classification using Feature computational dependencies heterogeneous Hypergraph (CAFH) model has achieved excellent performance on the effectiveness. In the big data era, the high dimensionality caused by the heterogeneous data sources leads to the difficulty in fitting the entire hypergraph into the main memory and the high computational cost during the optimization process. Simply partitioning the features into batches cannot give the optimal solution since it will lose some feature dependencies across the batches. To improve the high memory and computational costs in the CAFH model, we propose an equivalent Accelerated CAFH (ACAFH) model based on the lossless heterogeneous hypergraph decomposition. An efficient and effective nonconvex optimization algorithm based on the alternating direction method of multipliers (ADMM) is developed to optimize the ACAFH model. The time and space complexities of the optimization algorithm for the ACAFH model are three and one polynomial degrees less than our previous algorithm for the CAFH model, respectively. Extensive experiments demonstrate the proposed ACAFH model achieves competitive performance on the effectiveness and much better performance on the efficiency.

Non-smooth Non-convex Bregman Minimization: Unification and New Algorithms

We propose a unifying algorithm for non-smooth non-convex optimization. The algorithm approximates the objective function by a convex model function and finds an approximate (Bregman) proximal point of the convex model. This approximate minimizer of the model function yields a descent direction, along which the next iterate is found. Complemented with an Armijo-like line search strategy, we obtain a flexible algorithm for which we prove (subsequential) convergence to a stationary point under weak assumptions on the growth of the model function error. Special instances of the algorithm with a Euclidean distance function are, for example, Gradient Descent, Forward-Backward Splitting, ProxDescent, without the common requirement of a Lipschitz continuous gradient. In addition, we consider a broad class of Bregman distance functions (generated by Legendre functions), replacing the Euclidean distance. The algorithm has a wide range of applications including many linear and non-linear inverse problems in signal/image processing and machine learning.

Optimal Spectral Initialization for Signal Recovery With Applications to Phase Retrieval

We present the optimal design of a spectral method widely used to initialize nonconvex optimization algorithms for solving phase retrieval and other signal recovery problems. Our work leverages recent results that provide an exact characterization of the performance of the spectral method in the high-dimensional limit. This characterization allows us to map the task of optimal design to a constrained optimization problem in a weighted $L^2$ function space. The latter has a closed-form solution. Interestingly, under a mild technical condition, our results show that there exists a fixed design that is uniformly optimal over all sampling ratios. Numerical simulations demonstrate the performance improvement brought by the proposed optimal design over existing constructions in the literature. In a recent work, Mondelli and Montanari have shown the existence of a weak reconstruction threshold below which the spectral method cannot provide useful estimates. Our results serve to complement that work by deriving the fundamental limit of the spectral method beyond the aforementioned threshold.

A Bregman inexact linesearch–based forward–backward algorithm for nonsmooth nonconvex optimization

In this paper, we present a forward-backward linesearch-based algorithm suited for the minimization of the sum of a smooth (possibly nonconvex) function and a convex (possibly nonsmooth) term. Such algorithm first computes inexactly the proximal operator with respect to a given Bregman distance, and then ensures a sufficient decrease condition by performing a linesearch along the descent direction. The proposed approach can be seen as an instance of the more general class of descent methods presented in [7], however, unlike in [7], we do not assume the strong convexity of the Bregman distance used in the proximal evaluation. We prove that each limit point of the iterates sequence is stationary, we show how to compute an approximate proximal–gradient point with respect to a Bregman distance and, finally, we report the good numerical performance of the algorithm on a large scale image restoration problem.

Network resource information scheduling based on non-convex function optimization algorithm

Network resource information scheduling based on non-convex function optimization algorithm

Fundamental Limits of Weak Recovery with Applications to Phase Retrieval

In phase retrieval, we want to recover an unknown signal $${{\varvec{x}}}\in {{\mathbb {C}}}^d$$ from n quadratic measurements of the form $$y_i = |\langle {{\varvec{a}}}_i,{{\varvec{x}}}\rangle |^2+w_i$$ , where $${{\varvec{a}}}_i\in {{\mathbb {C}}}^d$$ are known sensing vectors and $$w_i$$ is measurement noise. We ask the following weak recovery question: What is the minimum number of measurements n needed to produce an estimator $${\hat{{{\varvec{x}}}}}({{\varvec{y}}})$$ that is positively correlated with the signal $${{\varvec{x}}}$$ ? We consider the case of Gaussian vectors $${{\varvec{a}}}_i$$ . We prove that—in the high-dimensional limit—a sharp phase transition takes place, and we locate the threshold in the regime of vanishingly small noise. For $$n\le d-o(d)$$ , no estimator can do significantly better than random and achieve a strictly positive correlation. For $$n\ge d+o(d)$$ , a simple spectral estimator achieves a positive correlation. Surprisingly, numerical simulations with the same spectral estimator demonstrate promising performance with realistic sensing matrices. Spectral methods are used to initialize non-convex optimization algorithms in phase retrieval, and our approach can boost the performance in this setting as well. Our impossibility result is based on classical information-theoretic arguments. The spectral algorithm computes the leading eigenvector of a weighted empirical covariance matrix. We obtain a sharp characterization of the spectral properties of this random matrix using tools from free probability and generalizing a recent result by Lu and Li. Both the upper bound and lower bound generalize beyond phase retrieval to measurements $$y_i$$ produced according to a generalized linear model. As a by-product of our analysis, we compare the threshold of the proposed spectral method with that of a message passing algorithm.