Unconstrained Convex Optimization Problem Research Articles

A warm-start FE-dABCD algorithm for elliptic optimal control problems with constraints on the control and the gradient of the state

In this paper, elliptic control problems with the integral constraint on the gradient of the state and the box constraint on the control are considered. The optimality conditions for the problem are proved. To numerically solve the problem, a finite element duality-based inexact majorized accelerated block coordinate descent (FE-dABCD) algorithm is proposed. Specifically, both the state and the control are discretized by piecewise linear functions. An inexact majorized ABCD algorithm is employed to solve the discretized problem via its dual, which is a multi-block unconstrained convex optimization problem, but the primal variables are also generated in each iteration. Thanks to the inexactness of the FE-dABCD algorithm, the subproblems at each iteration are allowed to be solved inexactly. For the smooth subproblem, we use the preconditioned generalized minimal residual (GMRES) method to solve it. For the two nonsmooth subproblems, one of them has a closed form solution through introducing an appropriate proximal term, and another one is solved by the line search Newton's method. Based on these efficient strategies, we prove that our proposed FE-dABCD algorithm enjoys O(1k2) iteration complexity. Moreover, to make the algorithm more efficient and further reduce its computation cost, based on the mesh-independence of ABCD method, we propose an FE-dABCD algorithm with a warm-start strategy (wFE-dABCD). Some numerical experiments are done and the numerical results show the efficiency of the FE-dABCD algorithm and wFE-dABCD algorithm.

Improving the Performance of Optimization Algorithms Using the Adaptive Fixed-Time Scheme and Reset Scheme

Optimization algorithms have now played an important role in many fields, and the issue of how to design high-efficiency algorithms has gained increasing attention, for which it has been shown that advanced control theories could be helpful. In this paper, the fixed-time scheme and reset scheme are introduced to design high-efficiency gradient descent methods for unconstrained convex optimization problems. At first, a general reset framework for existing accelerated gradient descent methods is given based on the systematic representation, with which both convergence speed and stability are significantly improved. Then, the design of a novel adaptive fixed-time gradient descent, which has fewer tuning parameters and maintains better robustness to initial conditions, is presented. However, its discrete form introduces undesirable overshoot and easily leads to instability, and the reset scheme is then applied to overcome the drawbacks. The linear convergence and better stability of the proposed algorithms are theoretically proven, and several dedicated simulation examples are finally given to validate the effectiveness.

Distributed convex optimization based on zero‐gradient‐sum algorithm under switching topology

AbstractThis paper designs a finite‐time convergence protocol and an event‐triggered control protocol based on Zero‐Gradient‐Sum (ZGS) algorithm under stochastic switching undirected topology, respectively, which greatly expands the theory of continuous‐time distributed optimization algorithms. With finite‐time stability and Lyapunov stability analysis, it is illustrated that the proposed method can finite‐time converge to the optimal solution of distributed unconstrained convex optimization problem and overcome the disturbances of the switching communication networks. In addition, the event‐triggered mechanism can effectively reduce the network burden and communication cost as well as avoid Zeno behaviour. Finally, two numerical simulations verify the advantages and effectiveness of these methods.

Riemannian representation learning for multi-source domain adaptation

Multi-Source Domain Adaptation (MSDA) aims at training a classification model that achieves small target error, by leveraging labeled data from multiple source domains and unlabeled data from a target domain. The source and target domains are described by related but different joint distributions, which lie on a Riemannian manifold named the statistical manifold. In this paper, we characterize the joint distribution difference by the Hellinger distance, which bears strong connection to the Riemannian metric defined on the statistical manifold. We show that the target error of a neural network classification model is upper bounded by the average source error of the model and the average Hellinger distance, i.e., the average of multiple Hellinger distances between the source and target joint distributions in the network representation space. Motivated by the error bound, we introduce Riemannian Representation Learning (RRL): An approach that trains the network model by minimizing (i) the average empirical Hellinger distance with respect to the representation function, and (ii) the average empirical source error with respect to the network model. Specifically, we derive the average empirical Hellinger distance by constructing and solving unconstrained convex optimization problems whose global optimal solutions are easy to find. With the network model trained, we expect it to achieve small error in the target domain. Our experimental results on several image datasets demonstrate that the proposed RRL approach is statistically better than the comparison methods.

Inertial primal-dual dynamics with damping and scaling for linearly constrained convex optimization problems

We propose an inertial primal-dual dynamic with damping and scaling coefficients, which involves inertial terms both for primal and dual variables, for a linearly constrained convex optimization problem in a Hilbert setting. With different choices of damping and scaling coefficients, by a Lyapunov analysis approach, we investigate the asymptotic properties of the dynamic and prove its fast convergence results. Our results can be viewed as extensions of the existing ones on inertial dynamical systems for the unconstrained convex optimization problem to the primal-dual one for the linearly constrained convex optimization problem.

A distributed continuous-time modified Newton–Raphson algorithm

We propose a continuous-time second-order optimization algorithm for solving unconstrained convex optimization problems with bounded Hessian. We show that this alternative algorithm has a comparable convergence rate to that of the continuous-time Newton–Raphson method, however structurally, it is amenable to a more efficient distributed implementation. We present a distributed implementation of our proposed optimization algorithm and prove its convergence via Lyapunov analysis. A numerical example demonstrates our results.

Bertram’s pairs trading strategy with bounded risk

Finding Bertram's optimal trading strategy for a pair of cointegrated assets following the Ornstein--Uhlenbeck price difference process can be formulated as an unconstrained convex optimization problem for maximization of expected profit per unit of time. This model is generalized to the form where the riskiness of profit, measured by its per-time-unit volatility, is controlled (e.g. in case of existence of limits on riskiness of trading strategies imposed by regulatory bodies). The resulting optimization problem need not be convex. In spite of this undesirable fact, it is demonstrated that the problem is still efficiently solvable. In addition, the problem that parameters of the price difference process are never known exactly and are imprecisely estimated from an observed finite sample is investigated (recalling that this problem is critical for practice). It is shown how the imprecision affects the optimal trading strategy by quantification of the loss caused by the imprecise estimate compared to a theoretical trader knowing the parameters exactly. The main results focus on the geometric and optimization-theoretic viewpoint of the risk-bounded trading strategy and the imprecision resulting from the statistical estimates.

A New Inexact Line Search Method for Convex Optimization Problems

In general one can say that line search procedure for the steplength and search direction are two important elements of a line search algorithm. The line search procedure requires much attention because of its far implications on the robustness and efficiency of the algorithm. The purpose of this paper is to propose a simple yet effective line search strategy in solving unconstrained convex optimization problems. This line search procedure does not require the evaluation of the objective function. Instead, it forces reduction in gradient norm on each direction. Hence it is suitable for problems when function evaluation is very costly. To illustrate the effectiveness of our line search procedure, we employ this procedure together with the symmetric rank one quasi-Newton update and test it against the same quasi-Newton method with the well-known Armijo line search. Numerical results on a set of standard unconstrained optimization problems showed that the proposed procedure is superior to the Armijo line search.

New inertial proximal gradient methods for unconstrained convex optimization problems

The proximal gradient method is a highly powerful tool for solving the composite convex optimization problem. In this paper, firstly, we propose inexact inertial acceleration methods based on the viscosity approximation and proximal scaled gradient algorithm to accelerate the convergence of the algorithm. Under reasonable parameters, we prove that our algorithms strongly converge to some solution of the problem, which is the unique solution of a variational inequality problem. Secondly, we propose an inexact alternated inertial proximal point algorithm. Under suitable conditions, the weak convergence theorem is proved. Finally, numerical results illustrate the performances of our algorithms and present a comparison with related algorithms. Our results improve and extend the corresponding results reported by many authors recently.

SurvLIME: A method for explaining machine learning survival models

A new method called SurvLIME for explaining machine learning survival models is proposed. It can be viewed as an extension or modification of the well-known method LIME. The main idea behind the proposed method is to apply the Cox proportional hazards model to approximate the survival model at the local area around a test example. The Cox model is used because it considers a linear combination of the example covariates such that coefficients of the covariates can be regarded as quantitative impacts on the prediction. Another idea is to approximate cumulative hazard functions of the explained model and the Cox model by using a set of perturbed points in a local area around the point of interest. The method is reduced to solving an unconstrained convex optimization problem. A lot of numerical experiments demonstrate the SurvLIME efficiency.

On Stochastic and Deterministic Quasi-Newton Methods for Nonstrongly Convex Optimization: Asymptotic Convergence and Rate Analysis

Motivated by applications arising from large-scale optimization and machine learning, we consider stochastic quasi-Newton (SQN) methods for solving unconstrained convex optimization problems. Much ...

Joint Texture/Depth Power Allocation for 3-D Video SoftCast

Recently, a novel uncoded (pseudoanalog) scheme called SoftCast is proposed for wireless video transmission, which eliminates the cliff effect of the state-of-the-art source-channel coding based schemes and achieves linear quality transition within a wide range of channel signal-to-noise ratio. Therefore, SoftCast-like uncoded and hybrid transmission has become an attractive research issue for natural 2-D video. However, very few studies focus on the SoftCast-based wireless transmission of the 3-D video (3DV) currently. One critical issue of 3DV SoftCast is how to allocate the limited power budget of the transmitter to the texture videos and depth maps of the 3DV to achieve the optimal overall quality on the receiver side, including the transmission quality of the reference views and the synthesis quality of the virtual views. This paper attempts to solve the optimal joint power allocation problem in an efficient way. First, we formulate the target problem as a constrained power-distortion optimization (PDO) problem mathematically. Then, each part of the distortion is analyzed and formulated in a closed form. Finally, the PDO problem is mapped to an unconstrained convex optimization problem and solved by the Lagrangian multiplier method. Simulation results demonstrate that the performance of the proposed method is close to that of the full search method, which can provide the best performance theoretically. Nevertheless, the complexity of the proposed method is negligible compared with that of the full search method. In addition, as compared with the fixed ratio (e.g., 1:1) power allocation between texture and depth, the proposed method can achieve a PNSR gain up to 1.8 dB.

Bounded perturbation resilience and superiorization techniques for a modified proximal gradient method

ABSTRACTIn this article, we combine the viscosity approximation method and proximal operator to propose modified proximal gradient methods for solving the unconstrained convex optimization problems. The bounded perturbation resilience of these methods is investigated in the general Hilbert space H. Under reasonable parameter conditions, we prove that our algorithms strongly converge to the unique solution of a variational inequality problem. Our results improve and extend the corresponding results reported by many authors recently.

A new infeasible proximal bundle algorithm for nonsmooth nonconvex constrained optimization

Proximal bundle method has usually been presented for unconstrained convex optimization problems. In this paper, we develop an infeasible proximal bundle method for nonsmooth nonconvex constrained optimization problems. Using the improvement function we transform the problem into an unconstrained one and then we build a cutting plane model. The resulting algorithm allows effective control of the size of quadratic programming subproblems via the aggregation techniques. The novelty in our approach is that the objective and constraint functions can be any arbitrary (regular) locally Lipschitz functions. In addition the global convergence, starting from any point, is proved in the sense that every accumulation point of the iterative sequence is stationary for the improvement function. At the end, some encouraging numerical results with a MATLAB implementation are also reported.

Energy-Efficient Noncooperative Power Control in Small-Cell Networks

In this paper, energy efficient power control for small cells underlaying a macro cellular network is investigated. We formulate the power control problem in self-organizing small cell networks as a non-cooperative game, and propose a distributed energy efficient power control scheme, which allows the small base stations (SBSs) to take individual decisions for attaining the Nash equilibrium (NE) with minimum information exchange. Specially, in the non-cooperative power control game, a non-convex optimization problem is formulated for each SBS to maximize their energy efficiency (EE). By exploiting the properties of parameter-free fractional programming and the concept of perspective function, the non-convex optimization problem for each SBS is transformed into an equivalent constrained convex optimization problem. Then, the constrained convex optimization problem is converted into an unconstrained convex optimization problem by exploiting the mixed penalty function method. The inequality constraints are eliminated by introducing the logarithmic barrier functions and the equality constraint is eliminated by introducing the quadratic penalty function. We also theoretically show the existence and the uniqueness of the NE in the non-cooperative power control game. Simulation results show remarkable improvements in terms of EE by using the proposed scheme.

Closedness Type Regularity Conditions in Convex Optimization and Beyond

The closedness type regularity conditions have proven during the last decade to be viable alternatives to their more restrictive interiority type counterparts, in both convex optimization and different areas where it was successfully applied. In this review article we de- and reconstruct some closedness type regularity conditions formulated by means of epigraphs and subdifferentials, respectively, for general optimization problems in order to stress that they arise naturally when dealing with such problems. The results are then specialized for constrained and unconstrained convex optimization problems. We also hint towards other classes of optimization problems where closedness type regularity conditions were successfully employed and discuss other possible applications of them.

A Class of Prediction-Correction Methods for Time-Varying Convex Optimization

This paper considers unconstrained convex optimization problems with time-varying objective functions. We propose algorithms with a discrete time-sampling scheme to find and track the solution trajectory based on prediction and correction steps, while sampling the problem data at a constant rate of $1/h$, where $h$ is the length of the sampling interval. The prediction step is derived by analyzing the iso-residual dynamics of the optimality conditions. The correction step adjusts for the distance between the current prediction and the optimizer at each time step, and consists either of one or multiple gradient steps or Newton steps, which respectively correspond to the gradient trajectory tracking (GTT) or Newton trajectory tracking (NTT) algorithms. Under suitable conditions, we establish that the asymptotic error incurred by both proposed methods behaves as $O(h^2)$, and in some cases as $O(h^4)$, which outperforms the state-of-the-art error bound of $O(h)$ for correction-only methods in the gradient-correction step. Moreover, when the characteristics of the objective function variation are not available, we propose approximate gradient and Newton tracking algorithms (AGT and ANT, respectively) that still attain these asymptotical error bounds. Numerical simulations demonstrate the practical utility of the proposed methods and that they improve upon existing techniques by several orders of magnitude.

Acceleration of block coordinate descent method achieves the $\bm{O(\frac{1}{k^2})}$ rate of convergence for a convex function with block coordinate strong convexity

Block coordinate descent method enjoys a long history and is a classic optimization method which has been extensively applied because it possesses some advantages, such as the simplicity, speed, and stability, etc. We prove that it can be accelerated and obtain an $O(\\frac{1}{k^2})$ rate of convergence for unconstrained convex optimization problem under a relatively mild assumption that the objective function is a convex function with block coordinate strong convexity.

General viscosity iterative approximation for solving unconstrained convex optimization problems

In this paper, we combine a sequence of contractive mappings $\{h_{n}\}$ with the proximal operator and propose a generalized viscosity approximation method for solving the unconstrained convex optimization problems in a real Hilbert space H. We show that, under reasonable parameter conditions, our algorithm strongly converges to the unique solution of a variational inequality problem. Our result presented in the paper improves and extends the corresponding results reported by many authors recently.

Sparsity-based correction of exponential artifacts

This paper describes an exponential transient excision algorithm (ETEA). In biomedical time series analysis, e.g., in vivo neural recording and electrocorticography (ECoG), some measurement artifacts take the form of piecewise exponential transients. The proposed method is formulated as an unconstrained convex optimization problem, regularized by smoothed ℓ1-norm penalty function, which can be solved by majorization–minimization (MM) method. With a slight modification of the regularizer, ETEA can also suppress more irregular piecewise smooth artifacts, especially, ocular artifacts (OA) in electroencephalography (EEG) data. Examples of synthetic signal, EEG data, and ECoG data are presented to illustrate the proposed algorithms.