A conjugate gradient algorithmic framework for unconstrained optimization with applications: Convergence and rate analyses

This paper presents a convergence analysis and experimental validation of an iterative design optimization framework that fuses numerical simulations with experiments. At every iteration, a G-optimal design generates a set of simulations and experiments that are used to characterize response surfaces. A subset of the experiments termed as the training points are used to fit a combined numerical/experimental response. This numerical response is obtained as a result of numerical model correction via experiments. The quality of fit for this combined response is evaluated using the remaining validation points. Based on the quality of fit, the feasible design space is reduced for a given confidence interval using hypothesis testing. A convergence analysis of the framework quantifies the closeness of the corrected numerical model to the true system as a function of response estimation error. This design optimization framework, along with the convergence result, is validated through an airborne wind energy (AWE) application using a lab-scale water channel setup. The quality of flight is greatly improved by optimizing the center of mass location, pitch angle set point, horizontal and vertical stabilizer areas using an effective experimental infusion as compared to a pure numerically optimized design.

Convergence Conditions for Ascent Methods

Synthetic aperture radar (SAR) frequently suffers from radio frequency interference (RFI) due to the simultaneous presence of numerous wireless communication signals. Recently, the narrowband RFI is found to possess the low-rank property benefiting from stable frequency occupancy, hence the reconsideration of RFI suppression as a joint sparse and low-rank optimization problem. The existing methods either use the non-sparse useful signal itself as the sparse regularizer, or employ the nuclear norm to approximate the rank function, which punishes all singular values with the same penalty via singular value thresholding (SVT), resulting in the improper punishment problem. Hence, both are consequentially subject to performance limitation. In this paper, a novel dictionary-based nonconvex low-rank minimization (DNLRM) optimization framework is proposed for RFI suppression, which concurrently considers the improvements for both the sparse regularizer and the low-rank regularizer. For the former, an over-completed dictionary is constructed, for which the sparse coefficient acts as the sparse regularizer. For the latter, the rank function is more accurately approximated by innovatively introducing the nonconvex function, for which the supergradient is synchronously used to generate the weighted penalty, thus solving the improper punishment problem. The derivation of the closed-form solution and the convergence analysis are described in detail. Additionally, the adaptive selection scheme for the model parameter is uniquely proposed for further ensuring the practicality of the DNLRM framework. The superiority of the proposed method is demonstrated via not only the RFI-free real SAR data combined with the measured RFI, but the RFI-contaminated real SAR data.

Vertical federated learning (FL) is a collaborative machine learning framework that enables devices to learn a global model from the feature-partition datasets without sharing local raw data. However, as the number of the local intermediate outputs is proportional to the training samples, it is critical to develop communication-efficient techniques for wireless vertical FL to support high-dimensional model aggregation with full device participation. In this paper, we propose a novel cloud radio access network (Cloud-RAN) based vertical FL system to enable fast and accurate model aggregation by leveraging over-the-air computation (AirComp) and alleviating communication straggler issue with cooperative model aggregation among geographically distributed edge servers. However, the model aggregation error caused by AirComp and quantization errors caused by the limited fronthaul capacity degrade the learning performance for vertical FL. To address these issues, we characterize the convergence behavior of the vertical FL algorithm considering both uplink and downlink transmissions. To improve the learning performance, we establish a system optimization framework by joint transceiver and fronthaul quantization design, for which successive convex approximation and alternate convex search based system optimization algorithms are developed. We conduct extensive simulations to demonstrate the effectiveness of the proposed system architecture and optimization framework for vertical FL.

Robustness, efficiency, and accuracy are qualities that excellent algorithms should have. Due to the simplicity and minimal storage requirements, Conjugate gradient (CG) methods are useful for solving large-scale, unconstrained optimization problems. Despite that, it has a few drawbacks. Even if they have high numerical performance, certain approaches lack global convergence properties; therefore, the solutions might not be the most accurate. Various methods and modifications have been done. Some formulations would be difficult to comprehend and apply, and would lead to high CPU time. The proving process would also be impacted by the complex formulations. Over the past years, researchers have developed various globally convergent CG methods, but with a complicated algorithm, it rather hampered the implementation. Therefore, new CG methods with a derivate-free approach that have good convergence properties and outperform the existing CG coefficients in terms of number of iterations (NOI), number of function evaluations (NFE), and central processing time per unit (CPU time) are proposed. The proposed method will employ a non-derivative approach. This approach should make the algorithm’s processing time as minimal as possible. The comparison for derivative-free tools among the existed derivative-free CG. The proposed approach was chosen because it combines the strengths of the CG method with derivate-free optimization to optimize complicated objective functions without explicitly computing derivatives. This paper will show the derivate-free CG, which was proven to fulfil both convergence analysis and numerical performance.

Random feature kernel least mean square RF KLMS) algorithms, like the random Fourier feature KLMS (RFF-KLMS), can effectively reduce the computation and storage burdens of the KLMS algorithm in the process of update. However, little work has been done to perform the convergence analysis for such algorithms. To this end, in this paper, we present a unified framework of RF-KLMS algorithms, and based on which, a universal model for convergence analysis is given. As two examples, the RFF-KLMS and the random Gaussian feature KLMS (RGF-KLMS) are discussed detailedly. Simulations demonstrate the validity of the theoretical analysis. Index Terms–Kernel least mean square, random feature, universal model, convergence analysis

Robust best approximation with interpolation constraints under ellipsoidal uncertainty: Strong duality and nonsmooth Newton methods

We present two frameworks for designing random search methods for discrete simulation optimization. One of our frameworks is very broad (in that it includes many random search methods), whereas the other one considers a special class of random search methods called point‐based methods, that move iteratively between points within the feasible region. Our frameworks involve averaging, in that all decisions that require estimates of the objective function values at various feasible solutions are based on the averages of all observations collected at these solutions so far. Also, the methods are adaptive in that they can use information gathered in previous iterations to decide how simulation effort is expended in the current iteration. We show that the methods within our frameworks are almost surely globally convergent under mild conditions. Thus, the generality of our frameworks and associated convergence guarantees makes the frameworks useful to algorithm developers wishing to design efficient and rigorous procedures for simulation optimization. We also present two variants of the simulated annealing (SA) algorithm and provide their convergence analysis as example application of our point‐based framework. Finally, we provide numerical results that demonstrate the empirical effectiveness of averaging and adaptivity in the context of SA. © 2012 Wiley Periodicals, Inc. Naval Research Logistics, 2012

A conjugate gradient (CG)-type algorithm CG_Plan is introduced for calculating an approximate solution of Newton's equation within large-scale optimization frameworks. The approximate solution must satisfy suitable properties to ensure global convergence. In practice, the CG algorithm is widely used, but it is not suitable when the Hessian matrix is indefinite, as it can stop prematurely. CG_Plan is a symmetric variant of the composite step Bi-CG method of Bank and Chan, suitably adapted for optimization problems. It is an alternative to CG that copes with the indefinite case. This work was published by MIUR, FIRB Research Program Large Scale Nonlinear Optimization, Rome, Italy We show convergence for CG_Plan, then prove that the practical implementation always provides a gradient related direction within a truncated Newton method (algorithm TN_Plan). Some preliminary numerical results support the theory.

We develop a discrete-time optimal control framework for systems evolving on Lie groups. Our article generalizes the original differential dynamic programming method, by employing a coordinate-free, Lie-theoretic approach for its derivation. A key element lies, specifically, in the use of quadratic expansion schemes for cost functions and dynamics defined on Lie groups. The obtained algorithm iteratively optimizes local approximations of the control problem, until reaching a (sub)optimal solution. On the theoretical side, we also study the conditions under which convergence is attained. Details about the behavior and implementation of our method are provided through a simulated example on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$TSO(3)$</tex-math></inline-formula> .

In this work, we propose a stochastic algorithm for solving $$ \mathcal{N}{\wp } - hard $$ combinatorial optimization problems. The procedure is formulated within the Ant Colony Optimization (ACO) framework, and extends the so-called Graph-based Ant System with time-dependent evaporation factor, (GBAS/tdev) studied in Gutjahr (2002). In particular, we consider an ACO search procedure which also takes into account the objective function value. We provide a rigorous theoretical study on the convergence of the proposed algorithm. Further, for a toy example, we compare by simulation the rate of convergence of the proposed algorithm with those from the Random Search (RS) and from the corresponding procedure in Gutjahr (2002).

As a critical infrastructure, the modern electrical network is faced with various types of threats, such as accidental natural disaster attacks and deliberate artificial attacks, thus the power system fortification has attracted great concerns in the community of academic, industry, and military. Nevertheless, the attacker is commonly assumed to be capable of accessing all information in the literature (e.g., network configuration and defensive plan are explicitly provided to the attacker), which might always be the truth since the grid data access permission is usually restricted. In this paper, the information asymmetry between defender and attacker is investigated, leading to an optimal deception strategy problem for power system fortification. Both the proposed deception and traditional protection strategies are formulated as a tri-level mixed-integer linear programming (MILP) problem and solved via two-stage robust optimization (RO) framework and the column-and-constraint generation (CCG) algorithm. Comprehensive case studies on the 6-bus system and IEEE 57-bus system are implemented to reveal the difference between these two strategies and identify the significance of information deception. Numerical results indicate that deception strategy is superior to protection strategy. In addition, detailed discussions on the performance evaluation and convergence analysis are presented as well.

Research on unconstrained and constrained optimization has been separately conducted for a long time. Normally, constrained optimization, especially the nonlinear equality constrained optimization problem is much more difficult to be researched. People became to recognize that both unconstrained optimization and constrained optimization are actually the optimization problem on Riemannian manifolds since 1982. Therefore, unconstrained optimization methods can be extended directly to constrained optimization cases under the condition of Riemannian manifolds. Recently, more and more scholars realize the importance of the optimization methods on Riemannian manifold and it has become a new direction of research on nonlinear optimization. This article reviews the development and new applications of the optimization problem on Riemannian manifolds and adduces correlative arithmetic and some examples in addition.

The conjugate gradient is a useful tool in solving large- and small-scale unconstrained optimization problems. In addition, the conjugate gradient method can be applied in many fields, such as engineering, medical research, and computer science. In this paper, a convex combination of two different search directions is proposed. The new combination satisfies the sufficient descent condition and the convergence analysis. Moreover, a new conjugate gradient formula is proposed. The new formula satisfies the convergence properties with the descent property related to Hestenes–Stiefel conjugate gradient formula. The numerical results show that the new search direction outperforms both two search directions, making it convex between them. The numerical result includes the number of iterations, function evaluations, and central processing unit time. Finally, we present some examples about image restoration as an application of the proposed conjugate gradient method.

We consider an iterative computation of negative curvature directions, in large-scale unconstrained optimization frameworks, needed for ensuring the convergence toward stationary points which satisfy second-order necessary optimality conditions. We show that to the latter purpose, we can fruitfully couple the conjugate gradient (CG) method with a recently introduced approach involving the use of the numeral called Grossone. In particular, recalling that in principle the CG method is well posed only when solving positive definite linear systems, our proposal exploits the use of grossone to enhance the performance of the CG, allowing the computation of negative curvature directions in the indefinite case, too. Our overall method could be used to significantly generalize the theory in state-of-the-art literature. Moreover, it straightforwardly allows the solution of Newton’s equation in optimization frameworks, even in nonconvex problems. We remark that our iterative procedure to compute a negative curvature direction does not require the storage of any matrix, simply needing to store a couple of vectors. This definitely represents an advance with respect to current results in the literature.