Augmented Lagrangian Function Research Articles

A Three-Block Inexact Heterogeneous Alternating Direction Method of Multipliers for Elliptic PDE-Constrained Optimization Problems with a Control Gradient Penalty Term

Optimization problems with PDE constraints are widely used in engineering and technical fields. In some practical applications, it is necessary to smooth the control variables and suppress their large fluctuations, especially at the boundary. Therefore, we propose an elliptic PDE-constrained optimization model with a control gradient penalty term. However, introducing this penalty term increases the complexity and difficulty of the problems. To solve the problems numerically, we adopt the strategy of “First discretize, then optimize”. First, the finite element method is employed to discretize the optimization problems. Then, a heterogeneous strategy is introduced to formulate the augmented Lagrangian function for the subproblems. Subsequently, we propose a three-block inexact heterogeneous alternating direction method of multipliers (three-block ihADMM). Theoretically, we provide a global convergence analysis of the three-block ihADMM algorithm and discuss the iteration complexity results. Numerical results are provided to demonstrate the efficiency of the proposed algorithm.

Convergence of Peaceman-Rachford splitting method with Bregman distance for three-block nonconvex nonseparable optimization

Abstract It is of strong theoretical significance and application prospects to explore three-block nonconvex optimization with nonseparable structure, which are often modeled for many problems in machine learning, statistics, and image and signal processing. In this article, by combining the Bregman distance and Peaceman-Rachford splitting method, we propose a novel three-block Bregman Peaceman-Rachford splitting method (3-BPRSM). Under a general assumption, global convergence is presented via optimality conditions. Furthermore, we prove strong convergence when the augmented Lagrange function satisfies Kurdyka-Łojasiewicz property. In addition, if the association function possessing the Kurdyka-Łojasiewicz property exhibits a distinctive structure, then linear and sublinear convergence rate of 3-BPRSM can be guaranteed. Finally, a preliminary numerical experiment demonstrates the effectiveness.

Global convergence of dislocation hyperbolic augmented Lagrangian algorithm for nonconvex optimization

In this work, we present a new augmented Lagrangian type algorithm based on the dislocation hyperbolic augmented Lagrangian function (DHALF). This algorithm is called dislocation hyperbolic augmented Lagrangian algorithm (DHALA). We ensure that DHALA converges to a global solution for the inequality constrained nonconvex optimization problem. We present DHALA considering the safeguards technique and ensure the convergence of this algorithm. Finally, we present a computational experiment to show the performance of our proposed algorithm.

A Collaborative Neurodynamic Optimization Approach to Distributed Chiller Loading.

In this article, we present a collaborative neurodynamic optimization approach to distributed chiller loading in the presence of nonconvex power consumption functions and binary variables associated with cardinality constraints. We formulate a cardinality-constrained distributed optimization problem with nonconvex objective functions and discrete feasible regions, based on an augmented Lagrangian function. To overcome the difficulty caused by the nonconvexity in the formulated distributed optimization problem, we develop a collaborative neurodynamic optimization method based on multiple coupled recurrent neural networks reinitialized repeatedly using a meta-heuristic rule. We elaborate on experimental results based on two multi-chiller systems with the parameters from the chiller manufacturers to demonstrate the efficacy of the proposed approach in comparison to several baselines.

Predictive Control via Augmented Lagrange Function for Autonomous Vehicles Trajectory Tracking with Dynamic Quantization

Predictive Control via Augmented Lagrange Function for Autonomous Vehicles Trajectory Tracking with Dynamic Quantization

The Alternating Direction Method of Multipliers for Sufficient Dimension Reduction

The minimum average variance estimation (MAVE) method has proven to be an effective approach to sufficient dimension reduction. In this study, we apply the computationally efficient optimization algorithm named alternating direction method of multipliers (ADMM) to a particular approach (MAVE or minimum average variance estimation) to the problem of sufficient dimension reduction (SDR). Under some assumptions, we prove that the iterative sequence generated by ADMM converges to some point of the associated augmented Lagrangian function. Moreover, that point is stationary. It also presents some numerical simulations on synthetic data to demonstrate the computational efficiency of the algorithm.

A Collaborative Neurodynamic Optimization Approach to Distributed Nash-Equilibrium Seeking in Multicluster Games With Nonconvex Functions.

In this article, we propose a collaborative neurodynamic optimization (CNO) method for the distributed seeking of generalized Nash equilibriums (GNEs) in multicluster games with nonconvex functions. Based on an augmented Lagrangian function, we develop a projection neural network for the local search of GNEs, and its convergence to a local GNE is proven. We formulate a global optimization problem to which a global optimal solution is a high-quality local GNE, and we adopt a CNO approach consisting of multiple recurrent neural networks for scattering searches and a metaheuristic rule for reinitializing states. We elaborate on an example of a price-bidding problem in an electricity market to demonstrate the viability of the proposed approach.

A Reentry Trajectory Planning Algorithm via Pseudo-Spectral Convexification and Method of Multipliers

The reentry trajectory planning problem of hypersonic vehicles is generally a continuous and nonconvex optimization problem, and it constitutes a critical challenge within the field of aerospace engineering. In this paper, an improved sequential convexification algorithm is proposed to solve it and achieve online trajectory planning. In the proposed algorithm, the Chebyshev pseudo-spectral method with high-accuracy approximation performance is first employed to discretize the continuous dynamic equations. Subsequently, based on the multipliers and linearization methods, the original nonconvex trajectory planning problem is transformed into a series of relaxed convex subproblems in the form of an augmented Lagrange function. Then, the interior point method is utilized to iteratively solve the relaxed convex subproblem until the expected convergence precision is achieved. The convex-optimization-based and multipliers methods guarantee the promotion of fast convergence precision, making it suitable for online trajectory planning applications. Finally, numerical simulations are conducted to verify the performance of the proposed algorithm. The simulation results show that the algorithm possesses better convergence performance, and the solution time can reach the level of seconds, which is more than 97% less than nonlinear programming algorithms, such as the sequential quadratic programming algorithm.

High-Dimensional Multivariate Linear Regression with Weighted Nuclear Norm Regularization

We consider a low-rank matrix estimation problem when the data is assumed to be generated from the multivariate linear regression model. To induce the low-rank coefficient matrix, we employ the weighted nuclear norm (WNN) penalty defined as the weighted sum of the singular values of the matrix. The weights are set in a nondecreasing order, which yields the non-convexity of the WNN objective function in the parameter space. Although the objective function has been widely applied, studies on the estimation properties of its resulting estimator are limited. We propose an efficient algorithm under the framework of the alternative directional method of multipliers (ADMM) to estimate the coefficient matrix. The estimator from the suggested algorithm converges to a stationary point of an augmented Lagrangian function. Under the orthogonal design setting, the effects of the weights for estimating the singular values of the ground-truth coefficient matrix are derived. Under the Gaussian design setting, a minimax convergence rate on the estimation error is derived. We also propose a generalized cross-validation (GCV) criterion for selecting the tuning parameter and an iterative algorithm for updating the weights. Simulations and a real data analysis demonstrate the competitive performance of our new method. Supplementary materials for this article are available online.

Stochastic nested primal-dual method for nonconvex constrained composition optimization

In this paper we study the nonconvex constrained composition optimization, in which the objective contains a composition of two expected-value functions whose accurate information is normally expensive to calculate. We propose a STochastic nEsted Primal-dual (STEP) method for such problems. In each iteration, with an auxiliary variable introduced to track the inner layer function values we compute stochastic gradients of the nested function using a subsampling strategy. To alleviate difficulties caused by possibly nonconvex constraints, we construct a stochastic approximation to the linearized augmented Lagrangian function to update the primal variable, which further motivates to update the dual variable in a weighted-average way. Moreover, to better understand the asymptotic dynamics of the update schemes we consider a deterministic continuous-time system from the perspective of ordinary differential equation (ODE). We analyze the Karush-Kuhn-Tucker measure at the output by the STEP method with constant parameters and establish its iteration and sample complexities to find an ϵ \epsilon -stationary point, ensuring that expected stationarity, feasibility as well as complementary slackness are below accuracy ϵ \epsilon . To leverage the benefit of the (near) initial feasibility in the STEP method, we propose a two-stage framework incorporating a feasibility-seeking phase, aiming to locate a nearly feasible initial point. Moreover, to enhance the adaptivity of the STEP algorithm, we propose an adaptive variant by adaptively adjusting its parameters, along with a complexity analysis. Numerical results on a risk-averse portfolio optimization problem and an orthogonal nonnegative matrix decomposition reveal the effectiveness of the proposed algorithms.

An improved alternating direction method of multipliers for solving deterministic user equilibrium

An improved alternating direction method of multipliers (iADMM) algorithm is proposed for solving the deterministic user equilibrium (DUE) problem. The iADMM algorithm introduces a key improvement wherein the primal variables can be updated sequentially, possibly multiple times, before updating the dual variables. Subsequently, a sensitivity analysis method is applied to optimise the update frequency of primal variable, and two indices are introduced to evaluate the performance of various policies. Additionally, eliminating equality constraints through the augmented Lagrangian function makes it challenging to completely satisfy flow conservation conditions, resulting in the presence of primal residual. To address this issue, two schemes are proposed to handle the primal residual, involving modifications to the primal variables based on iterative messages. Numerical experiments highlight that the convergence of ADMM algorithm is significantly influenced by the update frequency of primal variable, and addressing the primal residual after each cycle enhances the convergence of ADMM algorithm.

Incorporating STATCOM into a Hydro-Thermal Optimal Power Flow Algorithm

This paper presents the results of the inclusion of Synchronous Static Compensator (STATCOM) power flow models into a hydro-thermal optimal power flow (HTOPF) algorithm. STATCOM basically introduces the voltage magnitude and phase angle of the source converter into the algorithm. For each incorporated STATCOM, an augmented Lagrangian function was formed. The first and second derivatives of these functions were added to the gradient vector and Hessian matrix of an existing algorithm. The modified algorithm was implemented using MATLAB R2018a and tested on 30 and 57 bus systems. Two and three STATCOMs were, respectively, tested on 30 and 57 bus systems. The results obtained showed that one of the STATCOMs used on 30-bus system injected reactive power that ranges from 8.99 MVAR to 28.22 MVAR while the other one injected reactive power in the range of 8.20 MVAR to 33.20 MVAR. For the STATCOMs placed on the 57-bus system, the range of reactive power absorption by the one placed at bus 5 is 7.83 MVAR to 22.86 MVAR while the one at bus 55 absorbed from 5.06 MVAR to 12.92 MVAR. The STATCOM’s reactive power injection at bus 31 ranges from 4.63 MVAR to 10.35 MVAR. All the lower and higher values were obtained at hours 3 and 7, respectively. While the STATCOMs significantly improved the systems’ voltage profile, the impacts of STATCOM on the total systems daily energy loss, daily energy generations (from both plants), daily fuel cost and hydro plant water worth are insignificant.

Adaptive fixed-time proximal gradient method for non-smooth optimization: the fractional approach

Non-smooth optimization has now played an important role in many fields, such as l1-regulation problem and sparse recovery problem. In this paper, a novel fractional fixed-time proximal gradient method is proposed for solving the composite optimization problems with non-smooth terms. At first, according to the distribution of zeroes of Mittag-Leffler functions, a novel fixed-time convergence theory with a fractional adaptive gain is proposed. Based on the optimal condition analysis of the transformed problem using the augmented Lagrangian function, a novel fixed-time proximal gradient method is then proposed and the fixed-time convergence property is rigorously proven. Furthermore, the proposed algorithm is applied to solve the sparse recovery problem in compressive sensing, and the restricted isometry condition required for the application of the algorithm is analyzed to achieve fixed-time recovery of the signal. Finally, simulation examples are provided to validate all the results.

A Customized Augmented Lagrangian Method for Block-Structured Integer Programming.

Integer programming with block structures has received considerable attention recently and is widely used in many practical applications such as train timetabling and vehicle routing problems. It is known to be NP-hard due to the presence of integer variables. We define a novel augmented Lagrangian function by directly penalizing the inequality constraints and establish the strong duality between the primal problem and the augmented Lagrangian dual problem. Then, a customized augmented Lagrangian method is proposed to address the block-structures. In particular, the minimization of the augmented Lagrangian function is decomposed into multiple subproblems by decoupling the linking constraints and these subproblems can be efficiently solved using the block coordinate descent method. We also establish the convergence property of the proposed method. To make the algorithm more practical, we further introduce several refinement techniques to identify high-quality feasible solutions. Numerical experiments on a few interesting scenarios show that our proposed algorithm often achieves a satisfactory solution and is quite effective.

A hybrid approach for unit commitment with splitting technique and local search

In the modern electricity market, solving the unit commitment (UC) problem becomes more challenging due to its large number of binary variables and coupling constraints. Therefore, the goal of this paper is to obtain nearly optimal solutions for such complex UC problems within specific time limits. To achieve this, a hybrid approach is presented in this paper, which incorporates a splitting technique and a local search for UC. The hybrid approach involves two steps. In the first step, the augmented Lagrange function (ALF) of UC is decomposed using a splitting technique based on the alternating direction method of multipliers (ADMM). The resulting sub-problems can be effectively solved in a distributed manner by utilizing Lagrange duality theory and binary variable characteristics. Additionally, the penalty parameters of the ALF are updated effectively with the sub-gradient method to diminish the influence of the initial penalty parameters on the algorithm. In the second step, the appropriate relaxation unit is selected using the binary variable characteristics in the current solution of ADMM. Further, the high-quality feasibility of the final solution is ensured through local search. The hybrid approach utilizes ADMM to quickly fix most binary variables in the UC model and then leverages the CPLEX solver to solve the mixed-integer linear programming (MILP) formulation of UC by fixing a large number of binary variables. Consequently, the CPLEX solver needs to branch fewer binary variables, reducing the local search time. The hybrid approach is applied to solve several power systems consisting of up to 1000 units. Compared with the CPLEX solver, the solution of the hybrid approach is found to be stable, and the relative gap of the solution is less than 1 %, demonstrating the effectiveness and performance of the proposed approach while meeting the time limit requirements of the power industry dispatching.

Risk-averse stochastic capacity planning and P2P trading collaborative optimization for multi-energy microgrids considering carbon emission limitations: An asymmetric Nash bargaining approach

The increasing penetration of renewable energy with stochastic characteristics and continuous refinement of carbon emission policies put forward higher requirements for the construction of new power systems. This paper proposes a risk-averse stochastic capacity planning and peer-to-peer (P2P) trading collaborative optimization method for multi-energy microgrids (MEMGs) considering carbon emission limitations. First, a cooperative operation model for MEMGs considering the capacity planning of distributed generation units, uncertainty from renewable energy generations, carbon emission limitations and P2P electricity trading among MEMGs is formulated. Second, a risk-averse stochastic programming method is applied to avoid the potential risk losses caused by randomness and intermittence of renewable energy. Third, an asymmetric Nash bargaining approach is adopted to ensure the fair allocation of benefits and maintain the willingness of individual MEMGs to participate in cooperation. Then, to protect the privacy of individual MEMGs belonging to different stakeholders, the alternating direction method of multipliers is used to solve the two subproblems in a distributed manner. Meanwhile, to further alleviate the computation burdens, a diagonal quadratic approximation method is applied to linearize the quadratic penalty term in the augmented Lagrangian function and realize the parallel solution of all optimization subproblems. Moreover, the influence of different carbon emission targets on the optimal resource combination strategy of the system is investigated by introducing carbon emission factors. Simulations on different models, strategies, and distributed algorithms are conducted to verify the effectiveness and superiority of the proposed method.

Multi-priority HVAC temperature control with resource constrained based on coordinated distributed zone MPC

The conventional HVAC system for multi-area buildings includes both centralized and in-situ equipment to collectively regulate indoor air temperature to meet the preferences of its occupants. However, due to the limited capacity of the total supply air mass rate in centralized equipment, maintaining comfortable indoor air temperature for all areas can face conflicts. The first conflict arises between satisfying diverse preferences and limited global resources, while the second conflict is between centralized air supply and distributed regulation. In this paper, a distributed zone model predictive control (DZMPC) with priority conflict resolution is proposed for the multi-area HVAC systems. This strategy is composed of a zone parameter optimization with dynamic priority as a coordinated upper layer and a lower DZMPC layer. In the upper layer, a zone parameter optimization with priority is presented to coordinate sub-controllers by adjusting variable references’ bounds for DZMPC, to solve the resource competition when it occurs. In the lower layer, model predictive control with zone control is formed to mitigate the tension of the shared resource. By integrating dual decomposition and Augmented Lagrange function to address global constraints, a distributed solvable control structure is formed. The resulting optimization can be solved using a distributed primal–dual algorithm. The effectiveness of the proposed scheme is proved by simulation examples.

Prediction method for the truck's fault time in open-pit mines based on exponential smoothing neural network

The transport truck is one of the important equipment for open-pit mines, and predicting the truck's fault time is of great significance in improving the economic benefits of open-pit mines. In this paper, we discuss the reason for the large prediction error of the exponential smoothing method. Then, we propose a novel nonlinear exponential smoothing method (ESNN) for predicting the truck's fault time, and demonstrate the equivalence between our approach and the neural network structure. Finally, based on the augmented Lagrange function, the solving method of ESNN is proposed. We conduct experiments on real-world datasets and our results demonstrate the effectiveness of ESNN in comparison to existing state-of-the-art methods. Our approach makes it easier for maintenance personnel to predict fault situations in advance and provides a basis for enterprises to develop preventive maintenance plans.

Regularization Strategy Aided Robust Unsupervised Learning for Wireless Resource Allocation

Unsupervised learning (UL) is widely used in the wireless resource allocation problems due to its lower computational complexity and better performance compared with traditional optimization algorithms. Since wireless resource allocation problems usually have several constraints, primal-dual learning based UL framework are widely adopted. However, the primal-dual learning approach has the problem of oscillation around the constraint threshold while training and there may be serious constraint violations when deployment. In addition, although the output of the neural network can also be restricted to the feasible region by the penalty function method, the optimality of such training methods cannot be guaranteed. In this paper, we combine the primal dual learning method with the penalty function method and propose a regularized unsupervised learning (RUL) framework to enhance the robustness of the primal-dual learning based UL framework. In the proposed RUL framework, we use regularization techniques to improve the robustness of primal-dual learning by reducing the risk of constraint violations while training. A quadratic penalty term is introduced into the Lagrangian function of the wireless optimization problem where the constraints can be equivalent to equality constraints to form its augmented Lagrangian function. In the simulation, we give a simple point to point power optimization problem as an example to show that the proposed RUL can improve the robustness of constraint convergence, and can also accelerate training speed.

Collaborative neurodynamic optimization for solving nonlinear equations

A distributed optimization method for solving nonlinear equations with constraints is developed in this paper. The multiple constrained nonlinear equations are converted into an optimization problem and we solve it in a distributed manner. Due to the possible presence of nonconvexity, the converted optimization problem might be a nonconvex optimization problem. To this end, we propose a multi-agent system based on an augmented Lagrangian function and prove that it converges to a locally optimal solution to an optimization problem in the presence of nonconvexity. In addition, a collaborative neurodynamic optimization method is adopted to obtain a globally optimal solution. Three numerical examples are elaborated to illustrate the effectiveness of the main results.