Augmented Lagrangian Function Research Articles

The Alternating Direction Method of Multipliers for Sufficient Dimension Reduction

The minimum average variance estimation (MAVE) method has proven to be an effective approach to sufficient dimension reduction. In this study, we apply the computationally efficient optimization algorithm named alternating direction method of multipliers (ADMM) to a particular approach (MAVE or minimum average variance estimation) to the problem of sufficient dimension reduction (SDR). Under some assumptions, we prove that the iterative sequence generated by ADMM converges to some point of the associated augmented Lagrangian function. Moreover, that point is stationary. It also presents some numerical simulations on synthetic data to demonstrate the computational efficiency of the algorithm.

A Reentry Trajectory Planning Algorithm via Pseudo-Spectral Convexification and Method of Multipliers

The reentry trajectory planning problem of hypersonic vehicles is generally a continuous and nonconvex optimization problem, and it constitutes a critical challenge within the field of aerospace engineering. In this paper, an improved sequential convexification algorithm is proposed to solve it and achieve online trajectory planning. In the proposed algorithm, the Chebyshev pseudo-spectral method with high-accuracy approximation performance is first employed to discretize the continuous dynamic equations. Subsequently, based on the multipliers and linearization methods, the original nonconvex trajectory planning problem is transformed into a series of relaxed convex subproblems in the form of an augmented Lagrange function. Then, the interior point method is utilized to iteratively solve the relaxed convex subproblem until the expected convergence precision is achieved. The convex-optimization-based and multipliers methods guarantee the promotion of fast convergence precision, making it suitable for online trajectory planning applications. Finally, numerical simulations are conducted to verify the performance of the proposed algorithm. The simulation results show that the algorithm possesses better convergence performance, and the solution time can reach the level of seconds, which is more than 97% less than nonlinear programming algorithms, such as the sequential quadratic programming algorithm.

Stochastic nested primal-dual method for nonconvex constrained composition optimization

In this paper we study the nonconvex constrained composition optimization, in which the objective contains a composition of two expected-value functions whose accurate information is normally expensive to calculate. We propose a STochastic nEsted Primal-dual (STEP) method for such problems. In each iteration, with an auxiliary variable introduced to track the inner layer function values we compute stochastic gradients of the nested function using a subsampling strategy. To alleviate difficulties caused by possibly nonconvex constraints, we construct a stochastic approximation to the linearized augmented Lagrangian function to update the primal variable, which further motivates to update the dual variable in a weighted-average way. Moreover, to better understand the asymptotic dynamics of the update schemes we consider a deterministic continuous-time system from the perspective of ordinary differential equation (ODE). We analyze the Karush-Kuhn-Tucker measure at the output by the STEP method with constant parameters and establish its iteration and sample complexities to find an ϵ \epsilon -stationary point, ensuring that expected stationarity, feasibility as well as complementary slackness are below accuracy ϵ \epsilon . To leverage the benefit of the (near) initial feasibility in the STEP method, we propose a two-stage framework incorporating a feasibility-seeking phase, aiming to locate a nearly feasible initial point. Moreover, to enhance the adaptivity of the STEP algorithm, we propose an adaptive variant by adaptively adjusting its parameters, along with a complexity analysis. Numerical results on a risk-averse portfolio optimization problem and an orthogonal nonnegative matrix decomposition reveal the effectiveness of the proposed algorithms.

An improved alternating direction method of multipliers for solving deterministic user equilibrium

An improved alternating direction method of multipliers (iADMM) algorithm is proposed for solving the deterministic user equilibrium (DUE) problem. The iADMM algorithm introduces a key improvement wherein the primal variables can be updated sequentially, possibly multiple times, before updating the dual variables. Subsequently, a sensitivity analysis method is applied to optimise the update frequency of primal variable, and two indices are introduced to evaluate the performance of various policies. Additionally, eliminating equality constraints through the augmented Lagrangian function makes it challenging to completely satisfy flow conservation conditions, resulting in the presence of primal residual. To address this issue, two schemes are proposed to handle the primal residual, involving modifications to the primal variables based on iterative messages. Numerical experiments highlight that the convergence of ADMM algorithm is significantly influenced by the update frequency of primal variable, and addressing the primal residual after each cycle enhances the convergence of ADMM algorithm.

High-Dimensional Multivariate Linear Regression with Weighted Nuclear Norm Regularization

We consider a low-rank matrix estimation problem when the data is assumed to be generated from the multivariate linear regression model. To induce the low-rank coefficient matrix, we employ the weighted nuclear norm (WNN) penalty defined as the weighted sum of the singular values of the matrix. The weights are set in a non-decreasing order, which yields the non-convexity of the WNN objective function in the parameter space. Although the objective function has been widely applied, studies on the estimation properties of its resulting estimator are limited. We propose an efficient algorithm under the framework of the alternative directional method of multipliers (ADMM) to estimate the coefficient matrix. The estimator from the suggested algorithm converges to a stationary point of an augmented Lagrangian function. Under the orthogonal design setting, the effects of the weights for estimating the singular values of the ground-truth coefficient matrix are derived. Under the Gaussian design setting, a minimax convergence rate on the estimation error is derived. We also propose a generalized cross-validation (GCV) criterion for selecting the tuning parameter and an iterative algorithm for updating the weights. Simulations and a real data analysis demonstrate the competitive performance of our new method.

Incorporating STATCOM into a Hydro-Thermal Optimal Power Flow Algorithm

This paper presents the results of the inclusion of Synchronous Static Compensator (STATCOM) power flow models into a hydro-thermal optimal power flow (HTOPF) algorithm. STATCOM basically introduces the voltage magnitude and phase angle of the source converter into the algorithm. For each incorporated STATCOM, an augmented Lagrangian function was formed. The first and second derivatives of these functions were added to the gradient vector and Hessian matrix of an existing algorithm. The modified algorithm was implemented using MATLAB R2018a and tested on 30 and 57 bus systems. Two and three STATCOMs were, respectively, tested on 30 and 57 bus systems. The results obtained showed that one of the STATCOMs used on 30-bus system injected reactive power that ranges from 8.99 MVAR to 28.22 MVAR while the other one injected reactive power in the range of 8.20 MVAR to 33.20 MVAR. For the STATCOMs placed on the 57-bus system, the range of reactive power absorption by the one placed at bus 5 is 7.83 MVAR to 22.86 MVAR while the one at bus 55 absorbed from 5.06 MVAR to 12.92 MVAR. The STATCOM’s reactive power injection at bus 31 ranges from 4.63 MVAR to 10.35 MVAR. All the lower and higher values were obtained at hours 3 and 7, respectively. While the STATCOMs significantly improved the systems’ voltage profile, the impacts of STATCOM on the total systems daily energy loss, daily energy generations (from both plants), daily fuel cost and hydro plant water worth are insignificant.

A Collaborative Neurodynamic Optimization Approach to Distributed Nash-Equilibrium Seeking in Multicluster Games With Nonconvex Functions.

In this article, we propose a collaborative neurodynamic optimization (CNO) method for the distributed seeking of generalized Nash equilibriums (GNEs) in multicluster games with nonconvex functions. Based on an augmented Lagrangian function, we develop a projection neural network for the local search of GNEs, and its convergence to a local GNE is proven. We formulate a global optimization problem to which a global optimal solution is a high-quality local GNE, and we adopt a CNO approach consisting of multiple recurrent neural networks for scattering searches and a metaheuristic rule for reinitializing states. We elaborate on an example of a price-bidding problem in an electricity market to demonstrate the viability of the proposed approach.

A Collaborative Neurodynamic Optimization Approach to Distributed Chiller Loading.

In this article, we present a collaborative neurodynamic optimization approach to distributed chiller loading in the presence of nonconvex power consumption functions and binary variables associated with cardinality constraints. We formulate a cardinality-constrained distributed optimization problem with nonconvex objective functions and discrete feasible regions, based on an augmented Lagrangian function. To overcome the difficulty caused by the nonconvexity in the formulated distributed optimization problem, we develop a collaborative neurodynamic optimization method based on multiple coupled recurrent neural networks reinitialized repeatedly using a meta-heuristic rule. We elaborate on experimental results based on two multi-chiller systems with the parameters from the chiller manufacturers to demonstrate the efficacy of the proposed approach in comparison to several baselines.

A hybrid approach for unit commitment with splitting technique and local search

In the modern electricity market, solving the unit commitment (UC) problem becomes more challenging due to its large number of binary variables and coupling constraints. Therefore, the goal of this paper is to obtain nearly optimal solutions for such complex UC problems within specific time limits. To achieve this, a hybrid approach is presented in this paper, which incorporates a splitting technique and a local search for UC. The hybrid approach involves two steps. In the first step, the augmented Lagrange function (ALF) of UC is decomposed using a splitting technique based on the alternating direction method of multipliers (ADMM). The resulting sub-problems can be effectively solved in a distributed manner by utilizing Lagrange duality theory and binary variable characteristics. Additionally, the penalty parameters of the ALF are updated effectively with the sub-gradient method to diminish the influence of the initial penalty parameters on the algorithm. In the second step, the appropriate relaxation unit is selected using the binary variable characteristics in the current solution of ADMM. Further, the high-quality feasibility of the final solution is ensured through local search. The hybrid approach utilizes ADMM to quickly fix most binary variables in the UC model and then leverages the CPLEX solver to solve the mixed-integer linear programming (MILP) formulation of UC by fixing a large number of binary variables. Consequently, the CPLEX solver needs to branch fewer binary variables, reducing the local search time. The hybrid approach is applied to solve several power systems consisting of up to 1000 units. Compared with the CPLEX solver, the solution of the hybrid approach is found to be stable, and the relative gap of the solution is less than 1 %, demonstrating the effectiveness and performance of the proposed approach while meeting the time limit requirements of the power industry dispatching.

Risk-averse stochastic capacity planning and P2P trading collaborative optimization for multi-energy microgrids considering carbon emission limitations: An asymmetric Nash bargaining approach

The increasing penetration of renewable energy with stochastic characteristics and continuous refinement of carbon emission policies put forward higher requirements for the construction of new power systems. This paper proposes a risk-averse stochastic capacity planning and peer-to-peer (P2P) trading collaborative optimization method for multi-energy microgrids (MEMGs) considering carbon emission limitations. First, a cooperative operation model for MEMGs considering the capacity planning of distributed generation units, uncertainty from renewable energy generations, carbon emission limitations and P2P electricity trading among MEMGs is formulated. Second, a risk-averse stochastic programming method is applied to avoid the potential risk losses caused by randomness and intermittence of renewable energy. Third, an asymmetric Nash bargaining approach is adopted to ensure the fair allocation of benefits and maintain the willingness of individual MEMGs to participate in cooperation. Then, to protect the privacy of individual MEMGs belonging to different stakeholders, the alternating direction method of multipliers is used to solve the two subproblems in a distributed manner. Meanwhile, to further alleviate the computation burdens, a diagonal quadratic approximation method is applied to linearize the quadratic penalty term in the augmented Lagrangian function and realize the parallel solution of all optimization subproblems. Moreover, the influence of different carbon emission targets on the optimal resource combination strategy of the system is investigated by introducing carbon emission factors. Simulations on different models, strategies, and distributed algorithms are conducted to verify the effectiveness and superiority of the proposed method.

Multi-priority HVAC temperature control with resource constrained based on coordinated distributed zone MPC

The conventional HVAC system for multi-area buildings includes both centralized and in-situ equipment to collectively regulate indoor air temperature to meet the preferences of its occupants. However, due to the limited capacity of the total supply air mass rate in centralized equipment, maintaining comfortable indoor air temperature for all areas can face conflicts. The first conflict arises between satisfying diverse preferences and limited global resources, while the second conflict is between centralized air supply and distributed regulation. In this paper, a distributed zone model predictive control (DZMPC) with priority conflict resolution is proposed for the multi-area HVAC systems. This strategy is composed of a zone parameter optimization with dynamic priority as a coordinated upper layer and a lower DZMPC layer. In the upper layer, a zone parameter optimization with priority is presented to coordinate sub-controllers by adjusting variable references’ bounds for DZMPC, to solve the resource competition when it occurs. In the lower layer, model predictive control with zone control is formed to mitigate the tension of the shared resource. By integrating dual decomposition and Augmented Lagrange function to address global constraints, a distributed solvable control structure is formed. The resulting optimization can be solved using a distributed primal–dual algorithm. The effectiveness of the proposed scheme is proved by simulation examples.

Prediction method for the truck's fault time in open-pit mines based on exponential smoothing neural network

The transport truck is one of the important equipment for open-pit mines, and predicting the truck's fault time is of great significance in improving the economic benefits of open-pit mines. In this paper, we discuss the reason for the large prediction error of the exponential smoothing method. Then, we propose a novel nonlinear exponential smoothing method (ESNN) for predicting the truck's fault time, and demonstrate the equivalence between our approach and the neural network structure. Finally, based on the augmented Lagrange function, the solving method of ESNN is proposed. We conduct experiments on real-world datasets and our results demonstrate the effectiveness of ESNN in comparison to existing state-of-the-art methods. Our approach makes it easier for maintenance personnel to predict fault situations in advance and provides a basis for enterprises to develop preventive maintenance plans.

Regularization Strategy Aided Robust Unsupervised Learning for Wireless Resource Allocation

Unsupervised learning (UL) is widely used in the wireless resource allocation problems due to its lower computational complexity and better performance compared with traditional optimization algorithms. Since wireless resource allocation problems usually have several constraints, primal-dual learning based UL framework are widely adopted. However, the primal-dual learning approach has the problem of oscillation around the constraint threshold while training and there may be serious constraint violations when deployment. In addition, although the output of the neural network can also be restricted to the feasible region by the penalty function method, the optimality of such training methods cannot be guaranteed. In this paper, we combine the primal dual learning method with the penalty function method and propose a regularized unsupervised learning (RUL) framework to enhance the robustness of the primal-dual learning based UL framework. In the proposed RUL framework, we use regularization techniques to improve the robustness of primal-dual learning by reducing the risk of constraint violations while training. A quadratic penalty term is introduced into the Lagrangian function of the wireless optimization problem where the constraints can be equivalent to equality constraints to form its augmented Lagrangian function. In the simulation, we give a simple point to point power optimization problem as an example to show that the proposed RUL can improve the robustness of constraint convergence, and can also accelerate training speed.

Collaborative neurodynamic optimization for solving nonlinear equations

A distributed optimization method for solving nonlinear equations with constraints is developed in this paper. The multiple constrained nonlinear equations are converted into an optimization problem and we solve it in a distributed manner. Due to the possible presence of nonconvexity, the converted optimization problem might be a nonconvex optimization problem. To this end, we propose a multi-agent system based on an augmented Lagrangian function and prove that it converges to a locally optimal solution to an optimization problem in the presence of nonconvexity. In addition, a collaborative neurodynamic optimization method is adopted to obtain a globally optimal solution. Three numerical examples are elaborated to illustrate the effectiveness of the main results.

The Alternating Direction Search Pattern Method for Solving Constrained Nonlinear Optimization Problems

We adopt the alternating direction search pattern method to solve the equality and inequality constrained nonlinear optimization problems. Firstly, a new augmented Lagrangian function with a nonlinear complementarity function is proposed to transform the original constrained problem into a new unconstrained problem. Under appropriate conditions, it has been proven that there is a 1-1 correspondence between the local and global optimal solutions of the new unconstrained problem and the original constrained problem. In this way, the optimal solution of the original problem can be obtained by solving the new unconstrained optimization problem. Furthermore, based on the characteristics of the new problem, the alternating direction pattern search method was designed and its convergence was demonstrated. Numerical experiments were implemented to illustrate the availability of the new augmented Lagrangian function and the algorithm.

An alternating direction method of multipliers for solving user equilibrium problem

This paper introduces a new parallel computing algorithm to address the user equilibrium (UE) problem. Searching for efficient solution algorithms for UE has been a recurring study subject in transportation research and has attracted much attention in past decades. Existing solution algorithms can be classified into three categories: link-based, path-based, and origin-based. This paper introduces an alternating direction method of multipliers (ADMM) algorithm that is different from these categories. Based on the origin-based formulation of UE problem, an equivalent problem is proposed which eliminates the flow conservation conditions through the augmented Lagrangian function. In order to make use of the ADMM, the network links should be grouped into different blocks, where the links in the same block are disconnected. This link grouping problem falls into the category of edge-coloring problem in graph theory, and it follows the Vizing theorem. A novel approach is developed for the link grouping problem. For links in the same block, we have a separable subproblem, which is solved in parallel by the gradient projection algorithm. Numerical experiments are conducted to validate the proposed algorithm, which shows its computation efficiency.

A HYBRID MODIFIED SUBGRADIENT ALGORITHM THAT SELF-DETERMINES THE PROPER PARAMETER VALUES

A successful solution algorithm for non-convex optimization problems is the Modified Subgradient Algorithm (MSGA), which solves dual problems based on the sharp augmented lagrangian function. However, its performance highly depends on its parameter values, and determining the appropriate parameter values is difficult as they can be completely different for each problem. In this study, a new hybrid solution approach that a tabu search algorithm to find the appropriate MSGA parameter values and the MSGA algorithm run together is proposed. Although it seems like a contradiction to use an algorithm that also has its parameters to determine the most appropriate parameter values of an algorithm, this contradiction is eliminated by fixing the parameter values of the tabu search algorithm. The proposed algorithm does not need appropriate values of any algorithm parameter. It can find appropriate parameter values for each problem itself starting with the same fixed initial values. To show the success of the developed algorithm, especially on 0-1 quadratic problems, it is compared with the classical MSGA algorithm by using the quadratic knapsack test instances taken in the literature. According to the obtained solutions, the superiority of the hybrid algorithm has been demonstrated.

A First-Order Primal-Dual Method for Nonconvex Constrained Optimization Based on the Augmented Lagrangian

Nonlinearly constrained nonconvex and nonsmooth optimization models play an increasingly important role in machine learning, statistics, and data analytics. In this paper, based on the augmented Lagrangian function, we introduce a flexible first-order primal-dual method, to be called nonconvex auxiliary problem principle of augmented Lagrangian (NAPP-AL), for solving a class of nonlinearly constrained nonconvex and nonsmooth optimization problems. We demonstrate that NAPP-AL converges to a stationary solution at the rate of [Formula: see text], where k is the number of iterations. Moreover, under an additional error bound condition (to be called HVP-EB in the paper) with exponent [Formula: see text], we further show the global convergence of NAPP-AL. Additionally, if [Formula: see text], then we furthermore show that the convergence rate is in fact linear. Finally, we show that the well-known Kurdyka-Łojasiewicz property and the Hölderian metric subregularity imply the aforementioned HVP-EB condition. We demonstrate that under mild conditions, NAPP-AL can also be interpreted as a variant of the forward-backward operator splitting method in this context. Funding: This work was supported by the National Natural Science Foundation of China [Grant 71871140].

Convergence analysis of consensus-ADMM for general QCQP

We analyze the convergence properties of the consensus-alternating direction method of multipliers (ADMM) for solving general quadratically constrained quadratic programs. We prove that the augmented Lagrangian function value is monotonically non-increasing as long as the augmented Lagrangian parameter is chosen to be sufficiently large. Simulation results show that the augmented Lagrangian function is bounded from below when the matrix in the quadratic term of the objective function is positive definite. In such a case, the consensus-ADMM is convergent.

Iteration Complexity of an Inner Accelerated Inexact Proximal Augmented Lagrangian Method Based on the Classical Lagrangian Function

This paper establishes the iteration complexity of an inner accelerated inexact proximal augmented Lagrangian (IAIPAL) method for solving linearly constrained smooth nonconvex composite optimization problems that is based on the classical augmented Lagrangian (AL) function. More specifically, each IAIPAL iteration consists of inexactly solving a proximal AL subproblem by an accelerated composite gradient (ACG) method followed by a classical Lagrange multiplier update. Under the assumption that the domain of the composite function is bounded and the problem has a Slater point, it is shown that IAIPAL generates an approximate stationary solution in ACG iterations where is a tolerance for both stationarity and feasibility. Moreover, the above bound is derived without assuming that the initial point is feasible. Finally, numerical results are presented to demonstrate the strong practical performance of IAIPAL.