Equality Constrained Optimization Problems Research Articles

Fully Stochastic Trust-Region Sequential Quadratic Programming for Equality-Constrained Optimization Problems

Fully Stochastic Trust-Region Sequential Quadratic Programming for Equality-Constrained Optimization Problems

On the development of steady-state and dynamic mass-constrained neural networks using noisy transient data

This paper presents the development of algorithms for mass-constrained neural network models that can exactly satisfy mass conservation laws of chemical process systems, even if the training data violates the same. As opposed to approximately satisfying mass balance constraints of a system by considering additional penalty terms in the objective function, algorithms have been developed to solve an equality-constrained optimization problem, thus ensuring the exact satisfaction of the overall mass conservation laws. For developing dynamic mass-constrained networks, hybrid series and parallel all-nonlinear static-dynamic neural network models are leveraged. The proposed algorithms for solving both the inverse and forward problems are tested by considering both steady-state and dynamic data in presence of varieties of noise characterizations. The proposed structures and algorithms are applied to the development of data-driven models of two nonlinear dynamic chemical processes, namely the Van de Vusse reactor system as well as a solvent-based post-combustion CO2 capture process.

Exponential Convergence of Primal-Dual Dynamics Under General Conditions and Its Application to Distributed Optimization.

In this article, we establish the local and global exponential convergence of a primal-dual dynamics (PDD) for solving equality-constrained optimization problems without strong convexity and full row rank assumption on the equality constraint matrix. Under the metric subregularity of Karush-Kuhn-Tucker (KKT) mapping, we prove the local exponential convergence of the dynamics. Moreover, we establish the global exponential convergence of the dynamics in an invariant subspace under a technically designed condition which is weaker than strong convexity. As an application, the obtained theoretical results are used to show the exponential convergence of several existing state-of-the-art primal-dual algorithms for solving distributed optimization without strong convexity. Finally, we provide some experiments to demonstrate the effectiveness of our results.

Методы с суженной матрицей Гессе как возмущенный метод Ньютона–Лагранжа

For an equality-constrained optimization problem, we consider the possibility to interpret sequential quadratic programming methods employing the Hessian of the Lagrangian reduced to the null space of the constraints’ Jacobian, as a perturbed Newton–Lagrange method. We demonstrate that such interpretation with required estimates on perturbations is possible for certain sequences generated by variants of these methods making use of second-order corrections. This allows to establish, from a general perspective, superlinear convergence of such sequences, the property generally missing for the main sequences of the methods in question.

Parallel-in-time multiple shooting for optimal control problems governed by the Navier–Stokes equations

In the past decades, multiple shooting methods have proven to be a promising direction to speed up the optimization process, especially in the context of ODE-based optimization. Very recently, Fang et al. (Journal of Computational Physics, vol. 452, 110926, 2022) proposed a multiple shooting algorithm for large-scale PDE-constrained optimization. The current paper continues this line of work and explores the potential of multiple shooting methods for optimal control problems governed by the three-dimensional Navier–Stokes equations. Similar to Fang et al., the augmented Lagrangian (AL) method is used to solve the resulting equality-constrained optimization problem, and we employ the classical limited-memory BFGS method for the unconstrained subproblems inside the AL loop. In the current work, we exploit the multiple shooting paradigm in full by processing the shooting windows parallel-in-time, allowing for significant parallel speed-ups compared to single shooting. The proposed method is validated on a velocity tracking case, using up to 100 windows. Our analysis shows that the multiple shooting algorithm allows for considerable algorithmic and parallel speed-ups. While algorithmic speed-up depends on the exact tracking case and initialization of the shooting windows, the multiple shooting algorithm always outperforms single shooting in terms of computational time (if the number of windows is sufficiently high) due to the parallel-in-time implementation. For a given amount of resources, we also show that the proposed parallel-in-time strategy can outperform spatial parallelization alone, especially when the spatial parallelization is saturated.

Techniques for adjusting dual iterates in the presence of critical Lagrange multipliers for optimization problems with inequality constraints

The dual modification technique earlier proposed for equality-constrained optimization problems aims at avoiding attraction to critical Lagrange multipliers implying the loss of local fast convergence. In this paper, we extend this technique to problems involving inequality constraints. In particular, we analyze two approaches. One tackles directly the inequality-constrained problem at hand, while the second approach employs squared slacks for reformulating the inequality constraints as equalities. Some theoretical justifications for the corresponding two variants of the stabilized sequential programming algorithm equipped with dual modification procedures are provided. Theoretical expectations are supported by promising numerical results.

An adaptive moments-based interface reconstruction using intersection of the cell with one half-plane, two half-planes and a circle

We present a new adaptive moment-of-fluid (A-MOF) interface reconstruction method. It uses the zeroth, first, and second moments of the fragment of material inside a cell of the mesh to construct a shape that approximates the respective material fragment. The new method requires information about the material moments only for the cell under consideration. The adaptive method chooses between shapes obtained by the intersection of the cell with one half-plane, two half-planes, or a circle. The A-MOF method allows to exactly reproduce several convex shapes: corners, filaments, and their concave cell-complements; as well as pieces of the circles and its cell-compliments. Interface reconstruction is formulated as a local (for each cell), non-linear, equality constrained optimization problem, which does not require additional communication and allows for an efficient parallel implementation.We present an extensive set of test problems, both for interface reconstruction on a single cell, and for reconstruction of a variety of shapes on the entire mesh.

Additive autoregressive models for matrix valued time series

In this article, we develop additive autoregressive models (Add‐ARM) for the time series data with matrix valued predictors. The proposed models assume separable row, column and lag effects of the matrix variables, attaining stronger interpretability when compared with existing bilinear matrix autoregressive models. We utilize the Gershgorin's circle theorem to impose some certain conditions on the parameter matrices, which make the underlying process strictly stationary. We also introduce the alternating least squares estimation method to solve the involved equality constrained optimization problems. Asymptotic distributions of the parameter estimators are derived. In addition, we employ hypothesis tests to run diagnostics on the parameter matrices. The performance of the proposed models and methods is further demonstrated through simulations and real data analysis.

Evolutionary Padé Approximation for Heat and Mass Transfer Analysis of Falkner–Skan Flow of a Bio-Convective Casson Fluid

This study presents numerical work to investigate the Falkner–Skan flow of a bio-convective Casson fluid over a wedge using an Evolutionary Padé Approximation (EPA) scheme. The governing partial differential equations and boundary conditions of a Falkner–Skan flow model are transformed to a system of ordinary differential equations involving ten dimensionless parameters by using similarity transformations. In the proposed EPA framework, an equivalent constrained optimization problem is formed. The solution of the resulting optimization problem is analogous to the solution of the dimensionless system of ordinary differential equations. The solutions produced in this work, with respect to various combinations of the physical parameters, are found to be in good agreement with those reported in the previously published literature. The effects of a non-dimensional physical-parameter wedge, Casson fluid, fluid phase effective heat capacity, Brownian motion, thermophoresis, radiation, and magnetic field on velocity profile, temperature profile, fluid concentration profile, and the density of motile microorganisms are discussed and presented graphically. It is observed that the fluid velocity rises with a rise in the Casson fluid viscosity force parameter, and an increase in the Prandtl number causes a decrease in the heat transfer rate. Another significant observation is that the temperature and fluid concentration fields are greatly increased by an increase in the thermophoresis parameter. An increase in the Péclet number suppresses the microorganism density. Moreover, the increased values of the Prandtl number increase the local Nusslet number, whereas the skin friction is increased when an increase in the Prandtl number occurs.

Equality-Constrained Engineering Design Optimization Using a Novel Inexact Quasi-Newton Method

For engineering design optimization, the full-space formulation offers the potential for greater efficiency than the more commonly used reduced-space formulation. This potential is greater when the numerical model involves discretized partial differential equations or coupled disciplines. However, the full-space formulation results in a larger optimization problem with at least a factor of two increase in the number of optimization variables and equality constraints. Using Newton-type methods to solve such problems involves solving a large-scale and, often, ill-conditioned Karush–Kuhn–Tucker linear system at each optimization iteration. This can be time-consuming to solve even with a Krylov solver. If the number of iterations is reduced, the full-space formulation could be applied to a broader class of problems. This paper presents an inexact quasi-Newton algorithm with an adaptive extension for solving large-scale equality-constrained optimization problems. The new algorithm inexactly solves the Karush–Kuhn–Tucker system using new inexactness criteria that are derived to ensure a descent direction. The adaptive extension chooses the stopping condition of the Krylov solver by also taking its convergence rate into account. The paper presents results of numerical experiments applying this algorithm to three types of problems: six constrained optimization problems from the widely used CUTEst test suite, a bar thickness optimization problem, and a two-dimensional topology optimization problem. For all problems, the new algorithm consistently shows a roughly 50% reduction in the total number of Krylov solver iterations and a minimum of roughly 15% reduction in the optimization time. Moreover, the proposed approach for selecting the Krylov solver tolerance shows an improvement in all cases, whereas the existing forcing-parameter approach shows an increase in the number of Krylov iterations in some cases. These results indicate that this new method for selecting solver tolerances is effective and robust, and a good choice in algorithms that use a Krylov solver for solving the Karush–Kuhn–Tucker linear system.

A Contraction Analysis of Primal-Dual Dynamics in Distributed and Time-Varying Implementations

In this article, we provide an overarching analysis of primal-dual dynamics associated with linear equality-constrained optimization problems using contraction analysis. For the well-known standard version of the problem, we establish convergence under convexity and the contracting rate under strong convexity. Then, for a canonical distributed optimization problem, we use partial contractivity to establish global exponential convergence of its primal-dual dynamics. As an application, we propose a new distributed solver for the least-squares problem with the same convergence guarantees. Finally, for time-varying versions of both centralized and distributed primal-dual dynamics, we exploit their contractive nature to establish bounds on their tracking error. To support our analyses, we introduce novel results on contraction theory.

A Symmetric Inertial Alternating Direction Method of Multipliers for Elliptic Equation Constrained Optimization Problem

A Symmetric Inertial Alternating Direction Method of Multipliers for Elliptic Equation Constrained Optimization Problem

Resilient State Estimation and Control of Cyber-Physical Systems Against False Data Injection Attacks on Both Actuator and Sensors

This article studies the secure state estimation and control problem of cyber-physical systems, where there are multiple heterogeneous sensing nodes and one actuator which are both under the threat of intermittent false data injection attacks. A specific filter gain is proposed by solving the equality constrained optimization problem so that the attack on the actuator does not affect the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a posteriori</i> estimation error of the system. A new method to construct <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\chi ^2$</tex-math></inline-formula> detection variables is proposed, which does not require that the covariance matrix be invertible, and then the actuator and sensor attack detection schemes are developed, respectively. Based on the detection results, a multiple information fusion algorithm is developed and an estimator-based controller with the compensator, which can compensate for the actuator attack in the previous step, is further designed. A necessary and sufficient condition for complete compensation is provided, and the stability of the filter and the control system is proved. Finally, the effectiveness of the estimation algorithm and control algorithm is verified by numerical simulation.

Laplacian Smoothing Stochastic ADMMs With Differential Privacy Guarantees

Many machine learning tasks such as structured sparse coding and multi-task learning can be converted into an equality constrained optimization problem. The stochastic alternating direction method of multipliers (SADMM) is a popular algorithm to solve such large-scale problems, and has been successfully used in many real-world applications. However, existing SADMMs fail to take into consideration an important issue in their designs, i.e., protecting sensitive information. To address this challenging issue, this paper proposes a novel differential privacy stochastic ADMM framework for solving equality constrained machine learning problems. In particular, to further lift the utility in privacy-preserving equality constrained optimization, a Laplacian smoothing operation is also introduced into our differential privacy ADMM framework, and it can smooth out the Gaussian noise used in the Gaussian mechanism. Then we propose an efficient differentially private variance reduced stochastic ADMM (DP-VRADMM) algorithm with Laplacian smoothing for both strongly convex and general convex objectives. As a by-product, we also present a new differentially private stochastic ADMM algorithm with DP guarantees. In theory, we provide both private guarantees and utility guarantees for the proposed algorithms, which show that Laplacian smoothing can improve the utility bounds of our algorithms. Experimental results on real-world datasets verify our theoretical results and the effectiveness of our algorithms.

Optimization with learning-informed differential equation constraints and its applications

Inspired by applications in optimal control of semilinear elliptic partial differential equations and physics-integrated imaging, differential equation constrained optimization problems with constituents that are only accessible through data-driven techniques are studied. A particular focus is on the analysis and on numerical methods for problems with machine-learned components. For a rather general context, an error analysis is provided, and particular properties resulting from artificial neural network based approximations are addressed. Moreover, for each of the two inspiring applications analytical details are presented and numerical results are provided.

A parallel-in-time multiple shooting algorithm for large-scale PDE-constrained optimal control problems

Multiple shooting methods for solving optimal control problems governed by ODEs have been extensively studied in past decades. However, their application for solving large-scale PDE-based optimal control problems still faces many challenges, including the difficulty of solving large scale equality constrained optimization problems in an efficient parallelizable way. The current work proposes and analyzes a new parallel-in-time multiple shooting algorithm for large scale optimal control problems governed by parabolic PDEs. We solve the equality constrained optimization problems introduced by the multiple shooting strategy by using the augmented Lagrangian method, in which the unconstrained subproblems are solved using a classical limited-memory BFGS quasi-Newton method. An optimal control problem governed by the Nagumo equation is employed to validate the proposed algorithm and analyze its efficiency. The results demonstrate that substantial accelerations can be achieved for multiple shooting approaches when proper starting guesses of controls are provided, and when the control variables are scaled appropriately. A second test case consists of a two-dimensional velocity tracking problem that is governed by the Navier–Stokes equations. The influence of the flow complexity on the optimization method is studied, and the results illustrate that for a fluid field with more complex structures, the efficiency of the algorithm further increases. Overall, for the different cases considered, we find algorithmic speed-ups of up to 6 versus single shooting, depending on the starting guess, and the specific tracking problem.

Sequential Projected Newton method for regularization of nonlinear least squares problems

We develop a computationally efficient algorithm for the automatic regularization of nonlinear inverse problems based on the discrepancy principle. We formulate the problem as an equality constrained optimization problem, where the constraint is given by a least squares data fidelity term and expresses the discrepancy principle. The objective function is a convex regularization function that incorporates some prior knowledge, such as the total variation regularization function. Using the Jacobian matrix of the nonlinear forward model, we consider a sequence of quadratically constrained optimization problems that can all be solved using the Projected Newton method. We show that the solution of such a quadratically constrained sub-problem results in a descent direction for an exact merit function. This merit function can then be used to describe a formal line-search method. We also formulate a slightly more heuristic approach that simplifies the algorithm and allows for an inexact solution of the sequence of sub-problems. We illustrate the behavior of the algorithm using a number of numerical experiments, with Talbot-Lau x-ray phase contrast imaging as the main application. The numerical experiments confirm that the quadratically constrained sub-problems need not be solved with high accuracy in early iterations to make sufficient progress towards the solution. In addition, we show that the proposed method is able to produce reconstructions of similar quality compared to other state-of-the-art approaches with a significant reduction in computational time.

Structural Reliability Analysis Using Group Teaching Optimization Algorithm

Conventional First order reliability method (FORM) usually encounters divergence and non-convergence for high nonlinear performance functions. This paper puts forward an improved FORM coupling with a newly raised metaheuristic optimization algorithm, that is, Group teaching optimization algorithm (GTO). The GTO algorithm is inspired by grouping teaching strategy, featured by no additional control parameters, and has shown excellent global search ability for high nonlinear problems. Thus, the GTO algorithm is utilized to solve the equivalent constrained optimization problem according to the first order reliability theory. The implementation procedure of the proposed method is then presented in this paper. A numerical example is employed to verify the performance of the proposed method. Traditional reliability methods and PSO-based FORM are also introduced to compared with the proposed approach. The simulation results prove that the proposed method provides good performance in terms of computational efficiency and accuracy.

Continuation methods with the trusty time-stepping scheme for linearly constrained optimization with noisy data

The nonlinear optimization problem with linear constraints has many applications in engineering fields such as the visual-inertial navigation and localization of an unmanned aerial vehicle maintaining the horizontal flight. In order to solve this practical problem efficiently, this paper constructs a continuation method with the trusty time-stepping scheme for the linearly equality-constrained optimization problem at every sampling time. At every iteration, the new method only solves a system of linear equations other than the traditional optimization method such as the sequential quadratic programming (SQP) method, which needs to solve a quadratic programming subproblem. Consequently, the new method can save much more computational time than SQP. Numerical results show that the new method works well for this problem and its consumed time is about one fifth of that of SQP (the built-in subroutine fmincon.m of the MATLAB2018a environment) or that of the traditional dynamical method (the built-in subroutine ode15s.m of the MATLAB2018a environment). Furthermore, we also give the global convergence analysis of the new method.

Reformative Noise-Immune Neural Network for Equality-Constrained Optimization Applied to Image Target Detection

Equality-constrained optimization problem captures increasing attention in the fields of computer science, control engineering, and applied mathematics. Almost all of the relevant issues suffer from kinds of intense or weak noises during the solving process, so that how to realize the noise deduction even noise elimination has increasingly become a sticky and significant problem. A lot of corresponding solving models are established for the equality-constrained optimization problem. However, the majority of them can find the optimal solution to a certain extent in the absence of noise disturbance, but few can behave a brilliant noise-resistance proficiency. On account of this discovery, a reformative noise-immune neural network (RNINN) model is constructed. In addition, the conventional gradient-based recursive neural network model and the zeroing recursive neural network model are presented to compare with the proposed RNINN model on convergence properties and noise-resistance capabilities. Lastly, the relative numerical experiment simulation and image target detection application are implemented to further elaborate on the robustness and efficiency of the RNINN model.