Stochastic Optimization Algorithm Research Articles

A Stochastic Inexact Sequential Quadratic Optimization Algorithm for Nonlinear Equality-Constrained Optimization

A stochastic algorithm is proposed, analyzed, and tested experimentally for solving continuous optimization problems with nonlinear equality constraints. It is assumed that constraint function and derivative values can be computed but that only stochastic approximations are available for the objective function and its derivatives. The algorithm is of the sequential quadratic optimization variety. Distinguishing features of the algorithm are that it only employs stochastic objective gradient estimates that satisfy a relatively weak set of assumptions (while using neither objective function values nor estimates of them) and that it allows inexact subproblem solutions to be employed, the latter of which is particularly useful in large-scale settings when the matrices defining the subproblems are too large to form and/or factorize. Conditions are imposed on the inexact subproblem solutions that account for the fact that only stochastic objective gradient estimates are employed. Convergence results are established for the method. Numerical experiments show that the proposed method vastly outperforms a stochastic subgradient method and can outperform an alternative sequential quadratic programming algorithm that employs highly accurate subproblem solutions in every iteration. Funding: This material is based upon work supported by the National Science Foundation [Awards CCF-1740796 and CCF-2139735] and the Office of Naval Research [Award N00014-21-1-2532].

Efficient mini-batch stochastic gradient descent with Centroidal Voronoi Tessellation for PDE-constrained optimization under uncertainty

The study of optimal control problems under uncertainty plays an important role in scientific numerical simulations. This class of optimization problems is frequently utilized in engineering, biology and finance. Although stochastic gradient descent is a well-known stochastic optimization technique for solving the approximate optimal control problem, it often exhibits a slow convergence rate due to the inherent variance in gradient approximation. To address this issue, we propose a more efficient stochastic optimization algorithm based on Centroidal Voronoi Tessellation sampling. This strategy reduces the error in gradient estimation compared to the conventional mini-batch stochastic gradient descent method. Our approach involves dividing the whole snapshot set into several Voronoi cells with low variance and extracting samples with good uniformity from each region to construct an unbiased estimation of the full gradient. Our method can economically and stably approximate numerical optimal control functions even with a fixed step size. Numerical results demonstrate that the proposed method is a reliable algorithm with great potential for applications in the field of optimization.

Improve robustness of machine learning via efficient optimization and conformal prediction

AbstractThe advance of machine learning (ML) systems in real‐world scenarios usually expects safe deployment in high‐stake applications (e.g., medical diagnosis) for critical decision‐making process. To this end, provable robustness of ML is usually required to measure and understand how reliable the deployed ML system is and how trustworthy their predictions can be. Many studies have been done to enhance the robustness in recent years from different angles, such as variance‐regularized robust objective functions and conformal prediction (CP) for uncertainty quantification on testing data. Although these tools provably improve the robustness of ML model, there is still an inevitable gap to integrate them into an end‐to‐end deployment. For example, robust objectives usually require carefully designed optimization algorithms, while CP treats ML models as black boxes. This paper is a brief introduction to our recent research focusing on filling this gap. Specifically, for learning robust objectives, we designed sample‐efficient stochastic optimization algorithms that achieves the optimal (or faster compared to existing algorithms) convergence rates. Moreover, for CP‐based uncertainty quantification, we established a framework to analyze the expected prediction set size (smaller size means more efficiency) of CP methods in both standard and adversarial settings. This paper elaborates the key challenges and our exploration towards efficient algorithms with details of background methods, notions for robustness measure, concepts of algorithmic efficiency, our proposed algorithms and results. All of them further motivate our future research on risk‐aware ML that can be critical for AI–human collaborative systems. The future work mainly targets designing conformal robust objectives and their efficient optimization algorithms.

Adaptive switching and routing protocol design and optimization in internet of things based on probabilistic models

Through smart cities, Intelligent Transportation Systems (ITS), the agricultural sector, and wearable devices, the Internet of Things (IoT) has revolutionized several human interests. Through the development of new cluster tasks, the Decision-Making System (DMS) of Cluster Heads (CHs), and improving the accuracy of traffic prediction and reliability of transportation, the present study intends to improve the energy depletion of IoT devices. The paper explores the subject of data flow optimization using Fuzzy Assisted Cuckoo Search Optimization (FACSO), traffic flow using Gaussian Process Regression (GPR), and CH prediction using the Stochastic Optimization Algorithm (SOA). Optimizing network lifetime while minimizing Energy Consumption (EC) is feasible through the practical application of the SOA, GPR, and FACSO models. Increasing End-to-End Delay (EED), Network Throughput (NT), and energy efficiency can be rendered feasible through a real-time DMS regarding routing employing a novel approach referred to as FACSO. This approach has enhanced the efficacy and reliability of Wireless Sensor Networks (WSN). With up to 500 nodes and an EC of 0.3451 J, the experiment's findings demonstrate that a proposed SOA-FACSO model achieves superior EED.

Distributed gradient-free and projection-free algorithm for stochastic constrained optimization

Distributed stochastic zeroth-order optimization (DSZO), in which the objective function is allocated over multiple agents and the derivative of cost functions is unavailable, arises frequently in large-scale machine learning and reinforcement learning. This paper introduces a distributed stochastic algorithm for DSZO in a projection-free and gradient-free manner via the Frank-Wolfe framework and the stochastic zeroth-order oracle (SZO). Such a scheme is particularly useful in large-scale constrained optimization problems where calculating gradients or projection operators is impractical, costly, or when the objective function is not differentiable everywhere. Specifically, the proposed algorithm, enhanced by recursive momentum and gradient tracking techniques, guarantees convergence with just a single batch per iteration. This significant improvement over existing algorithms substantially lowers the computational complexity. Under mild conditions, we prove that the complexity bounds on SZO of the proposed algorithm are O(n/ϵ2)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{O}(n/\\epsilon ^{2})$\\end{document} and O(n(21ϵ))\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{O}(n(2^{\\frac{1}{\\epsilon}}))$\\end{document} for convex and nonconvex cases, respectively. The efficacy of the algorithm is verified on black-box binary classification problems against several competing alternatives.

Simultaneous Reconstruction of Multiple Time-Varying Thermal Properties Based on Translucent Materials.

In the realm of high-tech materials and energy applications, accurately measuring the transient heat flow at media boundaries and the internal thermal conductivity of materials in harsh heat exchange environments poses a significant challenge when using conventional direct measurement methods. Consequently, the study of photothermal parameter reconstruction in translucent media, which relies on indirect measurement techniques, has crucial practical value. Current research on reconstructing photothermal properties within participating media typically focuses on single-objective or time-invariant properties. There is a pressing need to develop effective methods for the simultaneous reconstruction of time-varying thermal flow fields and internal thermal conductivity at the boundaries of participating media. This paper introduces a computational model based on the numerical simulation theory of internal heat transfer systems in participating media, stochastic particle swarm optimization algorithms, and Kalman filter technology. The model aims to enable the simultaneous reconstruction of various thermal parameters within the target medium. Our results demonstrate that under varying levels of measurement noise, the inversion results for different target parameters exhibit slight oscillations around the true values, leading to a reduction in reconstruction accuracy. However, overall, the model demonstrates robustness and accuracy in ideal conditions, validating its effectiveness.

Simultaneous port number and location optimization of mold gates‐vents using in‐house coded stochastic multi‐objective optimization algorithm

AbstractThis research introduces multi‐objective stochastic optimization (MOSO) and non‐dominated sorting differential evolution (NSDE) in‐house coded algorithms for enhancing the optimization of the resin transfer molding (RTM) process during the mold‐fill phase for composite parts. Glass fiber‐vinyl ester‐based automotive bonnet and carbon fiber‐RTM6‐based aircraft wing flap parts were used as the case studies. Initially, the NSDE algorithm was developed for the simultaneous optimization of dry‐spot content and mold‐fill time by changing the locations of gates and vents with pre‐fixed port numbers. Then, the MOSO algorithm was designed for the concurrent optimization of the dry‐spot content, mold‐fill time and total number of ports by changing both the numbers and locations of gates and vents. The effect of race‐tracking was also investigated using higher permeability values at the composite part‐cut edges. When compared, the MOSO algorithm predicted less dry‐spot content, number of ports, mold‐fill time and uniform resin flow‐front progressions with lesser functional evaluations and computational time than the NSDE algorithm.Highlights The non‐dominated sorting differential evolution (NSDE) and multi‐objective stochastic optimization (MOSO) algorithms were developed for the optimization of the mold fill phase of resin transfer molded composite parts. The development of the NSDE algorithm aimed to concurrently optimize the dry‐spot content and mold‐fill time by changing the locations of gates and vents with pre‐fixed port numbers. The MOSO algorithm was designed to concurrently optimize dry‐spot content, mold‐fill time, and the total number of ports by varying both the numbers and locations of gates and vents. The MOSO algorithm predicted better‐automated mold‐fill phase optimalities for the resin transfer molded composite parts compared to the NSDE algorithm.

Optimal proportional-integral speed control for closed-loop engine timing system

In internal combustion engines, adjusting the air-fuel ratio is essential to control the speed and minimize the burnt fuel. The throttle opening is the actuator to control the air-fuel. A better design for the used/conventional controller can give a better response without additional cost. In this work, the proposed controller gains of the proportional-integral (PI) controller are tuned to enhance the speed in constant and variable drive cycle modes. The tuning process is conducted based on two of the most efficient performance indices used in this field. The performance indices are integral absolute error (IAE) and integral time absolute error (ITAE). The optimization problem is solved using three reliable stochastic optimization algorithms to ensure mature convergence of the solutions, to avoid local optima solutions, and to ensure effective shrinking of the search space. The optimization algorithms are teaching-learning-based optimization (TLBO), particle swarm optimization (PSO), and genetic algorithm (GA). Different simulations are conducted to validate the results. The results are compared with conventional tuning methods regarding the system's time response.

Improved adaptive regularization for simulated annealing inversion of transient electromagnetic

Geophysical inversion usually involves ill-posed problem. Regularization is the most commonly used method to mitigate this problem. There are many regularization parameter selection methods, among which the adaptive regularization method can automatically update parameters during iteration, reducing the difficulty of parameter selection. Therefore, it is widely used in linear inversion. However, there are very few studies on the use of adaptive regularization methods in stochastic optimization algorithms. The biggest difficulty is that in stochastic optimization algorithms, the search direction of any iteration is completely random. Data fitting term and stabilizing term vary in a wide range, making it difficult for traditional methods to work. In this paper, we consider the contributions of the data fitting term and the stabilizing term in the objective function and give an improved adaptive regularization method for very fast simulated annealing (VFSA) inversion for transient electromagnetic (TEM) data. The optimized method adjusts the two terms dynamically to make them in balance. We have designed several numerical experiments, and the experimental results demonstrate that the method in this paper not only accelerates the convergence, but also the inversion results are very little affected by the initial regularization parameter. Finally, we apply this method to field data, and the inversion results show very good agreements with nearby borehole data.

NH3 Emissions and Lifetime Estimated by Satellite Observations with Differential Evolution Algorithm

As an important irritant trace gas in the atmosphere, ammonia (NH3) significantly impacts human health and environment. Bottom-up emission inventories are widely used to estimate ammonia emissions and their geographical distributions over China. However, high uncertainties are still associated with emission inventories due to inaccurate emission factors used. The Differential Evolution (DE) algorithm is a population-based stochastic optimization algorithm used to solve complicated optimization problems. We quantify NH3 emissions and lifetime from Infrared Atmospheric Sounding Interferometer (IASI) NH3 observations together with MERRA-2 wind fields based on the DE algorithm. Two inland cities, Urumchi and Golmud in China, are chosen to study of the NH3 emissions based on the distributions of NH3 total columns and wind fields. The NH3 emissions rate estimated is about 5.84 × 10−11 and 4.19 × 10−11 kg·m−2s−1 in Urumchi and in the Golmud area from May to September from 2008 to 2023, respectively. The lifetime of NH3 estimated in the two areas is 4.31 and 9.19 h, respectively. We compare the NH3 emissions and lifetime estimated in this study with the values in other studies, and the results show the reliability of the method used. This work is one of few quantitative studies of NH3 emissions from cities using satellite observations in China.

An Algorithm Based on Compute Unified Device Architecture for Estimating Covering Functionals of Convex Bodies

In Chuanming Zong’s program to attack Hadwiger’s covering conjecture, which is a longstanding open problem from Convex and Discrete Geometry, it is essential to estimate covering functionals of convex bodies effectively. Recently, He et al. and Yu et al. provided two deterministic global optimization algorithms having high computational complexity for this purpose. Since satisfactory estimations of covering functionals will be sufficient in Zong’s program, we propose a stochastic global optimization algorithm based on CUDA and provide an error estimation for the algorithm. The accuracy of our algorithm is tested by comparing numerical and exact values of covering functionals of convex bodies including the Euclidean unit disc, the three-dimensional Euclidean unit ball, the regular tetrahedron, and the regular octahedron. We also present estimations of covering functionals for the regular dodecahedron and the regular icosahedron.

Online Distributed Stochastic Gradient Algorithm for Nonconvex Optimization With Compressed Communication

Online Distributed Stochastic Gradient Algorithm for Nonconvex Optimization With Compressed Communication

On the Inherent Distributional Robustness of Stochastic and Nominal Model Predictive Control

We define a notion of distributional robustness, via the Wasserstein metric, for closed-loop systems subject to errors in the disturbance distribution used to construct the controller. We then establish sufficient conditions for stochastic model predictive control (SMPC) to satisfy this definition of distributional robustness and establish a similar notion of distributional robustness for economic applications of SMPC. These results address incorrectly or unmodeled disturbances, demonstrate the efficacy of scenario optimization as a means to approximate and solve the SMPC problem, and unify the descriptions of robustness for stochastic and nominal model predictive control. This definition of distributional robustness for closed-loop systems is general and can be applied to other stochastic optimal control algorithms and, potentially, the developing field of distributionally robust control.

Scientometric analysis of quantum-inspired metaheuristic algorithms

Quantum algorithms, based on the principles of quantum mechanics, offer significant parallel processing capabilities with a wide range of applications. Nature-inspired stochastic optimization algorithms have long been a research hotspot. The fusion of quantum mechanics with optimization methods can potentially address NP-hard problems more efficiently and exponentially faster. The potential advantages provided by the ground-breaking paradigm have expedited the scientific output of quantum-inspired optimization algorithms locale. Consequently, a pertinent investigation is required to explain how ground-breaking scientific advancements have evolved. The scientometric approach utilizes quantitative and qualitative techniques to analyze research publications to evaluate the structure of scientific knowledge. Henceforth, the current research presents a scientometric and systematic analysis of quantum-inspired metaheuristic algorithms (QiMs) literature from the Scopus database since its inception. The scientometric implications of the article offer a detailed exploration of the publication patterns, keyword co-occurrence network analysis, author co-citation analysis and country collaboration analysis corresponding to each opted category of QiMs. The analysis reveals that QiMs solely account to 26.66% of publication share in quantum computing and have experienced an impressive 42.59% growth rate in the past decade. Notably, power management, adiabatic quantum computation, and vehicle routing are prominent emerging application areas. An extensive systematic literature analysis identifies key insights and research gaps in the QiMs knowledge domain. Overall, the findings of the current article provide scientific cues to researchers and the academic fraternity for identifying the intellectual landscape and latest research trends of QiMs, thereby fostering innovation and informed decision-making.

Phase Transition in Ant Colony Optimization

Ant colony optimization (ACO) is a stochastic optimization algorithm inspired by the foraging behavior of ants. We investigate a simplified computational model of ACO, wherein ants sequentially engage in binary decision-making tasks, leaving pheromone trails contingent upon their choices. The quantity of pheromone left is the number of correct answers. We scrutinize the impact of a salient parameter in the ACO algorithm, specifically, the exponent α, which governs the pheromone levels in the stochastic choice function. In the absence of pheromone evaporation, the system is accurately modeled as a multivariate nonlinear Pólya urn, undergoing phase transition as α varies. The probability of selecting the correct answer for each question asymptotically approaches the stable fixed point of the nonlinear Pólya urn. The system exhibits dual stable fixed points for α≥αc and a singular stable fixed point for α<αc where αc is the critical value. When pheromone evaporates over a time scale τ, the phase transition does not occur and leads to a bimodal stationary distribution of probabilities for α≥αc and a monomodal distribution for α<αc.

Stochastic first-order learning for large-scale flexibly tied Gaussian mixture models

Gaussian Mixture Models (GMMs) are one of the most potent parametric density models used extensively in many applications. Flexibly-tied factorization of the covariance matrices in GMMs is a powerful approach for coping with the challenges of common GMMs when faced with high-dimensional data and complex densities which often demand a large number of Gaussian components. However, the expectation–maximization algorithm for fitting flexibly-tied GMMs still encounters difficulties with streaming and very large dimensional data. To overcome these challenges, this paper suggests the use of first-order stochastic optimization algorithms. Specifically, we propose a new stochastic optimization algorithm on the manifold of orthogonal matrices. Through numerous empirical results on both synthetic and real datasets, we observe that stochastic optimization methods can outperform the expectation–maximization algorithm in terms of attaining better likelihood, needing fewer epochs for convergence, and consuming less time per each epoch.

Simulation and evaluation study of atmospheric aerosol nonsphericity as a function of particle size

Aerosol nonsphericity causes great uncertainty in radiative forcing assessments and climate simulations. Although considerable studies have attempted to quantify this uncertainty, the relationship between aerosol nonsphericity and particle size is usually not considered, thus reducing the accuracy of the results. In this study, a coupled inversion algorithm combining an improved stochastic particle swarm optimization algorithm and angular light scattering is used for the nonparametric estimation of aerosol nonsphericity variation with particle size, and the optimal sample selection method is employed to screen the data. Based on the verification of inversion accuracy, the variation of aerosol aspect ratio with particle size based on the ellipsoidal model in global regions has been obtained from Aerosol Robotic Network (AERONET) data, and the effect of nonsphericity on radiative forcing and dry deposition has been studied. The results show that the aspect ratio increases with particle size in all regions, with the maximum ranging from 1.4 to 1.8 in the desert, reflecting the differences in aerosol composition at different particle sizes. In radiation calculations, considering aerosol nonsphericity makes the aerosol cooling effect weaker and surface radiative fluxes increase, but hardly changes the aerosol absorption, with maximum differences of 9.22% and 22.12% at the bottom and top of the atmosphere, respectively. Meanwhile, the differences in radiative forcing between aspect ratios as a function of particle size and not varying with particle size are not significant, averaging less than 2%. Besides, the aspect ratio not varying with particle size underestimates the deposition velocity of small particles and overestimates that of large particles compared to that as a function of particle size, with maximum differences of 7% and 4%, respectively.

Distributed Stochastic Projection-Free Algorithm for Constrained Optimization

Distributed Stochastic Projection-Free Algorithm for Constrained Optimization

Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One.

Graph neural networks (GNN) suffer from severe inefficiency due to the exponential growth of node dependency with the increase of layers. It extremely limits the application of stochastic optimization algorithms so that the training of GNN is usually time-consuming. To address this problem, we propose to decouple a multi-layer GNN as multiple simple modules for more efficient training, which is comprised of classical forward training (FT) and designed backward training (BT). Under the proposed framework, each module can be trained efficiently in FT by stochastic algorithms without distortion of graph information owing to its simplicity. To avoid the only unidirectional information delivery of FT and sufficiently train shallow modules with the deeper ones, we develop a backward training mechanism that makes the former modules perceive the latter modules, inspired by the classical backward propagation algorithm. The backward training introduces the reversed information delivery into the decoupled modules as well as the forward information delivery. To investigate how the decoupling and greedy training affect the representational capacity, we theoretically prove that the error produced by linear modules will not accumulate on unsupervised tasks in most cases. The theoretical and experimental results show that the proposed framework is highly efficient with reasonable performance, which may deserve more investigation.

Sensor Set Decomposition for Enhanced Distributed Sensor Fault Isolability of Marine Propulsion Systems

This paper proposes a greedy stochastic optimization algorithm for the sensor set decomposition used in the sensor fault monitoring of marine propulsion systems, based on fault isolability criteria. These criteria are expressed mathematically in terms of the number of unique columns in the theoretical fault signature matrices (FSMs) used during the sensor fault isolation process. Due to the large scale and complexity of marine propulsion plants, the diagnostic layer follows a distributed architecture with a combinatorial logic used for fault isolation in two cyber levels; the local and global decision logic. As a result, the FSMs of both levels are formulated as an integrated optimization problem. Each solution regarding the sensor set decomposition is then used to generate the respective distributed monitoring architecture, using semantic (qualitative) knowledge for the propulsion plant. Thus, the need for an analytical model of the plant is removed. Moreover, based on the design of the distributed monitoring architecture, the respective theoretical FSMs (quantitative) are Automatically generated and used for the evaluation of the objective function. Finally, simulation results are used to illustrate the application of the greedy stochastic optimization algorithm and its efficiency.