Stochastic Optimization Algorithm Research Articles

Distributed stochastic optimization algorithm with non-consistent constraints in time-varying unbalanced networks

This study focuses on a class of distributed optimization problems with non-consistent local set constraints in time-varying unbalanced networks. Considering that each agent only has access to local gradient information with random errors, this paper proposes a distributed stochastic gradient projection algorithm. Under some conditions of random errors and a uniformly joint strongly connected topology, it is shown that the local decision states of all agents converge to a common optimal solution with a probability of one, achieving a sublinear convergence rate of O(lnk/k). Notably, under the condition that at least one local function exhibits strong convexity, the algorithm achieves a faster sublinear convergence rate of O(1/k). These theoretical results are validated through numerical simulations. Furthermore, this paper applies the proposed algorithm to solve for the unknown parameters in distributed linear regression problems with incomplete data. Numerical results indicate that the algorithm not only aligns with the convergence properties reported in existing literature but also demonstrates significant superiority in convergence speed. This important discovery not only lays a solid theoretical foundation for the further application and extension of the algorithm but also reveals its immense potential in solving practical problems.

An Improved Reacceleration Optimization Algorithm Based on the Momentum Method for Image Recognition

The optimization algorithm plays a crucial role in image recognition by neural networks. However, it is challenging to accelerate the model’s convergence and maintain high precision. As a commonly used stochastic gradient descent optimization algorithm, the momentum method requires many epochs to find the optimal parameters during model training. The velocity of its gradient descent depends solely on the historical gradients and is not subject to random fluctuations. To address this issue, an optimization algorithm to enhance the gradient descent velocity, i.e., the momentum reacceleration gradient descent (MRGD), is proposed. The algorithm utilizes the point division of the current momentum and the gradient relationship, multiplying it with the gradient. It can adjust the update rate and step size of the parameters based on the gradient descent state, so as to achieve faster convergence and higher precision in training the deep learning model. The effectiveness of this method is further proven by applying the reacceleration mechanism to the Adam optimizer, resulting in the MRGDAdam algorithm. We verify both algorithms using multiple image classification datasets, and the experimental results show that the proposed optimization algorithm enables the model to achieve higher recognition accuracy over a small number of training epochs, as well as speeding up model implementation. This study provides new ideas and expansions for future optimizer research.

Wear parameters identification of hydrodynamic bearing based on operational modal analysis and on-rotor sensing technology

Abrasive wear is a predominant fault in hydrodynamic bearings that can deteriorate bearing performance and damage the rotating machine eventually. The present work proposes a novel wear parameters identification method for hydrodynamic bearings based on operational modal analysis (OMA) and on-rotor sensing (ORS) technology. With this framework, the objective function is defined as the sum of squared errors between the theoretical modal frequencies and the identified modal frequencies. For the experiment verification, the ORS technology is used to collect rotor vibration directly. Additionally, the cepstrum editing procedure (CEP) is applied to enhance modal information. The different wear parameters are successfully identified using combined stochastic subspace identification (SSI) and particle swarm optimization (PSO) algorithms in both simulation and experimental signals.

Minimizing voltage fluctuation in stand-alone microgrid system using a Kriging-based multi-objective stochastic optimization algorithm

Minimizing voltage fluctuation in stand-alone microgrid system using a Kriging-based multi-objective stochastic optimization algorithm

A Review of Stochastic Optimization Algorithms Applied in Food Engineering

A Review of Stochastic Optimization Algorithms Applied in Food Engineering

A Stochastic Inexact Sequential Quadratic Optimization Algorithm for Nonlinear Equality-Constrained Optimization

A stochastic algorithm is proposed, analyzed, and tested experimentally for solving continuous optimization problems with nonlinear equality constraints. It is assumed that constraint function and derivative values can be computed but that only stochastic approximations are available for the objective function and its derivatives. The algorithm is of the sequential quadratic optimization variety. Distinguishing features of the algorithm are that it only employs stochastic objective gradient estimates that satisfy a relatively weak set of assumptions (while using neither objective function values nor estimates of them) and that it allows inexact subproblem solutions to be employed, the latter of which is particularly useful in large-scale settings when the matrices defining the subproblems are too large to form and/or factorize. Conditions are imposed on the inexact subproblem solutions that account for the fact that only stochastic objective gradient estimates are employed. Convergence results are established for the method. Numerical experiments show that the proposed method vastly outperforms a stochastic subgradient method and can outperform an alternative sequential quadratic programming algorithm that employs highly accurate subproblem solutions in every iteration. Funding: This material is based upon work supported by the National Science Foundation [Awards CCF-1740796 and CCF-2139735] and the Office of Naval Research [Award N00014-21-1-2532].

Improve robustness of machine learning via efficient optimization and conformal prediction

AbstractThe advance of machine learning (ML) systems in real‐world scenarios usually expects safe deployment in high‐stake applications (e.g., medical diagnosis) for critical decision‐making process. To this end, provable robustness of ML is usually required to measure and understand how reliable the deployed ML system is and how trustworthy their predictions can be. Many studies have been done to enhance the robustness in recent years from different angles, such as variance‐regularized robust objective functions and conformal prediction (CP) for uncertainty quantification on testing data. Although these tools provably improve the robustness of ML model, there is still an inevitable gap to integrate them into an end‐to‐end deployment. For example, robust objectives usually require carefully designed optimization algorithms, while CP treats ML models as black boxes. This paper is a brief introduction to our recent research focusing on filling this gap. Specifically, for learning robust objectives, we designed sample‐efficient stochastic optimization algorithms that achieves the optimal (or faster compared to existing algorithms) convergence rates. Moreover, for CP‐based uncertainty quantification, we established a framework to analyze the expected prediction set size (smaller size means more efficiency) of CP methods in both standard and adversarial settings. This paper elaborates the key challenges and our exploration towards efficient algorithms with details of background methods, notions for robustness measure, concepts of algorithmic efficiency, our proposed algorithms and results. All of them further motivate our future research on risk‐aware ML that can be critical for AI–human collaborative systems. The future work mainly targets designing conformal robust objectives and their efficient optimization algorithms.

Adaptive switching and routing protocol design and optimization in internet of things based on probabilistic models

Through smart cities, Intelligent Transportation Systems (ITS), the agricultural sector, and wearable devices, the Internet of Things (IoT) has revolutionized several human interests. Through the development of new cluster tasks, the Decision-Making System (DMS) of Cluster Heads (CHs), and improving the accuracy of traffic prediction and reliability of transportation, the present study intends to improve the energy depletion of IoT devices. The paper explores the subject of data flow optimization using Fuzzy Assisted Cuckoo Search Optimization (FACSO), traffic flow using Gaussian Process Regression (GPR), and CH prediction using the Stochastic Optimization Algorithm (SOA). Optimizing network lifetime while minimizing Energy Consumption (EC) is feasible through the practical application of the SOA, GPR, and FACSO models. Increasing End-to-End Delay (EED), Network Throughput (NT), and energy efficiency can be rendered feasible through a real-time DMS regarding routing employing a novel approach referred to as FACSO. This approach has enhanced the efficacy and reliability of Wireless Sensor Networks (WSN). With up to 500 nodes and an EC of 0.3451 J, the experiment's findings demonstrate that a proposed SOA-FACSO model achieves superior EED.

Distributed gradient-free and projection-free algorithm for stochastic constrained optimization

Distributed stochastic zeroth-order optimization (DSZO), in which the objective function is allocated over multiple agents and the derivative of cost functions is unavailable, arises frequently in large-scale machine learning and reinforcement learning. This paper introduces a distributed stochastic algorithm for DSZO in a projection-free and gradient-free manner via the Frank-Wolfe framework and the stochastic zeroth-order oracle (SZO). Such a scheme is particularly useful in large-scale constrained optimization problems where calculating gradients or projection operators is impractical, costly, or when the objective function is not differentiable everywhere. Specifically, the proposed algorithm, enhanced by recursive momentum and gradient tracking techniques, guarantees convergence with just a single batch per iteration. This significant improvement over existing algorithms substantially lowers the computational complexity. Under mild conditions, we prove that the complexity bounds on SZO of the proposed algorithm are O(n/ϵ2)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{O}(n/\\epsilon ^{2})$\\end{document} and O(n(21ϵ))\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{O}(n(2^{\\frac{1}{\\epsilon}}))$\\end{document} for convex and nonconvex cases, respectively. The efficacy of the algorithm is verified on black-box binary classification problems against several competing alternatives.

Efficient mini-batch stochastic gradient descent with Centroidal Voronoi Tessellation for PDE-constrained optimization under uncertainty

The study of optimal control problems under uncertainty plays an important role in scientific numerical simulations. This class of optimization problems is frequently utilized in engineering, biology and finance. Although stochastic gradient descent is a well-known stochastic optimization technique for solving the approximate optimal control problem, it often exhibits a slow convergence rate due to the inherent variance in gradient approximation. To address this issue, we propose a more efficient stochastic optimization algorithm based on Centroidal Voronoi Tessellation sampling. This strategy reduces the error in gradient estimation compared to the conventional mini-batch stochastic gradient descent method. Our approach involves dividing the whole snapshot set into several Voronoi cells with low variance and extracting samples with good uniformity from each region to construct an unbiased estimation of the full gradient. Our method can economically and stably approximate numerical optimal control functions even with a fixed step size. Numerical results demonstrate that the proposed method is a reliable algorithm with great potential for applications in the field of optimization.

Simultaneous Reconstruction of Multiple Time-Varying Thermal Properties Based on Translucent Materials.

In the realm of high-tech materials and energy applications, accurately measuring the transient heat flow at media boundaries and the internal thermal conductivity of materials in harsh heat exchange environments poses a significant challenge when using conventional direct measurement methods. Consequently, the study of photothermal parameter reconstruction in translucent media, which relies on indirect measurement techniques, has crucial practical value. Current research on reconstructing photothermal properties within participating media typically focuses on single-objective or time-invariant properties. There is a pressing need to develop effective methods for the simultaneous reconstruction of time-varying thermal flow fields and internal thermal conductivity at the boundaries of participating media. This paper introduces a computational model based on the numerical simulation theory of internal heat transfer systems in participating media, stochastic particle swarm optimization algorithms, and Kalman filter technology. The model aims to enable the simultaneous reconstruction of various thermal parameters within the target medium. Our results demonstrate that under varying levels of measurement noise, the inversion results for different target parameters exhibit slight oscillations around the true values, leading to a reduction in reconstruction accuracy. However, overall, the model demonstrates robustness and accuracy in ideal conditions, validating its effectiveness.

Simultaneous port number and location optimization of mold gates‐vents using in‐house coded stochastic multi‐objective optimization algorithm

AbstractThis research introduces multi‐objective stochastic optimization (MOSO) and non‐dominated sorting differential evolution (NSDE) in‐house coded algorithms for enhancing the optimization of the resin transfer molding (RTM) process during the mold‐fill phase for composite parts. Glass fiber‐vinyl ester‐based automotive bonnet and carbon fiber‐RTM6‐based aircraft wing flap parts were used as the case studies. Initially, the NSDE algorithm was developed for the simultaneous optimization of dry‐spot content and mold‐fill time by changing the locations of gates and vents with pre‐fixed port numbers. Then, the MOSO algorithm was designed for the concurrent optimization of the dry‐spot content, mold‐fill time and total number of ports by changing both the numbers and locations of gates and vents. The effect of race‐tracking was also investigated using higher permeability values at the composite part‐cut edges. When compared, the MOSO algorithm predicted less dry‐spot content, number of ports, mold‐fill time and uniform resin flow‐front progressions with lesser functional evaluations and computational time than the NSDE algorithm.Highlights The non‐dominated sorting differential evolution (NSDE) and multi‐objective stochastic optimization (MOSO) algorithms were developed for the optimization of the mold fill phase of resin transfer molded composite parts. The development of the NSDE algorithm aimed to concurrently optimize the dry‐spot content and mold‐fill time by changing the locations of gates and vents with pre‐fixed port numbers. The MOSO algorithm was designed to concurrently optimize dry‐spot content, mold‐fill time, and the total number of ports by varying both the numbers and locations of gates and vents. The MOSO algorithm predicted better‐automated mold‐fill phase optimalities for the resin transfer molded composite parts compared to the NSDE algorithm.

Optimal proportional-integral speed control for closed-loop engine timing system

In internal combustion engines, adjusting the air-fuel ratio is essential to control the speed and minimize the burnt fuel. The throttle opening is the actuator to control the air-fuel. A better design for the used/conventional controller can give a better response without additional cost. In this work, the proposed controller gains of the proportional-integral (PI) controller are tuned to enhance the speed in constant and variable drive cycle modes. The tuning process is conducted based on two of the most efficient performance indices used in this field. The performance indices are integral absolute error (IAE) and integral time absolute error (ITAE). The optimization problem is solved using three reliable stochastic optimization algorithms to ensure mature convergence of the solutions, to avoid local optima solutions, and to ensure effective shrinking of the search space. The optimization algorithms are teaching-learning-based optimization (TLBO), particle swarm optimization (PSO), and genetic algorithm (GA). Different simulations are conducted to validate the results. The results are compared with conventional tuning methods regarding the system's time response.

Improved adaptive regularization for simulated annealing inversion of transient electromagnetic

Geophysical inversion usually involves ill-posed problem. Regularization is the most commonly used method to mitigate this problem. There are many regularization parameter selection methods, among which the adaptive regularization method can automatically update parameters during iteration, reducing the difficulty of parameter selection. Therefore, it is widely used in linear inversion. However, there are very few studies on the use of adaptive regularization methods in stochastic optimization algorithms. The biggest difficulty is that in stochastic optimization algorithms, the search direction of any iteration is completely random. Data fitting term and stabilizing term vary in a wide range, making it difficult for traditional methods to work. In this paper, we consider the contributions of the data fitting term and the stabilizing term in the objective function and give an improved adaptive regularization method for very fast simulated annealing (VFSA) inversion for transient electromagnetic (TEM) data. The optimized method adjusts the two terms dynamically to make them in balance. We have designed several numerical experiments, and the experimental results demonstrate that the method in this paper not only accelerates the convergence, but also the inversion results are very little affected by the initial regularization parameter. Finally, we apply this method to field data, and the inversion results show very good agreements with nearby borehole data.

NH3 Emissions and Lifetime Estimated by Satellite Observations with Differential Evolution Algorithm

As an important irritant trace gas in the atmosphere, ammonia (NH3) significantly impacts human health and environment. Bottom-up emission inventories are widely used to estimate ammonia emissions and their geographical distributions over China. However, high uncertainties are still associated with emission inventories due to inaccurate emission factors used. The Differential Evolution (DE) algorithm is a population-based stochastic optimization algorithm used to solve complicated optimization problems. We quantify NH3 emissions and lifetime from Infrared Atmospheric Sounding Interferometer (IASI) NH3 observations together with MERRA-2 wind fields based on the DE algorithm. Two inland cities, Urumchi and Golmud in China, are chosen to study of the NH3 emissions based on the distributions of NH3 total columns and wind fields. The NH3 emissions rate estimated is about 5.84 × 10−11 and 4.19 × 10−11 kg·m−2s−1 in Urumchi and in the Golmud area from May to September from 2008 to 2023, respectively. The lifetime of NH3 estimated in the two areas is 4.31 and 9.19 h, respectively. We compare the NH3 emissions and lifetime estimated in this study with the values in other studies, and the results show the reliability of the method used. This work is one of few quantitative studies of NH3 emissions from cities using satellite observations in China.

An Algorithm Based on Compute Unified Device Architecture for Estimating Covering Functionals of Convex Bodies

In Chuanming Zong’s program to attack Hadwiger’s covering conjecture, which is a longstanding open problem from Convex and Discrete Geometry, it is essential to estimate covering functionals of convex bodies effectively. Recently, He et al. and Yu et al. provided two deterministic global optimization algorithms having high computational complexity for this purpose. Since satisfactory estimations of covering functionals will be sufficient in Zong’s program, we propose a stochastic global optimization algorithm based on CUDA and provide an error estimation for the algorithm. The accuracy of our algorithm is tested by comparing numerical and exact values of covering functionals of convex bodies including the Euclidean unit disc, the three-dimensional Euclidean unit ball, the regular tetrahedron, and the regular octahedron. We also present estimations of covering functionals for the regular dodecahedron and the regular icosahedron.

Online Distributed Stochastic Gradient Algorithm for Nonconvex Optimization With Compressed Communication

Online Distributed Stochastic Gradient Algorithm for Nonconvex Optimization With Compressed Communication

On the Inherent Distributional Robustness of Stochastic and Nominal Model Predictive Control

We define a notion of distributional robustness, via the Wasserstein metric, for closed-loop systems subject to errors in the disturbance distribution used to construct the controller. We then establish sufficient conditions for stochastic model predictive control (SMPC) to satisfy this definition of distributional robustness and establish a similar notion of distributional robustness for economic applications of SMPC. These results address incorrectly or unmodeled disturbances, demonstrate the efficacy of scenario optimization as a means to approximate and solve the SMPC problem, and unify the descriptions of robustness for stochastic and nominal model predictive control. This definition of distributional robustness for closed-loop systems is general and can be applied to other stochastic optimal control algorithms and, potentially, the developing field of distributionally robust control.

Scientometric analysis of quantum-inspired metaheuristic algorithms

Quantum algorithms, based on the principles of quantum mechanics, offer significant parallel processing capabilities with a wide range of applications. Nature-inspired stochastic optimization algorithms have long been a research hotspot. The fusion of quantum mechanics with optimization methods can potentially address NP-hard problems more efficiently and exponentially faster. The potential advantages provided by the ground-breaking paradigm have expedited the scientific output of quantum-inspired optimization algorithms locale. Consequently, a pertinent investigation is required to explain how ground-breaking scientific advancements have evolved. The scientometric approach utilizes quantitative and qualitative techniques to analyze research publications to evaluate the structure of scientific knowledge. Henceforth, the current research presents a scientometric and systematic analysis of quantum-inspired metaheuristic algorithms (QiMs) literature from the Scopus database since its inception. The scientometric implications of the article offer a detailed exploration of the publication patterns, keyword co-occurrence network analysis, author co-citation analysis and country collaboration analysis corresponding to each opted category of QiMs. The analysis reveals that QiMs solely account to 26.66% of publication share in quantum computing and have experienced an impressive 42.59% growth rate in the past decade. Notably, power management, adiabatic quantum computation, and vehicle routing are prominent emerging application areas. An extensive systematic literature analysis identifies key insights and research gaps in the QiMs knowledge domain. Overall, the findings of the current article provide scientific cues to researchers and the academic fraternity for identifying the intellectual landscape and latest research trends of QiMs, thereby fostering innovation and informed decision-making.

Phase Transition in Ant Colony Optimization

Ant colony optimization (ACO) is a stochastic optimization algorithm inspired by the foraging behavior of ants. We investigate a simplified computational model of ACO, wherein ants sequentially engage in binary decision-making tasks, leaving pheromone trails contingent upon their choices. The quantity of pheromone left is the number of correct answers. We scrutinize the impact of a salient parameter in the ACO algorithm, specifically, the exponent α, which governs the pheromone levels in the stochastic choice function. In the absence of pheromone evaporation, the system is accurately modeled as a multivariate nonlinear Pólya urn, undergoing phase transition as α varies. The probability of selecting the correct answer for each question asymptotically approaches the stable fixed point of the nonlinear Pólya urn. The system exhibits dual stable fixed points for α≥αc and a singular stable fixed point for α<αc where αc is the critical value. When pheromone evaporates over a time scale τ, the phase transition does not occur and leads to a bimodal stationary distribution of probabilities for α≥αc and a monomodal distribution for α<αc.