Stochastic Optimization Methods Research Articles

Advancing Parameter Estimation in Differential Equations: A Hybrid Approach Integrating Levenberg–Marquardt and Luus–Jaakola Algorithms

This study presents an integrated approach that combines the Levenberg–Marquardt (LM) and Luus–Jaakola (LJ) algorithms to enhance parameter estimation for various applications. The LM algorithm, known for its precision in solving non-linear least squares problems, is effectively paired with the LJ algorithm, a robust stochastic optimization method, to improve accuracy and computational efficiency. This hybrid LM-LJ methodology is demonstrated through several case studies, including the optimization of MESH equations in distillation processes, modeling controlled diffusion in biopolymer films, and analyzing heat and mass transfer during the drying of cylindrical quince slices. By overcoming the convergence issues typical of gradient-based methods and performing global searches without initial parameter bounds, this approach effectively handles complex models and closely aligns simulation results with experimental data. These capabilities highlight the versatility of this approach in engineering and environmental modeling, significantly enhancing parameter estimation in systems governed by differential equations.

Two‐stage stochastic energy and reserve market clearing model considering real‐time offering strategy of renewable energy

AbstractWith the introduction of decarbonization policies worldwide, renewable energy, especially wind and solar power, is gradually taking a significant share in the power system. The two‐stage stochastic optimization methods are generally adopted to address the uncertainty issues and the two‐stage stochastic market clearing models are established with the participation of renewable energy. However, most existing two‐stage stochastic market clearing models usually intuitively presume the offering quantities of renewable energy are equal to their forecast values, neglecting the potential impacts of the strategic offerings on the market results and thus affecting the secure operation of power systems. To fill the gaps, this paper incorporates the strategic offering models of renewable energy into the two‐stage stochastic energy and reserve market clearing model. Within the two‐stage framework, the first day‐ahead stage serves for energy and reserve market clearing, while the second real‐time stage considers the adjustments on offering quantities of renewable energy according to the fluctuations of renewable generations. The two‐stage stochastic market clearing model is formulated as a mixed‐integer non‐linear programming problem, and an iterative algorithm is devised to obtain market clearing results after proper linearizations. The effectiveness of the proposed method is validated using the extended IEEE‐118 test system.

Model for estimating red mulberry leaf area using a genetic algorithm

The success of sericulture depends directly on the quality and quantity of mulberry leaves, as it is essential for the feeding and development of silkworm caterpillars and, consequently, influences the quality of the silk thread manufactured. The estimation of mulberry leaf area is important to have plant development and growth indicators, such as transpiration intensity, net assimilation rate, leaf area ratio, specific leaf area and leaf area index, which allow predicting crop productivity. Thus, the objective of this study was to develop and test a model capable of estimating the red mulberry leaf area using a genetic algorithm. The model was adjusted with the proposed stochastic optimization method. The mean error found for the tested dataset was approximately 228.17 mm2 in sample space with mean leaf area of 6515.55 mm2. The information generated allows applying the model to estimate red mulberry leaf area in future studies.

Stochastic two-stage multi-objective unit commitment of distributed resource energy systems considering uncertainties and unit failures

Compared to centralized generation technology, distributed energy resource systems are susceptible to energy risks caused by boundary uncertainties and unit failures. This study introduces a stochastic two-stage multi-objective optimization method to address reliability-based unit commitment issues. In the day-ahead stage, operational state and reserve capacity are determined to minimize prescheduled operation costs based on forecasted parameters. In the real-time stage, a decision-dependent stochastic reliability method is proposed to simulate outage scenarios. Reserve resources within available units are allocated to mitigate forecasting errors and unit failures. Additionally, the grid interaction ratio and penalty cost are added to restrict the depth and frequency access to the grid. Four comparative cases analyze the effects of the proposed methodology. This method innovatively achieves the simulation of stochastic multi-unit outages and delete faulty units in the operation scheme. The optimal results show that the risks of electricity and cooling supply are underestimated, while the risks of heating are overestimated, compared to N-1 reliability. Furthermore, Pareto analysis of the multi-objective problem enhances independent operational capacity through utilization of reserve resources. Grid dispatch pressure is reduced since purchased power can be used as day-ahead planning. Thus, the methodology achieves collaborative optimization of reliability with a reduction of operation costs, offering effective guidance for engineering applications.

Optimization by linear kinetic equations and mean-field Langevin dynamics

One of the most striking examples of the close connections between global optimization processes and statistical physics is the simulated annealing method, inspired by the famous Monte Carlo algorithm devised by Metropolis et al. in the middle of last century. In this paper, we show how the tools of linear kinetic theory allow the description of this gradient-free algorithm from the perspective of statistical physics and how convergence to the global minimum can be related to classical entropy inequalities. This analysis highlights the strong link between linear Boltzmann equations and stochastic optimization methods governed by Markov processes. Thanks to this formalism, we can establish the connections between the simulated annealing process and the corresponding mean-field Langevin dynamics characterized by a stochastic gradient descent approach. Generalizations to other selection strategies in simulated annealing that avoid the acceptance–rejection dynamic are also provided.

A Barrier Lagrangian Dual Method for Multi-stage Stochastic Convex Semidefinite Optimization

A Barrier Lagrangian Dual Method for Multi-stage Stochastic Convex Semidefinite Optimization

Optimal Battery Storage Configuration for High-Proportion Renewable Power Systems Considering Minimum Inertia Requirements

With the continuous development of renewable energy worldwide, the issue of frequency stability in power systems has become increasingly serious. Enhancing the inertia level of power systems by configuring battery storage to provide virtual inertia has garnered significant research attention in academia. However, addressing the non-linear characteristics of frequency stability constraints, which complicate model solving, and managing the uncertainties associated with renewable energy and load, are the main challenges in planning energy storage for high-proportion renewable power systems. In this context, this paper proposes a battery storage configuration model for high-proportion renewable power systems that considers minimum inertia requirements and the uncertainties of wind and solar power. First, frequency stability constraints are transformed into minimum inertia constraints, primarily considering the rate of change of frequency (ROCOF) and nadir frequency (NF) indicators during the transformation process. Second, using historical wind and solar data, a time-series probability scenario set is constructed through clustering methods to model the uncertainties of wind and solar power. A stochastic optimization method is then adopted to establish a mixed-integer linear programming (MILP) model for the battery storage configuration of high-proportion renewable power systems, considering minimum inertia requirements and wind-solar uncertainties. Finally, through a modified IEEE-39 bus system, it was verified that the proposed method is more economical in addressing frequency stability issues in power systems with a high proportion of renewable energy compared to traditional scheduling methods.

A new strategy for manufacturing, modeling, and optimization of 3D printed polylactide based on multiple nonlinear neuro regression analysis and stochastic optimization methods

Although there are many studies related to design, modeling, and optimization of fused deposition modeling (FDM) process parameters in the literature, the absence of a systematic approach to increase the reliability of model selection and optimization results is an important shortcoming that must be addressed. To make up for this deficiency, a new strategy was proposed to obtain desired quality on mechanical properties by adjusting FDM process parameters. This attempt involves manufacturing, modeling, and optimization point of view. The D-optimal method was employed to form experiment set, including process parameters. A hybrid approach neuro regression combining artificial neural network (ANN) and regression analysis together was used for modeling of FDM process. The most significant advantage of the neuro-regression approach compared to ANN is that the mathematical models can be used directly without needing any transformation. This is not possible in neural networks and, therefore, significantly limits and complicates the use of models obtained using ANN. The present study aims at optimization of the FDM process parameters, including infill density, infill pattern, layer thickness, and print speed on ultimate strength, fracture strength, and fracture strain for polylactide (PLA). In this regard, modified versions of the optimization algorithms Differential Evolution, Nelder Mead, and Simulated Annealing were used to find the best or elite designs. Linear or nonlinear models consisting of polynomial, trigonometric, and logarithmic expressions and their hybrid forms were employed to define the strength and strain behavior of PLA. It is concluded that (a) implementations of the optimization algorithms provide a 19% improving the minimum strain value if it is compared with the experimental results, (b) infill pattern types ( x2) were found as honeycomb, triangle, and cubic for the designs in terms of maximum fracture strength, minimum strain, and maximum ultimate tensile strength respectively, (c) many alternatives near optimum local designs could be obtained based on Nelder Mead algorithm for fracture strength and ultimate strength parameters. Thus, this allows work in a wide range of applications without depending on a single result for production.

Parallel PSO for Efficient Neural Network Training Using GPGPU and Apache Spark in Edge Computing Sets

The training phase of a deep learning neural network (DLNN) is a computationally demanding process, particularly for models comprising multiple layers of intermediate neurons.This paper presents a novel approach to accelerating DLNN training using the particle swarm optimisation (PSO) algorithm, which exploits the GPGPU architecture and the Apache Spark analytics engine for large-scale data processing tasks. PSO is a bio-inspired stochastic optimisation method whose objective is to iteratively enhance the solution to a (usually complex) problem by approximating a given objective. The expensive fitness evaluation and updating of particle positions can be supported more effectively by parallel processing. Nevertheless, the parallelisation of an efficient PSO is not a simple process due to the complexity of the computations performed on the swarm of particles and the iterative execution of the algorithm until a solution close to the objective with minimal error is achieved. In this study, two forms of parallelisation have been developed for the PSO algorithm, both of which are designed for execution in a distributed execution environment. The synchronous parallel PSO implementation guarantees consistency but may result in idle time due to global synchronisation. In contrast, the asynchronous parallel PSO approach reduces the necessity for global synchronization, thereby enhancing execution time and making it more appropriate for large datasets and distributed environments such as Apache Spark. The two variants of PSO have been implemented with the objective of distributing the computational load supported by the algorithm across the different executor nodes of the Spark cluster to effectively achieve coarse-grained parallelism. The result is a significant performance improvement over current sequential variants of PSO.

An investigation of stochastic trust-region based algorithms for finite-sum minimization

This work elaborates on the TRust-region-ish (TRish) algorithm, a stochastic optimization method for finite-sum minimization problems proposed by Curtis et al. in [F.E. Curtis, K. Scheinberg, and R. Shi, A stochastic trust region algorithm based on careful step normalization, INFORMS. J. Optim. 1(3) (2019), pp. 200–220; F.E. Curtis and R. Shi, A fully stochastic second-order trust region method, Optim. Methods Softw. 37(3) (2022), pp. 844–877]. A theoretical analysis that complements the results in the literature is presented, and the issue of tuning the involved hyper-parameters is investigated. Our study also focuses on a practical version of the method, which computes the stochastic gradient by means of the inner product test and the orthogonality test proposed by Bollapragada et al. in [R. Bollapragada, R. Byrd, and J. Nocedal, Adaptive sampling strategies for stochastic optimization, SIAM. J. Optim. 28(4) (2018), pp. 3312–3343]. It is shown experimentally that this implementation improves the performance of TRish and reduces its sensitivity to the choice of the hyper-parameters.

Stochastic Optimization of Onboard Photovoltaic Hybrid Power System Considering Environmental Uncertainties

Environmental uncertainties present a significant challenge in the design of onboard photovoltaic hybrid power systems (PV-HPS), a pivotal decarbonization technology garnering widespread attention in the shipping industry. Neglecting environmental uncertainties associated with photovoltaic (PV) output and hull resistance can lead to suboptimal solutions. To address this issue, this paper proposes a stochastic optimization method for PV-HPS, aiming to minimize greenhouse gas (GHG) emissions and lifecycle costs. Copula functions are employed to establish joint distributions of uncertainties in solar irradiance, ambient temperature, significant wave height, and wave period. Monte Carlo simulation, the bi-bin method, and the multi-objective particle swarm optimization (MOPSO) algorithm are utilized for scenario generation, scenario reduction, and design space exploration. The efficacy of the proposed method is demonstrated through a case study involving an unmanned ship. Additionally, deterministic optimization and two partial stochastic optimizations are conducted to underscore the importance of simultaneously considering environmental uncertainties related to power sources and hull resistance. The results affirm the proposed approach’s capability to reduce GHG emissions and lifecycle costs. A sensitivity analysis of bin number is performed to investigate the tradeoff between optimality and computation time.

You only compress once: Towards effective and elastic BERT compression via exploit–explore stochastic nature gradient

Despite superior performance on various natural language processing tasks, pre-trained models such as BERT are challenged by deploying on resource-constraint devices. Most existing model compression approaches require re-compression or fine-tuning across diverse constraints to accommodate various hardware deployments. This practically limits the further application of model compression. Moreover, the ineffective training and searching process of existing elastic compression paradigms (Wang et al., 2020; Cai et al., 2020) prevents the direct migration to BERT compression. Motivated by the necessity of efficient inference across various constraints on BERT, we propose a novel approach, YOCO-BERT, to achieve compress once and deploy everywhere. Specifically, we first construct a huge search space with 1013 architectures, which covers nearly all configurations in BERT model. Then, we propose a novel stochastic nature gradient optimization method to guide the generation of optimal candidate architecture which could keep a balanced trade-off between explorations and exploitation. When a certain resource constraint is given, a lightweight distribution optimization approach is utilized to obtain the optimal network for target deployment without fine-tuning. Compared with state-of-the-art algorithms, YOCO-BERT provides more compact models, yet achieving 2.1%–4.5% average accuracy improvement on the GLUE benchmark. Besides, YOCO-BERT is also more effective, e.g., the training complexity is O(1) for N different devices. Codes available https://github.com/MAC-AutoML/YOCO-BERT.

Stochastic Optimization of Distribution Networks: Application of Probabilistic Graphical Models in Electrical Grid Management

In electricity grid management, optimizing distribution networks is a must for making sure that the grid is reliable, efficient, and resilient. Stochastic optimization methods have become very useful for dealing with the unknowns that come up in grid operations because of things like adding green energy, changing demand, and broken equipment. We present a new way to improve distribution networks when there is doubt in this study. It uses probabilistic graphical models (PGMs). Using PGMs lets us describe the complicated connections between loads, producers, and grid infrastructure, as well as the relationships between these parts of the distribution network. By recording these relationships, we can accurately show how unclear the grid is and make smart choices to make it work better. In particular, we use Bayesian networks (BNs) and Markov random fields (MRFs) to describe how the different factors in the network are likely to be related to each other. We show how well our method works by using it on a real-life delivery network problem. We look at a case study of a distribution network that has a lot of green energy sources and changing load levels. We use PGMs to build a statistical model of the distribution network by combining past data, weather forecasts, and real-time measures. Then, we create a stochastic optimization problem to find the best way to reduce the predicted operational cost while still meeting different operational restrictions, like voltage limits, power balance, and equipment limitations. We use advanced optimization algorithms, like stochastic gradient descent and genetic algorithms, to quickly solve the optimization problem that was given. We show that our method works and can be scaled up for handling distribution networks when there is doubt by doing a lot of computer tests and risk analyses. The suggested method can make delivery networks much more reliable, cost-effective, and resilient than traditional linear optimization methods, as shown by our results. Overall, this study shows that probabilistic graphical models can be a very useful tool for managing the electricity grid and finding the best ways to use distribution networks in the face of randomness. Including unknowns in the modeling process helps we make stronger and more dependable choices that will help current distribution systems work well.

Estimation of P(X < Y) Stress–Strength Reliability Measures for a Class of Asymmetric Distributions: The Case of Three-Parameter p-Max Stable Laws

Asymmetric distributions are frequently seen in real-world datasets due to a number of factors, such as sample biases and nonlinear interactions between the variables observed. Thus, in order to better characterize real-world phenomena, studying asymmetric distribution is of great interest. In this work, we derive stress–strength reliability formulas of the type P(X<Y) when both X and Y follow p-max stable laws with three parameters, which are inherently asymmetric. The new relations are given in terms of extreme-value H-functions and have been obtained under fewer parameter restrictions when compared to similar results in the literature. We estimate the parameters of the p-max stable laws by a stochastic optimization method and the stress–strength probability by a maximum likelihood procedure. The performance of the analytical models is evaluated through simulations and real-life dataset modeling.

A forecast-driven stochastic optimization method for proactive activation of manual reserves

Reducing operating balancing costs is paramount for an affordable transition towards renewable-dominated power systems. In European balancing markets, operating balancing costs are driven by the activation of automatic and manual frequency restoration reserves, respectively aFRR and mFRR. An inadequate combination of both products for resolving grid imbalances may result in economic inefficiencies where, e.g., saturated aFRR can lead to balancing price spikes. To avoid such situation, we propose a proactive activation policy of manual reserves, aiming at an optimal trade-off between aFRR and mFRR products via a stochastic optimization method. The tool is fed with 1-min time trajectories of system imbalances covering the next quarter hour. The one minute temporal granularity allows modeling the ramping phenomena of mFRR products, while keeping track of the faster activation of aFRR products. The proposed balancing energy activation methodology is tested on Belgian market data, which currently adopts a reactive balancing strategy. Ex-post comparisons of the proposed balancing strategy with a reactive one show that our methodology allows a decrease of the balancing activation costs. This is expected as early activation of mFRR, when appropriately provided, allows avoiding the activation of extremely high aFRR bids.

Stochastic Models for Cyber Attack Detection and Response: A Mathematical Approach to Intrusion Detection Systems

Cyberattacks are a big problem in today's world because everything is linked online. They can damage the security, privacy, and access of private information. Intrusion Detection Systems (IDS) are very important for keeping networks safe because they quickly find and stop harmful activity. To successfully find and stop attacks, however, we need more advanced methods because online risks are always changing. This paper suggests a new mathematical approach for improving IDS that is based on random models. Traditional ways of finding intrusions often use signature- or anomaly-based methods, which might not be able to keep up with how attackers' strategies change all the time. By using random processes, our method provides a more flexible and adaptable way to find online threats. IDS can tell the difference between normal network traffic and hostile activities because stochastic models use a statistical framework to capture the uncertainty that comes with cyberattack behaviors. We use methods from probability theory, Markov chains, and queue theory to make models of how network traffic and possible cyberattacks might behave. Our random models can find differences that could mean someone is doing something bad by looking at the statistical features of different network factors, like the rate at which packets arrive, the length of a link, and the size of the payloads. Using Markov models, the IDS can also guess how likely it is that an attack will happen in the future based on past data, which helps with strategic strategies for reducing threats. Along with monitoring, our system includes ways to respond to cyberattacks and lessen their effects. By using stochastic optimization methods, we can change how resources are used and how reactions are prioritized based on how likely and how bad the threats are to be. This flexible method makes the network infrastructure more resistant to complex attack routes, which lowers the damage and downtime that can happen during cyber events. We show that our stochastic models can improve the performance of IDS in real-world situations by analyzing them theoretically and running simulations. Adopting a scientific approach to cyber security makes it possible for stronger and smarter defenses against cyber risks that are always changing.

Stochastic Capacity Optimization of an Integrated BFGCC–MSHS–Wind–Solar Energy System for the Decarbonization of a Steelmaking Plant

Deploying renewable generation to replace conventional fossil-fuel-based energy supplies provides an important pathway for the decarbonization of steelmaking plants. Meanwhile, it is also crucial to improve the flexibility of blast-furnace-gas-fired combined-cycle power plants (BFGCCs) to ease the accommodation of uncertain renewable generation. To this end, this paper proposes the deployment of a molten salt heat storage (MSHS) system in BFGCCs to store the heat of gas turbine flue gas so that the power–heat coupling of these BFGCCs can be unlocked to enhance the flexibility of the energy supply. A stochastic capacity optimization of an integrated BFGCC–MSHS–wind–solar (BMWS) energy system is presented to determine the optimal installed capacities of a BFG holder, MSHS, wind turbine, and PV panel, aiming to achieve an economic and safe energy supply for the entire system. Multiple scenarios considering uncertain fluctuations in load demands and renewable generation are generated with the Monte Carlo method based on a typical scenario. These scenarios are then reduced to representative scenarios using the synchronous substitution and reduction method for stochastic capacity optimization to enhance the reliability of the results. The case study results demonstrate that configuring MSHS reduces the total annualized cost of the BMWS system by 2.28%. Furthermore, considering the uncertainties of the power/heating load and wind/PV generation can reduce the expected annualized total cost of the BMWS system and the corresponding standard deviation by 5.66% and 81.45%, respectively. The BMWS system can achieve 730.68 tons of equivalent CO2 reduction in 24 h due to the successful utilization of renewable energy. This paper provides an effective approach for the decarbonization of energy generation systems in steelmaking plants.

Demand Response Power-Gas Interconnection Energy System and Power Metering Based on SOGA

Under the double background of global energy crisis and environmental pollution, China vigorously develops renewable energy while accelerating the construction of energy Internet. Taking demand response into account, this paper proposes the optimal design of power-gas interconnection energy system and power metering, and establishes the optimization research model of IEGES (integrated electricity-gas energy system) with demand response. Firstly, the structure model of the electric-gas interconnection energy system with CHP (Cogeneration, combined heat and power) as the core is constructed, and the energy conversion relationship and different energy flow directions of the coupling equipment are expounded from three aspects. The natural gas source point, pipeline equation, power side branch equation, voltage and current equation are modeled and sorted out, and the square term in the equation is linearized by second-order cone programming method, and the mixed integer nonlinear programming problem is transformed into mixed integer linear programming problem. A single objective genetic algorithm with “elite strategy” was selected to solve the equipment capacity optimization problem of IEGES system with system economy as the optimization objective. After a long time of parameter combination attempts, the current population size is 30, the number of iterations is 600, the crossover rate is 0.8, and the heritability is 0.3. The above parameters can obtain better convergence results on the basis of considering the operation time. Finally, a stochastic optimization method of energy Internet considering integrated demand response and uncertainty of wind power is proposed, which aims to meet the energy demand of end users while minimizing the operating cost of the system. The comprehensive demand response strategy including internal and external demand response is considered in the model. Internal demand response is realized by adjusting the internal operation mode of EH, while external demand response is implemented by the end user’s active response, and the load is time-shifted or interrupted under the guidance of external signals.

Is Learning in Biological Neural Networks Based on Stochastic Gradient Descent? An Analysis Using Stochastic Processes.

In recent years, there has been an intense debate about how learning in biological neural networks (BNNs) differs from learning in artificial neural networks. It is often argued that the updating of connections in the brain relies only on local information, and therefore a stochastic gradient-descent type optimization method cannot be used. In this article, we study a stochastic model for supervised learning in BNNs. We show that a (continuous) gradient step occurs approximately when each learning opportunity is processed by many local updates. This result suggests that stochastic gradient descent may indeed play a role in optimizing BNNs.

A two-stage collaborative optimization method for emergency repair station planning and recovery strategy of cyber-physical-transportation networks

In this paper, a two-stage collaborative stochastic optimization method for pre-disaster emergency repair station planning and post-disaster recovery strategy of cyber-physical-transportation networks is proposed in this paper to enhance the resilience under extreme scenarios. In the first stage, based on the principle of hierarchical portioning, the responsible area for the emergency repair station is divided by the community theory, and the emergency repair station planning model is developed. In the second stage, the time sequence of repairing the faulty components is first established based on the graph theory. Then, based on the spatial distribution of adjacent time-series faulty components on the transportation network, a faulty component restoration time model is established to accurately reflect the faulty restoration time. On this basis, the functional coupling effects of economic dispatch control and substation automation system failures on load recovery are analyzed to establish a load recovery model. The case studies of the IEEE RTS-79 test system demonstrate that, compared with traditional independent recovery model, the model proposed in this paper performs better in enhancing resilience under extreme scenarios.