Derivative-free Optimization Technique Research Articles

Hyperparameter Optimization of a Parallelized LSTM for Time Series Prediction

Long Short-Term Memory (LSTM) Neural Network has great potential to predict sequential data. Time series prediction is one of the most popular experimental subjects of LSTM. To this end, various LSTM algorithms have been developed to predict time series data. However, there are a few works considering the hyperparameter optimization of LSTM along with parallelization approaches. To address this problem, a parallelized classic LSTM is proposed to predict time series. In the preprocessing phase, it first replaces missing values with zero and then normalizes the time series matrix. The transposed matrix is divided into training and testing parts. Consequently, a core-based parallelism is established, thereby utilizing forking to split prediction into multiple processes. Derivative-free optimization techniques are also analyzed to find out what sort of hyperparameter optimization technique is much more feasible for a parallelized LSTM. Further, a state-of-the-art comparison is included in the study. Experimental results show that training loss is optimal when using Nelder–Mead. Employing effort-intensive optimization methods such as genetic algorithms results in a remarkable CPU time reduction in parallelized designs according to the results. Last, the proposed algorithm outperformed the comparison methods with respect to the prediction errors.

Calibration of high-fidelity hydrodynamic models utilizing on-site vessel response measurements

With a focus on weather-restricted marine operations, a comprehensive work on numerical model calibration is conducted to improve the vessel response prediction accuracy in real environmental conditions. This framework involves using existing physics-based models in which the metocean and system parameters are entered as inputs to derive vessel responses. The numerical models based on linear potential flow theory resolve only a part of the physics, i.e., it does not account for viscous damping effects. Moreover, significant uncertainties are associated with the system parameters related to the vessel operational conditions that are not known or measured directly. Therefore, the present paper integrates the vessel model with a derivative-free Lipschitz optimization technique to calibrate the vessel system parameters. For efficient optimization, the most influential variables are identified by conducting a sensitivity study for relevant quantities of interest using a cost-effective Polynomial Chaos model. The sensitivity analysis assumes independent uniform distributions for the system parameters with proper lower and upper limits. Subsequently, estimates of the highly influential variables are optimized using on-site vessel measurements, and in turn, utilized for superior response estimation. The response computations based on the numerical wave spectrum produced superior results in comparison with the results based on the parametric wave spectrum. The calibration algorithm is validated with full-scale measurements using frequency and time domain approaches for simulating vessel motions. It has been quantitatively revealed that the Center of Gravity parameters associated with the vessel operational characteristics and sea state-dependent additional damping coefficients govern the Roll response variation to a great extent. The calibration of these parameters resulted in improved Roll spectra estimations that match the measured spectra closely.

Hyperparameter Black-Box Optimization to Improve the Automatic Classification of Support Tickets

User requests to a customer service, also known as tickets, are essentially short texts in natural language. They should be grouped by topic to be answered efficiently. The effectiveness increases if this semantic categorization becomes automatic. We pursue this goal by using text mining to extract the features from the tickets, and classification to perform the categorization. This is however a difficult multi-class problem, and the classification algorithm needs a suitable hyperparameter configuration to produce a practically useful categorization. As recently highlighted by several researchers, the selection of these hyperparameters is often the crucial aspect. Therefore, we propose to view the hyperparameter choice as a higher-level optimization problem where the hyperparameters are the decision variables and the objective is the predictive performance of the classifier. However, an explicit analytical model of this problem cannot be defined. Therefore, we propose to solve it as a black-box model by means of derivative-free optimization techniques. We conduct experiments on a relevant application: the categorization of the requests received by the Contact Center of the Italian National Statistics Institute (Istat). Results show that the proposed approach is able to effectively categorize the requests, and that its performance is increased by the proposed hyperparameter optimization.

Design, Technical and Economic Optimization of Renewable Energy-Based Electric Vehicle Charging Stations in Africa: The Case of Nigeria

The transportation sector accounts for more than 70% of Nigeria’s energy consumption. This sector has been the major consumer of fossil fuels in the past 20 years. In this study, the technical and economic feasibility of an electrical vehicle (EV) charging scheme is investigated based on the availability of renewable energy (RE) sources in six sites representing diverse geographic and climatic conditions in Nigeria. The HOMER Pro® microgrid software with the grid-search and proprietary derivative-free optimization techniques is used to assess the viability of the proposed EV charging scheme. The PV/WT/battery charging station with a quantity of two WT, 174 kW of PV panels, a quantity of 380 batteries storage, and a converter of 109 kW located in Sokoto provide the best economic metrics with the lowest NPC, electricity cost, and initial costs of USD547,717, USD0.211/kWh, and USD449,134, respectively. The optimal charging scheme is able to reliably satisfy most of the EV charging demand as it presents a small percentage of the unmet load, which is the lowest when compared with the corresponding values of the other charging stations. Moreover, the optimal charging system in all six locations is able to sufficiently meet the EV charge requirement with maximum uptime. A sensitivity analysis was conducted to check the robustness of the optimum charging scheme. This sensitivity analysis reveals that the technical and economic performance indicators of the optimum charging station are sensitive to the changes in the sensitivity variables. Furthermore, the outcomes ensure that the hybrid system of RE sources and EVs can minimize carbon and other pollutant emissions. The results and findings in this study can be implemented by all relevant parties involved to accelerate the development of EVs not only in Nigeria but also in other parts of the African continent and the rest of the world.

Estimating the optimal linear combination of predictors using spherically constrained optimization

BackgroundIn the context of a binary classification problem, the optimal linear combination of continuous predictors can be estimated by maximizing the area under the receiver operating characteristic curve. For ordinal responses, the optimal predictor combination can similarly be obtained by maximization of the hypervolume under the manifold (HUM). Since the empirical HUM is discontinuous, non-differentiable, and possibly multi-modal, solving this maximization problem requires a global optimization technique. Estimation of the optimal coefficient vector using existing global optimization techniques is computationally expensive, becoming prohibitive as the number of predictors and the number of outcome categories increases.ResultsWe propose an efficient derivative-free black-box optimization technique based on pattern search to solve this problem, which we refer to as Spherically Constrained Optimization Routine (SCOR). Through extensive simulation studies, we demonstrate that the proposed method achieves better performance than existing methods including the step-down algorithm. Finally, we illustrate the proposed method to predict the severity of swallowing difficulty after radiation therapy for oropharyngeal cancer based on radiation dose to various structures in the head and neck.ConclusionsOur proposed method addresses an important challenge in combining multiple biomarkers to predict an ordinal outcome. This problem is particularly relevant to medical research, where it may be of interest to diagnose a disease with various stages of progression or a toxicity with multiple grades of severity. We provide the implementation of our proposed SCOR method as an R package, available online at https://CRAN.R-project.org/package=SCOR.

Quantifying safety risks of deep neural networks

Safety concerns on the deep neural networks (DNNs) have been raised when they are applied to critical sectors. In this paper, we define safety risks by requesting the alignment of network’s decision with human perception. To enable a general methodology for quantifying safety risks, we define a generic safety property and instantiate it to express various safety risks. For the quantification of risks, we take the maximum radius of safe norm balls, in which no safety risk exists. The computation of the maximum safe radius is reduced to the computation of their respective Lipschitz metrics—the quantities to be computed. In addition to the known adversarial example, reachability example, and invariant example, in this paper, we identify a new class of risk—uncertainty example—on which humans can tell easily, but the network is unsure. We develop an algorithm, inspired by derivative-free optimization techniques and accelerated by tensor-based parallelization on GPUs, to support an efficient computation of the metrics. We perform evaluations on several benchmark neural networks, including ACSC-Xu, MNIST, CIFAR-10, and ImageNet networks. The experiments show that our method can achieve competitive performance on safety quantification in terms of the tightness and the efficiency of computation. Importantly, as a generic approach, our method can work with a broad class of safety risks and without restrictions on the structure of neural networks.

Design and optimization of energy-efficient single mixed refrigerant LNG liquefaction process

The optimal design of single mixed refrigerant (SMR) natural gas liquefaction process is decisive for upgrading the liquefied natural gas (LNG) value chain and can be achieved using process simulators and derivative-free optimization techniques in simulation–optimization frameworks. The process work consumption can be reduced by modifying process flowsheet and manipulating refrigeration cycle thermodynamic conditions and refrigerant composition. The goal of this paper is the design of energy-efficient SMR liquefaction process for LNG production using simulation–optimization approach, in Aspen HYSYS® and MATLAB®, that considers more decision variables to be optimized with a Nelder–Mead simplex algorithm and small flowsheet modifications, such as hydraulic turbines and flash separators in between compression stages. The present approach resulted in liquefaction process with network duty of 750.2 kJ per kg of natural gas, which is considerably smaller than those reported in the literature.

An Inverse-Adjusted Best Response Algorithm for Nash Equilibria

Regarding the approximation of Nash equilibria in games where the players have a continuum of strategies, there exist various algorithms based on best response dynamics and on its relaxed variants: from one step to the next, a player's strategy is updated by using explicitly a best response to the strategies of the other players that come from the previous steps. These iterative schemes generate sequences of strategy profiles which are constructed by using continuous optimization techniques and they have been shown to converge in the following situations: in zero-sum games or, in non-zero-sum ones, under contraction assumptions or under linearity of best response functions. In this paper, we propose an algorithm which guarantees the convergence to a Nash equilibrium in two-player non-zero-sum games when the best response functions, called $r_1$ and $r_2$, are not necessarily linear, neither the composition $r_1\circ r_2$ nor $r_2\circ r_1$ is a contraction, and the strategy sets are Hilbert spaces. First, we address the issue of uniqueness of the Nash equilibrium extending to a more general class the result obtained by Caruso, Ceparano, and Morgan [J. Math. Anal. Appl., 459 (2018), pp. 1208--1221] for weighted potential games. Then, we describe a theoretical approximation scheme based on a nonstandard (nonconvex) relaxation of best response iterations which converges to the unique Nash equilibrium of the game. Finally, we define a numerical approximation scheme relying on a derivative-free continuous optimization technique applied in a finite dimensional setting and we provide convergence results and error bounds.

Derivative-Free Direct Search Optimization Method for Enhancing Performance of Analytical Design Approach-Based Digital Controller for Switching Regulator

Although an analytical design approach-based digital controller—which is essentially a deadbeat controller—shows zero steady-state error and no intersampling oscillations, it takes a finite number of sampling periods to settle down to a steady-state value. This paper describes the application of a derivative-free Nelder–Mead (N–M) simplex method to the digital controller for retuning of its coefficients intelligently to ensure improved settling and rise times without disturbing the deadbeat controller characteristics (i.e., no ripples between the sampling periods and no steady-state error). A switching-mode buck regulator working at 1 MHz in continuous conduction mode (CCM) is considered as a plant. Numerical simulation results depict that the N–M algorithm-based optimized digital controller not only shows improved steady-state and transient performance but also guarantees rigorous robustness against model uncertainty and disturbance as compared to its traditional counterpart, as well as the other optimized digital controller fine-tuned through other derivative-free metaheuristic optimization techniques, such as the genetic algorithm (GA). A system generator-based hardware software co-simulation is also performed to validate the simulation results.

Practical optimization for cost reduction of a liquefier in an industrial air separation plant

Abstract Commercial and in-house simulation software used by industrial practitioners are often of a “black box” type from which derivatives cannot be directly obtained. This paper demonstrates a linkage between available industrial tools and cost reduction opportunity creation through the application of a derivative-free optimization technique. An operational liquefier in an air separation unit is used in our study due to the increasing importance of liquid production in the plant's overall operation strategy, and limited evaluation on the operation of such systems under disturbances. Particle swarm optimization is implemented, and optimization results show that when the plant is forced to operate away from its nominal operating/design conditions, it is possible to reduce the unit power consumption by adjusting different operation set-points. A reference map is generated to guide the operation under selected realizations of cooling water temperature, production load and feed conditions.

A reliable modifier-adaptation strategy for real-time optimization

In model-based real-time optimization, plant-model mismatch can be handled by applying bias- and gradient-corrections to the cost and constraint functions in an iterative optimization procedure. One of the major challenges in practice is the estimation of the plant gradients from noisy measurement data, in particular for several optimization variables. In this paper we propose a new real-time optimization scheme that explores the inherent smoothness of the plant mapping to enable a reliable optimization. The idea here is to combine the quadratic approximation approach used in derivative-free optimization techniques with the iterative gradient-modification optimization scheme. The convergence of the scheme is analyzed. Simulation studies for the optimization of a ten-variable synthetic example and a reactor benchmark problem with considerable plant-model mismatch show its promising performance.

The theory of variational hybrid quantum-classical algorithms

Many quantum algorithms have daunting resource requirements when compared to what is available today. To address this discrepancy, a quantum-classical hybrid optimization scheme known as ‘the quantum variational eigensolver’ was developed (Peruzzo et al 2014 Nat. Commun. 5 4213) with the philosophy that even minimal quantum resources could be made useful when used in conjunction with classical routines. In this work we extend the general theory of this algorithm and suggest algorithmic improvements for practical implementations. Specifically, we develop a variational adiabatic ansatz and explore unitary coupled cluster where we establish a connection from second order unitary coupled cluster to universal gate sets through a relaxation of exponential operator splitting. We introduce the concept of quantum variational error suppression that allows some errors to be suppressed naturally in this algorithm on a pre-threshold quantum device. Additionally, we analyze truncation and correlated sampling in Hamiltonian averaging as ways to reduce the cost of this procedure. Finally, we show how the use of modern derivative free optimization techniques can offer dramatic computational savings of up to three orders of magnitude over previously used optimization techniques.

Optimizing damper connectors for adjacent buildings

Many theoretical and experimental studies have used heuristic methods to investigate the dynamic behaviour of the passive coupling of adjacent structures. However, few papers have used optimization techniques with guaranteed convergence in order to increase the efficiency of the passive coupling of adjacent structures. In this paper, the combined problem of optimal arrangement and mechanical properties of dampers placed between two adjacent buildings is considered. A new bi-level optimization approach is presented. The outer-loop of the approach optimizes damper configuration and is solved using the ``inserting dampers'' method, which was recently shown to be a very effective heuristic method. Under the assumption that the dampers have varying damper coefficients, the inner-loop finds the optimal damper coefficients by solving an $n$-dimensional optimization problem, where derivative information of the objective function is not available. Three different non-gradient methods are compared for solving the inner loop: a genetic algorithm (GA), the mesh adaptive direct search (MADS) algorithm, and the robust approximate gradient sampling (RAGS) algorithm. It is shown that by exploiting this new bi-level problem formulation, modern derivative free optimization techniques with guaranteed convergence (such as MADS and RAGS) can be used. The results indicate a great increase in the efficiency of the retrofitting system, as well as the existence of a threshold on the number of dampers inserted with respect to the efficiency of the retrofitting system.

AC-DC Optimal Power Flow Incorporating Shunt FACTS Devices Using HVDC Model and Particle Swarm Optimization Method

This paper presents an efficient and reliable evolutionary-based approach to solve the AC-DC optimal power flow (OPF) incorporating shunt FACTS devices in electrical power networks. Our main goal is to minimize the objective function necessary for a best balance between the energy production and its consumption which is presented in a nonlinear function, taking into account the constraints of equality and of inequality. The objective is to minimize the total cost of the generations and also the active power losses with incorporating shunt FACTS devices such as the static VAr compensator (SVC). Furthermore, we will consider in detail the study of the AC-DC OPF under the use of the HVDC-link and the SVC. This optimization is based, on the application of the particle swarm optimization (PSO) method. Incorporation of the PSO method as a derivative-free optimization technique in solving AC-DC OPF problem relieves, significantly, the assumptions imposed on the optimized objective functions. The PSO method has been examined and tested on the standard IEEE 14-bus test system with different objectives that reflect active power losses minimization and active power generation cost minimization. The PSO results have been compared to those that reported in the literature recently. The results are promising and show the effectiveness and robustness of PSO method.

Behavioral analysis of the leader particle during stagnation in a particle swarm optimization algorithm

Concept of the particle swarms emerged from a simulation of the collective behavior of social creatures. It gradually evolved into a powerful derivative-free optimization techniques, now known as Particle Swarm Optimization (PSO) for solving multi-dimensional, multi-modal, and non-convex optimization problems. The dynamics governing the movement of the particles in PSO has invoked a great deal of research interest over the last decade. Theoretical investigations of PSO has mostly focused on particle trajectories in the search space and the parameter-selection. This work looks into the PSO algorithm from the perspective of the leader particle and takes into account stagnation, a situation where particles are trapped at less coveted local optima, thus preventing them from reaching more coveted global optima. We show that the points sampled by the leader particle satisfy a simple mathematical relation which demonstrates that they lie on a specific line. We demonstrate the condition under which for certain values of the parameters, particles stick to exploring one side of the stagnation point only and ignore the other side, and also the case where both sides are explored. We also obtain information about the gradient of the objective function during stagnation in PSO. We provide a large number of machine simulations which support our claims over several ranges of the control parameters. This sheds light on possible modifications to the basic PSO algorithm which would help future researchers to work with even more efficient and state-of-the-art PSO variants.

An Optimal PID Controller for a Bidirectional Inductive Power Transfer System Using Multiobjective Genetic Algorithm

Bidirectional inductive power transfer (IPT) systems are suitable for applications that require wireless and two-way power transfer. However, these systems are high-order resonant networks in nature and, hence, design and implementation of an optimum proportional–integral–derivative (PID) controller using various conventional methods is an onerous exercise. Further, the design of a PID controller, meeting various and demanding specifications, is a multiobjective problem and direct optimization of the PID gains often lead to a nonconvex problem. To overcome the difficulties associated with the traditional PID tuning methods, this paper, therefore, proposes a derivative-free optimization technique, based on genetic algorithm (GA), to determine the optimal parameters of PID controllers used in bidirectional IPT systems. The GA determines the optimal gains at a reasonable computational cost and often does not get trapped in a local optimum. The performance of the GA-tuned controller is investigated using several objective functions and under various operating conditions in comparison to other traditional tuning methods. It was observed that the performance of the GA-based PID controller is dependent on the nature of the objective function and therefore an objective function, which is a weighted combination of rise time, settling time, and peak overshoot, is used in determining the parameters of the PID controller using multiobjective GA. Simulated and experimental results of a 1-kW prototype bidirectional IPT system are presented to demonstrate the effectiveness of the GA-tuned controller as well as to show that gain selection through multiobjective GA using the weighted objective function yields the best performance of the PID controller.

A survey of non-gradient optimization methods in structural engineering

In this paper, we present a review on non-gradient optimization methods with applications to structural engineering. Due to their versatility, there is a large use of heuristic methods of optimization in structural engineering. However, heuristic methods do not guarantee convergence to (locally) optimal solutions. As such, recently, there has been an increasing use of derivative-free optimization techniques that guarantee optimality. For each method, we provide a pseudo code and list of references with structural engineering applications. Strengths and limitations of each technique are discussed. We conclude with some remarks on the value of using methods customized for a desired application.

The impact of uncertainty on shape optimization of idealized bypass graft models in unsteady flow

It is well known that the fluid mechanics of bypass grafts impacts biomechanical responses and is linked to intimal thickening and plaque deposition on the vessel wall. In spite of this, quantitative information about the fluid mechanics is not currently incorporated into surgical planning and bypass graft design. In this work, we use a derivative-free optimization technique for performing systematic design of bypass grafts. The optimization method is coupled to a three-dimensional pulsatile Navier–Stokes solver. We systematically account for inevitable uncertainties that arise in cardiovascular simulations, owing to noise in medical image data, variable physiologic conditions, and surgical implementation. Uncertainties in the simulation input parameters as well as shape design variables are accounted for using the adaptive stochastic collocation technique. The derivative-free optimization framework is coupled with a stochastic response surface technique to make the problem computationally tractable. Two idealized numerical examples, an end-to-side anastomosis, and a bypass graft around a stenosis, demonstrate that accounting for uncertainty significantly changes the optimal graft design. Results show that small changes in the design variables from their optimal values should be accounted for in surgical planning. Changes in the downstream (distal) graft angle resulted in greater sensitivity of the wall-shear stress compared to changes in the upstream (proximal) angle. The impact of cost function choice on the optimal solution was explored. Additionally, this work represents the first use of the stochastic surrogate management framework method for robust shape optimization in a fully three-dimensional unsteady Navier–Stokes design problem.

Particle Swarm Optimization Optimized Controller for Static Synchronous Series Compensator to Improve Torsional Damping

This article uses particle swarm optimization, one of the derivative-free optimization techniques, to optimize the proportional-integral controller parameters of the DC-link voltage controller of a 48-pulse static synchronous series compensator. This particle-swarm-optimization-based proportional-integral controller is used to keep the voltage across the DC-link capacitor of a 48-pulse voltage-source-inverter-based static synchronous series compensator constant. The particle swarm optimization approach avoids the repeated trials that are needed in a trial-and-error approach to obtain near-optimum proportional-integral controller parameters. An objective function is proposed, which helps to improve the damping of modes near to the imaginary axis. The study system considered is the IEEE first benchmark model for sub-synchronous resonance. The stability of the system is determined from eigenvalue analysis. The article also shows that an optimized proportional-integral controller can improve the performance of the system with lower value of DC-link capacitor. Non-linear simulations are performed to verify the results of eigenvalue analysis.

High-level approach to modeling of observed system behavior

Current computer systems and communication networks tend to be highly complex, and they typically hide their internal structure from their users. Thus, for selected aspects of capacity planning, overload control and related applications, it is useful to have a method allowing one to find good and relatively simple approximations for the observed system behavior with little knowledge of the system internal structure or operation. This paper investigates one such approach where we attempt to represent the observed behavior by adequately selecting the parameters of a set of queueing models. We identify a limited number of queueing models that we use as Building Blocks in our procedure. The selected Building Blocks allow us to accurately approximate the measured behavior of a range of different systems. The models used include a small selection of elementary queueing models, as well as original models of system congestion and saturation. We propose an approach for selecting suitable Building Blocks, as well as for their calibration. In particular, we automate the calibration step and make it systematic through the use of a Derivative-Free Optimization technique. We are able to successfully validate our methodology for a number of case studies. Finally, we discuss the potential and the limitations of the proposed approach.