Surrogate Objectives Research Articles

Trust region policy optimization via entropy regularization for Kullback–Leibler divergence constraint

Trust region policy optimization (TRPO) is one of the landmark policy optimization algorithms in deep reinforcement learning. Its purpose is to maximize a surrogate objective based on an advantage function, subject to the limited Kullback–Leibler (KL) divergence of two consecutive policies. Although there have been many successful applications of this algorithm in the literature, the approach has often been criticized for suppressing the exploration ability of some application environments due to its strict divergence constraint. As such, most researchers prefer to use entropy regularization, which is added to the expected discounted rewards or the surrogate objectives. That said, there is much debate about whether there might be an alternative strategy for regularizing TRPOs. In this paper, we present just that. Our strategy is to regularize the KL divergence-based constraint via Shannon entropy. This approach enlarges the difference between two consecutive policies and thus derives a new TRPO scheme with entropy regularization for use with KL divergence constraint. Next, the surrogate objective and Shannon entropy are approximated linearly, while the KL divergence is expanded quadratically. An efficient conjugate gradient optimization procedure then solves two sets of linear equations, providing a detailed code-level implementation that can be used for a fair experimental comparison. Extensive experiments within eight benchmark environments demonstrate that our proposed method is superior to both the original TRPO and the entropy regularized objective TRPO. Further, theoretical and experimental analysis shows that three TRPO-like methods have an equal time complexity and a close computational burden.

Empirical bias-reducing adjustments to estimating functions

Abstract We develop a novel, general framework for reduced-bias M-estimation from asymptotically unbiased estimating functions. The framework relies on an empirical approximation of the bias by a function of derivatives of estimating function contributions. Reduced-bias M-estimation operates either implicitly, solving empirically adjusted estimating equations, or explicitly, subtracting the estimated bias from the original M-estimates, and applies to partially or fully specified models with likelihoods or surrogate objectives. Automatic differentiation can abstract away the algebra required to implement reduced-bias M-estimation. As a result, the bias-reduction methods, we introduce have broader applicability, straightforward implementation, and less algebraic or computational effort than other established bias-reduction methods that require resampling or expectations of products of log-likelihood derivatives. If M-estimation is by maximising an objective, then there always exists a bias-reducing penalised objective. That penalised objective relates to information criteria for model selection and can be enhanced with plug-in penalties to deliver reduced-bias M-estimates with extra properties, like finiteness for categorical data models. Inferential procedures and model selection procedures for M-estimators apply unaltered with the reduced-bias M-estimates. We demonstrate and assess the properties of reduced-bias M-estimation in well-used, prominent modelling settings of varying complexity.

Modeling of an unseeded reactive crystallization process using multiobjective optimization

Modeling of reactive crystallization processes is considered to be a complex task where the reaction and crystallization processes occuring together presents additional challenges. A simplified approach is shown in this work to model such reactive crystallization processes using limited number of experiments covering the design space. In the current application, a case study is demonstrated using unseeded semi-batch reactive crystallization of salicylic acid. Multiobjective optimization (MOO) approach is used to determine the kinetic model parameters considering surrogate objectives related to solution concentration and mean size of crystals using the experimental data collected. The proposed first principles model is in good agreement with the experimental data and thus can be further applied for model predictions to determine optimal control strategies.

Structured Output Learning with Conditional Generative Flows

Traditional structured prediction models try to learn the conditional likelihood, i.e., p(y|x), to capture the relationship between the structured output y and the input features x. For many models, computing the likelihood is intractable. These models are therefore hard to train, requiring the use of surrogate objectives or variational inference to approximate likelihood. In this paper, we propose conditional Glow (c-Glow), a conditional generative flow for structured output learning. C-Glow benefits from the ability of flow-based models to compute p(y|x exactly and efficiently. Learning with c-Glow does not require a surrogate objective or performing inference during training. Once trained, we can directly and efficiently generate conditional samples. We develop a sample-based prediction method, which can use this advantage to do efficient and effective inference. In our experiments, we test c-Glow on five different tasks. C-Glow outperforms the state-of-the-art baselines in some tasks and predicts comparable outputs in the other tasks. The results show that c-Glow is versatile and is applicable to many different structured prediction problems.

Optimization Approaches for the Numerical Design of Structures Under Consideration of Polymorphic Uncertain Data

Abstract In this paper, optimization approaches for the numerical design of structures are presented. The uncertainty of structural parameters is considered by means of random variables and interval design parameters within the optimization. A new space subdividing technique is introduced to substitute time-consuming failure probability constraint evaluations. In order to solve optimization problems with polymorphic uncertain parameters, surrogate objectives are formulated and solved by means of a particle swarm optimization (PSO) approach in combination with Monte Carlo simulation and interval analysis. Two computational schemes are presented and verified by a benchmark example. Finally, an application is shown, where the reinforcement layout of a reinforced concrete bridge structure is optimized by minimizing the crack widths at the reinforcement layers in order to improve the durability of the structure. A nonlinear finite element (FE) model is used to compute the uncertain crack patterns and the load bearing capacity. The stochastic objective function and the failure probability constraint are approximated by neural network based surrogate models.

Surrogate measures to optimize structures for robust and predictable progressive failure

Optimizing complex structures for robust and predictable progressive failure using probabilistic approaches is computationally expensive. In this paper we investigate the progressive failure characteristics of structures subjected to random variability and deduce patterns to identify surrogate measures that correlate with robustness and predictability of the design’s progressive failure. The procedure is demonstrated for the optimization of robustness and predictability in progressive failure of truss structures. Deterministic optimization of trusses was used to generate candidate designs to compare and contrast robustness and predictability. The stochastic analyses of the candidate designs are then used to identify surrogate features that correlate to robustness and predictability of progressive failure response. These features are converted to numerical surrogate objectives or constraints and used in optimization to demonstrate their effectiveness and computational efficiency. The example shows that surrogate measures can be developed for robustness and predictability optimization, and that such measures are computationally efficient compared to robustness optimization using sampling based methods.

A comparative study of single-phase and two-phase approaches for the layout problem with material handling costs

A comparative study of one- and two-phase procedures for material handling cost-based layout problems is presented. A cost function that explicitly accounts for heterogeneous material handling equipment is used as the objective criterion and simulated annealing based search procedures are used to determine layout solutions. Simulated annealing is applied directly in the one-phase procedure and augmented with volume distance based clustering of work centres in the two-phase procedure to reduce computational complexity. The performance of the two approaches are tested on 180 sample problems with 40 work centres that differ in terms of material flow matrix density, work centre space requirements, and variation in the distribution of material flow matrix elements. The results indicate that complexity reduction techniques using surrogate objectives based on volume distance might not always result in layouts that reduce material handling cost.

Finding Optimal Algorithmic Parameters Using Derivative‐Free Optimization

The objectives of this paper are twofold. We devise a general framework for identifying locally optimal algorithmic parameters. Algorithmic parameters are treated as decision variables in a problem for which no derivative knowledge or existence is assumed. A derivative-free method for optimization seeks to minimize some measure of performance of the algorithm being fine-tuned. This measure is treated as a black-box and may be chosen by the user. Examples are given in the text. The second objective is to illustrate this framework by specializing it to the identification of locally optimal trust-region parameters in unconstrained optimization. The derivative-free method chosen to guide the process is the mesh adaptive direct search, a generalization of pattern search methods. We illustrate the flexibility of the latter and in particular make provision for surrogate objectives. Locally, optimal parameters with respect to overall computational time on a set of test problems are identified. Each function call may take several hours and may not always return a predictable result. A tailored surrogate function is used to guide the search towards a local solution. The parameters thus identified differ from traditionally used values, and allow one to solve a problem that remained otherwise unsolved in a reasonable time using traditional values.

A study on surrogate objectives for loading a certain type of flexible manufacturing systems

Abstract Various objectives are used for the loading problem in flexible manufacturing systems. Because of the complexity, objectives commonly used in real fields may not be employed when solving the problem. This paper examines relationships between these objectives and ones that are relatively easy to deal with and are used in most research on this problem, i.e. balancing workloads of machines. Also defined are pertinent measures of balance for an objective function of the problem. Computational tests show that balancing workloads helps to reduce tardiness and makespan while enhance system throughput in a system with the characteristics of open job shops.

Scheduling Players in Team Competitions: Theory and Computational Results

SYNOPTIC ABSTRACTConsider two teams, each with n players, that must match up players to compete in n events. In each event, a given player either wins or loses (with known probabilities) thereby earning his team 1 or 0 points, respectively. The team with the larger point total wins the match. We formulate the problem of maximizing the probability of winning for a team that has perfect information on the opposing team's assignment of players to events. We propose surrogate objective functions that one may use in lieu of the probability of winning to make the problem computationally tractable and we provide algorithms based on fractional programming to maximize these surrogate objectives.

Sequential games of status

In this paper we use classical and behavioral game theory to predict coalition behavior in a sequential three-person laboratory game. In the laboratory situation used for the three studies reported in this paper, each player seeks to maximize the rank of his final accumulated point score in relation to the total scores of the other members of his triad. We assume that, unable to solve the sequential game as a single supergame, each player forms a simplified representation of the situation by adopting short-run, surrogate objectives for the outcome of each game in the sequence. In particular, we assume that each player seeks to maximize the status of his current total score during each game in the sequence. Given these surrogate objectives, we represent each trial in the sequential game as a distinct game of status in characteristic set function form. Using this formal model of the situation, we extend a solution concept from n-person game theory to identify which of the possible outcomes of each game in the sequence are, in a game theoretic sense, stable. We then add the assumption that each player obeys a bargaining maxim prescribing that coalition members should agree to an outcome in which one partner outranks the other if and only if he also enjoys higher rank when the coalition forms. We modify the game theoretic solution by considering only outcomes that conform to the bargaining maxim. The behavioral modification substantially improves the success game theory achieves in predicting rank outcomes observed in the three laboratory studies. However, with or without the modification, the game theoretic solution concept fails to predict which coalitions form in these studies. Finally, we develop an alternative theory that predicts not only the rank outcomes of each game in the sequence but also the probability that each coalition forms. The empirical success enjoyed by this theory in predicting the ranks acquired on each game is greater than that attained by the unmodified game theoretic predictions, and is nearly comparable to that achieved by the behaviorally modified game theoretic solution. Moreover, the alternative theory's robust coalition predictions are strongly supported by the results from the three laboratory studies.