Gradient Optimization Procedure Research Articles

Trust region policy optimization via entropy regularization for Kullback–Leibler divergence constraint

Trust region policy optimization (TRPO) is one of the landmark policy optimization algorithms in deep reinforcement learning. Its purpose is to maximize a surrogate objective based on an advantage function, subject to the limited Kullback–Leibler (KL) divergence of two consecutive policies. Although there have been many successful applications of this algorithm in the literature, the approach has often been criticized for suppressing the exploration ability of some application environments due to its strict divergence constraint. As such, most researchers prefer to use entropy regularization, which is added to the expected discounted rewards or the surrogate objectives. That said, there is much debate about whether there might be an alternative strategy for regularizing TRPOs. In this paper, we present just that. Our strategy is to regularize the KL divergence-based constraint via Shannon entropy. This approach enlarges the difference between two consecutive policies and thus derives a new TRPO scheme with entropy regularization for use with KL divergence constraint. Next, the surrogate objective and Shannon entropy are approximated linearly, while the KL divergence is expanded quadratically. An efficient conjugate gradient optimization procedure then solves two sets of linear equations, providing a detailed code-level implementation that can be used for a fair experimental comparison. Extensive experiments within eight benchmark environments demonstrate that our proposed method is superior to both the original TRPO and the entropy regularized objective TRPO. Further, theoretical and experimental analysis shows that three TRPO-like methods have an equal time complexity and a close computational burden.

Numerical and experimental investigations on shape optimization of submerged floating tunnels with a discrete adjoint method

The shape of the tube in submerged floating tunnels (SFTs) plays a critical role in determining their performance and safety in marine environments. In this study, a gradient optimization procedure based on the discrete adjoint method is performed to minimize the drag force under uniform currents action. The free form deformation approach is employed to parameterize the design variables. The physical mechanisms of the optimization process are revealed via unsteady numerical simulations and experimental tests. Moreover, the hydrodynamic performance of the tube with the optimal shapes subjected to a wave–current combination is also evaluated. The results show that the drag coefficient is significantly reduced through optimization by reducing the pressure amplitude in the negative-pressure area. Additionally, the lift oscillation is also suppressed to delay structural fatigue, owing to the strength reduction and increased distance of the wake vortex. The experimental analysis indicates the advantage of the optimal shape in reducing the loads under wave–current actions, accompanied by changes in the frequency distribution of the force and vortex structure. The elliptical shape of the SFT's tube provides significant advantages in drag reduction at high Reynold number. Suggestions on the shape design of the section of SFT are given according to different types of constraints.

A Fault Diagnosis Method for the Autonomous Underwater Vehicle via Meta-Self-Attention Multi-Scale CNN

Autonomous underwater vehicles (AUVs) are an important equipment for ocean investigation. Actuator fault diagnosis is essential to ensure the sailing safety of AUVs. However, the lack of failure data for training due to unknown ocean environments and unpredictable failure occurrences is challenging for fault diagnosis. In this paper, a meta-self-attention multi-scale convolution neural network (MSAMS–CNN) is proposed for the actuator fault diagnosis of AUVs. Specifically, a two-dimensional spectrogram of the vibration signals obtained by a vibration sensor is used as the neural network’s inputs. The diagnostic model is fitted by executing a subtask-based gradient optimization procedure to generate more general degradation knowledge. A self-attentive multi-scale feature extraction approach is used to utilize both global and local features for learning important parameters autonomously. In addition, a meta-learning method is utilized to train the diagnostic model without a large amount of labeled data, which enhances the generalization ability and allows for cross-task training. Experimental studies with real AUV data collected by vibration sensors are conducted to validate the effectiveness of the MSAMS–CNN. The results show that the proposed method can diagnose the rudder and thruster faults of AUVs in the cases of few-shot diagnosis.

Joint Communication and Action Learning in Multi-Target Tracking of UAV Swarms with Deep Reinforcement Learning

Communication is the cornerstone of UAV swarms to transmit information and achieve cooperation. However, artificially designed communication protocols usually rely on prior expert knowledge and lack flexibility and adaptability, which may limit the communication ability between UAVs and is not conducive to swarm cooperation. This paper adopts a new data-driven approach to study how reinforcement learning can be utilized to jointly learn the cooperative communication and action policies for UAV swarms. Firstly, the communication policy of a UAV is defined, so that the UAV can autonomously decide the content of the message sent out according to its real-time status. Secondly, neural networks are designed to approximate the communication and action policies of the UAV, and their policy gradient optimization procedures are deduced, respectively. Then, a reinforcement learning algorithm is proposed to jointly learn the communication and action policies of UAV swarms. Numerical simulation results verify that the policies learned by the proposed algorithm are superior to the existing benchmark algorithms in terms of multi-target tracking performance, scalability in different scenarios, and robustness under communication failures.

Efficient assimilation of sparse data into RANS-based turbulent flow simulations using a discrete adjoint method

Turbulent flow simulations based on the Reynolds-averaged Navier–Stokes (RANS) equations continue to be the workhorse approach for industrial flow problems. However, due to the inherent averaging and the closure models required for the Reynolds stresses, their accuracy is limited. Experimental data, on the other hand, represents the true flow features but is potentially only available in limited locations or at low resolution. Data assimilation (DA) can be used to combine closure models with such experimental data to obtain accurate simulation results. In this scenario, the result of the DA process can be interpreted as a physics-based interpolation and serve as a basis for further studies of the flow. Moreover, such tuned models may be used in machine learning approaches aiming at a priori closure model enhancement. The main objective of this work is to recover a spatially varying eddy viscosity correction factor from sparsely distributed reference data.We use an efficient in-house data assimilation implementation based on the discrete adjoint method for RANS simulations in OpenFOAM to recover an optimal eddy viscosity field. A gradient optimization procedure is used to minimize a velocity-based cost function with a regularization term. All the partial derivatives in the gradient computation are evaluated with an efficient semi-analytical approach at the cost of approximately one forward solution step.The effectiveness of the proposed approach is demonstrated for a periodic hill test case at Re=10595 with different spatial distributions of the reference data. The results show improvements in the agreement between simulation and reference for a range of reference data configurations and highlight the performance of linear eddy viscosity models.

Inverse design and demonstration of high-performance wide-angle diffractive optical elements.

Diffractive optical elements are ultra-thin optical components required for constructing very compact optical 3D sensors. However, the required wide-angle diffractive 2D fan-out gratings have been elusive due to design challenges. Here, we introduce a new strategy for optimizing such high-performance and wide-angle diffractive optical elements, offering unprecedented control over the power distribution among the desired diffraction orders with only low requirements with respect to computational power. The microstructure surfaces were designed by an iterative gradient optimization procedure based on an adjoint-state method, capable to account for application-dependent target functions while ensuring compatibility with existing fabrication processes. The results of the experimental characterization confirm the simulated tailored power distributions and optical efficiencies of the fabricated elements.

Experimental Design via Generalized Mean Objective Cost of Uncertainty

The mean objective cost of uncertainty (MOCU) quantifies the performance cost of using an operator that is optimal across an uncertainty class of systems as opposed to using an operator that is optimal for a particular system. MOCU-based experimental design selects an experiment to maximally reduce MOCU, thereby gaining the greatest reduction of uncertainty impacting the operational objective. The original formulation applies to finding optimal system operators, where optimality is with respect to a cost function, such as mean-square error; and the prior distribution governing the uncertainty class relates directly to the underlying physical system. Here, we provide a generalized MOCU and the corresponding experimental design. We, then, demonstrate how this new formulation includes as special cases MOCU-based experimental design methods developed for materials science and genomic networks, when there is experimental error. Most importantly, we show that the classical knowledge gradient and efficient global optimization procedures are specific implementations of MOCU-based experimental design under their modeling assumptions.

Direct Minimization for Ensemble Electronic Structure Calculations

We consider a direct optimization approach for ensemble density functional theory electronic structure calculations. The update operator for the electronic orbitals takes the structure of the Stiefel manifold into account and we present an optimization scheme for the occupation numbers that ensures that the constraints remain satisfied. We also compare sequential and simultaneous quasi-Newton and nonlinear conjugate gradient optimization procedures, and demonstrate that simultaneous optimization of the electronic orbitals and occupation numbers improve performance compared to the sequential approach.

Robust PSS Parameters Design Using a Trajectory Sensitivity Approach

A new model for coordinated optimal design of power system stabilizers (PSSs) is proposed for damping low frequency power oscillation of interconnected power systems and enhancing overall performance of both transient and small-signal stability. In this model, a novel objective function is constructed using rotor speeds of generators in a post disturbed period. Trajectory sensitivity mapping technique is proposed to evaluate the approximate gradients of the objective function relative to the PSS parameters. With the gradients, conjugate gradient method is employed to assess the PSS parameters for minimizing the objective function. Two schemes of gradient formulation and optimization procedure are suggested and compared. The model and effectiveness of the method are verified by eigenvalue assessments and time domain nonlinear simulations on the IEEE four-generator and the ten-generator New England test systems.

Optimization of cyclically operated reactors and separators

An efficient version of a numerical gradient optimization procedure for the computation of solutions to periodic optimal control problems is presented. This procedure consists of a first order gradient method in combination with the Newton–Picard shooting method. The first order gradient method is able to compute the optimal control of a periodic process under one or more nonlocal constraints, such as a minimum purity constraint. The Newton–Picard method computes very efficiently periodic solutions to the state and adjoint equations. The presented numerical procedure is used to optimize a rapid pressure swing reactor and a rapid pressure swing adsorber. It is shown that the optimal cycle scheme consists of four steps: a no-inflow pre-pressurization step, a pressurization step, a no-inflow post-pressurization step, and a depressurization step.

Properties of small- to medium-sized mercury clusters from a combined ab initio, density-functional, and simulated-annealing study.

Relativistic coupled-cluster and second-order many-body perturbation theories were used to construct two- and three-body potentials for the interaction between mercury atoms. A subsequent combined simulated-annealing downhill simplex and conjugate gradient-optimization procedure gave global minima for mercury clusters with up to 30 atoms. The calculations reveal magic cluster numbers of 6, 13, 19, 23, 26, and 29 atoms. At these cluster sizes, the static dipole polarizability obtained from density-functional theory has a minimum. The calculations also reveal a fast convergence of the polarizability towards the bulk limit in contrast to the singlet-triplet gap or the ionization potential.

Enzyme catalysis: Transition structures and quantum dynamical aspects: Modeling rubisco's oxygenation and carboxylation mechanisms

Quantum dynamics is introduced with the help of molecular states (MS) for systems decomposable into n electrons and N nuclei. These states, when projected onto the electronic and nuclear configuration spaces, r and R, respectively, are represented as products of electronic and nuclear wave functions of the type Φk(r;αk°)χkj(R;αk°). The 3N coordinates αk° are the positions of positive charges in real space that are equivalent to the nuclear charges. The stationary geometry is derived from quantum chemical analytical gradient optimization procedures. The set {Φk(r;αk°)χkj(R;αk°)} is assumed to contain all cluster partitioning including corresponding asymptotic states. These MSs provide a basis to represent a quantum state as a particular linear superposition. Quantum states are (row) vectors in a dual space whose components are the complex coefficients in the linear superposition of MSs. A reactive system is represented in the direct product space of the MS and surrounding medium: ∣MS〉⊗∣E〉. The surrounding basis states {∣E〉} may be other molecular state systems (protein, solvent) or electromagnetic fields. A chemical reaction is represented as a time evolution of vectors in the dual space driven by the interaction with an energy source or sink system. A theory of catalysis is constructed on this new basis for which the protein–substrate interaction operator drives the quantum change of state. The chemical reaction catalyzed by rubisco is examined and used to introduce the theoretical scheme. The mechanism of carboxylation and oxygenation expressed as sets of transition structures is discussed from this new quantum mechanical perspective. For a protein in thermal equilibrium with a thermal bath at absolute temperature T, in so far as time evolution in the reactant system is concerned, the protein may be replaced by a blackbody radiation field. In real situations, both the protein–substrate and electromagnetic field–substrate interactions provide mechanisms to favor particular time evolutions of the quantum system. A mapping to a unit hypersphere with the axis represented by (orthogonalized) MSs permits a simple “visualization” of all bound quantum states related to the system as points on the surface of the hypersphere. Time evolution of complex systems can be systematized in this new way. © 2002 John Wiley & Sons, Inc. Int J Quantum Chem, 2002

Overview of blind signal separation, part I: Criteria and algorithms

In blind signal separation, multiple independent source signals are separated from multiple linear mixtures of these signals without specific knowledge of either the source signal characteristics or the mixing conditions. This talk provides an introduction to the blind signal separation task. Three different problem formulations—signal separation of instantaneous mixtures, signal separation of convolutive mixtures, and multichannel blind deconvolution—are described, and their similarities and differences are highlighted. An overview of both information-theoretic and contrast-based separation criteria is then given. Natural gradient optimization procedures, when combined with such criteria, yield simple and useful blind signal separation algorithms. An example of speech separation of real room recordings illustrates the capabilities and limitations of one such approach.

High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system

Some imaging systems employ detector arrays which are not su‐ciently dense so as to meet the Nyquist criteria during image acquisition. This is particularly true for many staring infrared imagers. Thus, the full resolution afiorded by the optics is not being realized in such a system. This paper presents a technique for estimating a high resolution image, with reduced aliasing, from a sequence of undersampled rotated and translationally shifted frames. Such an image sequence can be obtained if an imager is mounted on a moving platform, such as an aircraft. Several approaches to this type of problem have been proposed in the literature. Here we extend some of this previous work. In particular, we deflne an observation model which incorporates knowledge of the optical system and detector array. The high resolution image estimate is formed by minimizing a new regularized cost function which is based on the observation model. We show that with the proper choice of a tuning parameter, our algorithm exhibits robustness in the presence of noise. We consider both gradient descent and conjugate gradient optimization procedures to minimize the cost function. Detailed experimental results are provided to illustrate the performance of the proposed algorithm using digital video from an infrared imager.

Acoustic shape optimization using the finite element method

Information to predict the sensitivity of acoustical response to geometric shape variations can be calculated using the acoustic finite element formulation [R. J. Bernhard, J. Sound Vib. 98, 55–65 (1985)]. Such sensitivity information can be utilized to indicate to a designer the change of each geometric design variable necessary to achieve a better solution. The sensitivity information also indicates which design variables have minimal effect on the solution and how close the current solution is to an optimal condition. When the condition of the design can be quantified with a performance equation, numerical optimization procedures can be used to systematically find the optimal solution. Sensitivity information also permits the use of efficient gradient optimization algorithms. This paper will illustrate the incorporation of the acoustic finite element shape sensitivity information into a gradient optimization procedure. The design procedures for reactive mufflers will be illustrated. When modal techniques are appropriate for the solution of a particular problem, the shape sensitivity information may also be utilized to find modal sensitivity information.

Optimization of three-dimensional computational grids

A method for generating and optimizing arbitrary three-dimensional boundary-confo rming computational grids has been developed. The smoothness and local orthogonality of the grid are maximized using a fast iterative procedure, and provision is made for clustering the optimized grid in selected regions. An opimal grid can be obtained iteratively, irrespective of the method used to generate the initial grid. Unacceptable grids and even singular grids (i.e., grids containing regions of overlap) can be made useful for computation using this method. Application of the method to several test cases shows that grids containing regions of overlap are typically untangled in 2-5 iterations and that the conjugate gradient optimization procedure converges to an optimized grid within 25 iterations. Taking advantage of the original properties of this method, a new concept for generating optimal three-dimensional computational grids is proposed. It consists in optimizing a first guess of the desired grid, using an imperfect grid generated by a simple, inexpensive method as input.

A theoretical reexamination of carbanions adjacent to sulfoxide, sulfone and sulfonium centres

The structures and proton affinities of CH 2SR 2 (R = H,CH 3), −H 2SOR (R = H,CH 3) and −CH 2SO 2R (R = H,CH 3) have been found at the 3-21G* (d-orbitals on sulfur) and 3-21G (no d-orbitals on sulfur) levels, using gradient optimization procedures. d-Orbital effects are important in these optimized structures. The factors leading to the structures now found, and the role of the d-orbitals, have been evaluated by quantitative PMO analyses of the wavefunctions.

The Lanczos phenomenon—An interpretation based upon conjugate gradient optimization

The equivalence in exact arithmetic of the Lanczos tridiagonalization procedure and the conjugate gradient optimization procedure for solving Ax = b, where A is a real symmetric, positive definite matrix, is well known. We demonstrate that a relaxed equivalence is valid in the presence of errors. Specifically we demonstrate that local ε-orthonormality of the Lanczos vectors guarantees local ε- A-conjugacy of the direction vectors in the associated conjugate gradient procedure. Moreover we demonstrate that all the conjugate gradient relationships are satisfied approximately. Therefore, any statements valid for the conjugate gradient optimization procedure, which we show converges under very weak conditions, apply directly to the Lanczos procedure. We then use this equivalence to obtain an explanation of the Lanczos phenomenon: the empirically observed “convergence” of Lanczos eigenvalue procedures despite total loss of the global orthogonality of the Lanczos vectors.

Geometry optimization in ab initio molecular orbital theory

Abstract A gradient optimization procedure has been used to calculate equilibrium geometries for several small molecules in the framework of ab initio molecular orbital theory. The gradient method was found to be faster and more reliable than two direct search procedures.

A New Algorithm for Automatic History Matching

Abstract History-matching problems, in which reservoir parameters are to be estimated from well pressure parameters are to be estimated from well pressure data, are formulated as optimal control problems. The necessary conditions for optimality lead naturally to gradient optimization methods for determining the optimal parameter estimates. the key feature of the approach is that reservoir properties are considered as continuous functions properties are considered as continuous functions of position rather than as uniform in a certain number of zones. The optimal control approach is illustrated on a hypothetical reservoir and on an actual Saudi Arabian reservoir, both characterized by single-phase flow. A significant saving in computing time over conventional constant-zone gradient optimization methods is demonstrated. Introduction The process of determining in a mathematical reservoir model unknown parameter valuessuch as permeability and porositythat give the closest permeability and porositythat give the closest fit of measured and calculated pressures is commonly called "history matching." In principle, one would like an automatic routine for history matching, applicable to simulators of varying complexity, one that does not require inordinate amounts of computing time to achieve a set of parameter estimates. In recent years a number of authors have investigated the subject of history matching. All the reported approaches involve dividing the reservoir into a number of zones, in each of which the properties to be estimated are assumed to be uniform. (These zones may, in fact, correspond to the spatial grid employed for the finite-difference solution of the simulator.) Then the history-matching problem becomes that of determining the parameter problem becomes that of determining the parameter values in each of, say, N zones, k1, k2, ..., kN, in such a way that some measure (usually a sum of squares) of the deviation between calculated and observed pressures is minimized. A typical measure of deviation pressures is minimized. A typical measure of deviation is(1) where p obs (j, ti) and p cal (j, ti) are the observed and calculated pressures at the jth well, which is at location j=(xj, yj), j = 1,2,......, M, and where we have n1 measurements at Well 1 at n1 different times, n2 measurements at Well 2 at n2 different times, . . ., and nM measurements at Well M at nM different times. To carry out the minimization of Eq. 1 with respect to the vector k, most methods rely on some type of gradient optimization procedure that requires computation of the gradient of J with respect to each ki, i = 1, 2, . . ., N. The calculation of J/ ki usually requires, in turn, that one obtain the sensitivity coefficients, p cal/ ki, i = 1, 2, . . ., N; i.e., the first partial derivative of pressure with respect to each parameter. The sensitivity coefficients can be computed, in principle, in several ways. 1. Make a simulator base run with all N parameters at their initial values. Then, perturbing each parameter a small amount, make an additional simulator run for each parameter in the system. parameter in the system. SPEJ P. 593