Gradient Approximation Research Articles

A Risk-Sensitive Approach to Policy Optimization

Standard deep reinforcement learning (DRL) aims to maximize expected reward, considering collected experiences equally in formulating a policy. This differs from human decision-making, where gains and losses are valued differently and outlying outcomes are given increased consideration. It also fails to capitalize on opportunities to improve safety and/or performance through the incorporation of distributional context. Several approaches to distributional DRL have been investigated, with one popular strategy being to evaluate the projected distribution of returns for possible actions. We propose a more direct approach whereby risk-sensitive objectives, specified in terms of the cumulative distribution function (CDF) of the distribution of full-episode rewards, are optimized. This approach allows for outcomes to be weighed based on relative quality, can be used for both continuous and discrete action spaces, and may naturally be applied in both constrained and unconstrained settings. We show how to compute an asymptotically consistent estimate of the policy gradient for a broad class of risk-sensitive objectives via sampling, subsequently incorporating variance reduction and regularization measures to facilitate effective on-policy learning. We then demonstrate that the use of moderately "pessimistic" risk profiles, which emphasize scenarios where the agent performs poorly, leads to enhanced exploration and a continual focus on addressing deficiencies. We test the approach using different risk profiles in six OpenAI Safety Gym environments, comparing to state of the art on-policy methods. Without cost constraints, we find that pessimistic risk profiles can be used to reduce cost while improving total reward accumulation. With cost constraints, they are seen to provide higher positive rewards than risk-neutral approaches at the prescribed allowable cost.

Implicit Bilevel Optimization: Differentiating through Bilevel Optimization Programming

Bilevel Optimization Programming is used to model complex and conflicting interactions between agents, for example in Robust AI or Privacy preserving AI. Integrating bilevel mathematical programming within deep learning is thus an essential objective for the Machine Learning community. Previously proposed approaches only consider single-level programming. In this paper, we extend existing single-level optimization programming approaches and thus propose Differentiating through Bilevel Optimization Programming (BiGrad) for end-to-end learning of models that use Bilevel Programming as a layer. BiGrad has wide applicability and can be used in modern machine learning frameworks. BiGrad is applicable to both continuous and combinatorial Bilevel optimization problems. We describe a class of gradient estimators for the combinatorial case which reduces the requirements in terms of computation complexity; for the case of the continuous variable, the gradient computation takes advantage of the push-back approach (i.e. vector-jacobian product) for an efficient implementation. Experiments show that the BiGrad successfully extends existing single-level approaches to Bilevel Programming.

SSPAttack: A Simple and Sweet Paradigm for Black-Box Hard-Label Textual Adversarial Attack

Hard-label textual adversarial attack is a challenging task, as only the predicted label information is available, and the text space is discrete and non-differentiable. Relevant research work is still in fancy and just a handful of methods are proposed. However, existing methods suffer from either the high complexity of genetic algorithms or inaccurate gradient estimation, thus are arduous to obtain adversarial examples with high semantic similarity and low perturbation rate under the tight-budget scenario. In this paper, we propose a simple and sweet paradigm for hard-label textual adversarial attack, named SSPAttack. Specifically, SSPAttack first utilizes initialization to generate an adversarial example, and removes unnecessary replacement words to reduce the number of changed words. Then it determines the replacement order and searches for an anchor synonym, thus avoiding going through all the synonyms. Finally, it pushes substitution words towards original words until an appropriate adversarial example is obtained. The core idea of SSPAttack is just swapping words whose mechanism is simple. Experimental results on eight benchmark datasets and two real-world APIs have shown that the performance of SSPAttack is sweet in terms of similarity, perturbation rate and query efficiency.

Revisiting Item Promotion in GNN-Based Collaborative Filtering: A Masked Targeted Topological Attack Perspective

Graph neural networks (GNN) based collaborative filtering (CF) has attracted increasing attention in e-commerce and financial marketing platforms. However, there still lack efforts to evaluate the robustness of such CF systems in deployment. Fundamentally different from existing attacks, this work revisits the item promotion task and reformulates it from a targeted topological attack perspective for the first time. Specifically, we first develop a targeted attack formulation to maximally increase a target item's popularity. We then leverage gradient-based optimizations to find a solution. However, we observe the gradient estimates often appear noisy due to the discrete nature of a graph, which leads to a degradation of attack ability. To resolve noisy gradient effects, we then propose a masked attack objective that can remarkably enhance the topological attack ability. Furthermore, we design a computationally efficient approach to the proposed attack, thus making it feasible to evaluate large-large CF systems. Experiments on two real-world datasets show the effectiveness of our attack in analyzing the robustness of GNN-based CF more practically.

A note on gradient estimates for p-Laplacian equations

The aim of this short paper is to show that some assumptions in Guarnotta et al. (Adv Nonlinear Anal 11:741–756, 2022) can be relaxed and even dropped when looking for weak solutions instead of strong ones. This improvement is a consequence of two results concerning gradient terms: an L∞\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$L^\\infty $$\\end{document} estimate, which exploits nonlinear potential theory, and a compactness result, based on the classical Riesz-Fréchet-Kolmogorov theorem.

Correlation between energy band transition and optical absorption spectrum in bilayer armchair graphene nanoribbons

In this work, we investigate the intrinsic as well as modulated optical properties of the AB-stacking bilayer armchair graphene ribbons in the absence and presence of external electric fields. Single-layer ribbons are also considered for comparison. By using a tight-binding model in combination with the gradient approximation, we examine the energy bands, the density of states and the absorption spectra of the studied structures. Our results demonstrate that when external fields are not present, the low-frequency optical absorption spectra display numerous peaks and they vanish at the zero point. In addition, the number, the position, and the intensity of the absorption peaks are strongly associated with the ribbon width. With the wider ribbon width, more absorption peaks are present and a lower threshold absorption frequency is observed. Interestingly, in the presence of electric fields, bilayer armchair ribbons exhibit a lower threshold absorption frequency, more absorption peaks, and weaker spectral intensity. When increasing the strength of the electric field, the prominent peaks of the edge-dependent selection rules are lowered, and the sub-peaks satisfying the extra selection rules come to exist. The obtained results certainly provide a more comprehensive understanding of the correlation between the energy band transition and the optical absorption, in both single-layer and bilayer graphene armchair ribbons, and could provide new insights into developments of optoelectronic device applications based on graphene bilayer ribbons.

Optimal estimates for transmission problems including relative conductivities with different signs

We study the gradient and higher order derivative estimates for the transmission problem in the presence of closely located inclusions. We show that in two dimensions, when relative conductivities of circular inclusions have different signs, the gradient and higher order derivatives are bounded independent of ε, the distance between the inclusions. We also show that for general smooth strictly convex inclusions, when one inclusion is an insulator and the other one is a perfect conductor, the derivatives of any order is bounded independent of ε in any dimensions n≥2.

GoodCore: Data-effective and Data-efficient Machine Learning through Coreset Selection over Incomplete Data

Given a dataset with incomplete data (e.g., missing values), training a machine learning model over the incomplete data requires two steps. First, it requires a data-effective step that cleans the data in order to improve the data quality (and the model quality on the cleaned data). Second, it requires a data-efficient step that selects a core subset of the data (called coreset) such that the trained models on the entire data and the coreset have similar model quality, in order to improve the training efficiency. The first-data-effective-then-data-efficient methods are too costly, because they are expensive to clean the whole data; while the first-data-efficient-then-data-effective methods have low model quality, because they cannot select high-quality coreset for incomplete data. In this paper, we investigate the problem of coreset selection over incomplete data for data-effective and data-efficient machine learning. The essential challenge is how to model the incomplete data for selecting high-quality coreset. To this end, we propose the GoodCore framework towards selecting a good coreset over incomplete data with low cost. To model the unknown complete data, we utilize the combinations of possible repairs as possible worlds of the incomplete data. Based on possible worlds, GoodCore selects an expected optimal coreset through gradient approximation without training ML models. We formally define the expected optimal coreset selection problem, prove its NP-hardness, and propose a greedy algorithm with an approximation ratio. To make GoodCore more efficient, we further propose optimization methods that incorporate human-in-the-loop imputation or automatic imputation method into our framework. Experimental results show the effectiveness and efficiency of our framework with low cost.

Automatic Optimization of Multi-Well Multi-Stage Fracturing Treatments Combining Geomechanical Simulation, Reservoir Simulation and Intelligent Algorithm

Shale reserves have become an ever-increasing component of the world’s energy map. The optimal design of multi-well multi-stage fracturing (MMF) treatments is essential to the economic development of such resources. However, optimizing MMF treatments is a complex process. It requires geomechanical simulation, reservoir simulation, and automatic optimization. In this work, an integrated workflow is proposed to optimize MMF treatments in an unconventional reservoir, and the net present value (NPV) of reserves was treated as the objective function. The forward model consists of two submodels: a hydraulic fracturing model and a reservoir simulation model. The stochastic simplex approximation gradient (StoSAG) is used with the steepest ascent algorithm to maximize the NPV function. The computational results show that optimizing the fracture design can achieve a 20% higher NPV than that obtained with the field reference case. The drainage area of the optimal design is larger than that of the initial design. The maximum gas production rate increases from 23.75 MMSCF/day to 34.43 MMSCF/day and the maximum oil production rate increases from 497 STB/day to 692 STB/day. Therefore, new optimization paths can accelerate fracture design and help increase well production. This paper innovatively proposes a coupled workflow that can reduce the waste of manpower and improve the optimization results.

The perfect conductivity problem with arbitrary vanishing orders and non-trivial topology

The perfect conductivity problem concerns optimal bounds for the magnitude of an electric field in the presence of almost touching perfect conductors. This reduces to obtaining gradient estimates for harmonic functions with Dirichlet boundary conditions in the narrow region between the conductors. In this paper we extend estimates of Bao-Li-Yin to deal with the case when the boundaries of the conductors are given by graphs with arbitrary vanishing orders. Our estimates allow us to deal with globally defined narrow regions with possibly non-trivial topology.We also prove the sharpness of our estimates in terms of the distance between the perfect conductors. The precise optimality statement we give is new even in the setting of Bao-Li-Yin.

Elliptic gradient estimates for a nonlinear equation with Dirichlet boundary condition

We prove local elliptic type gradient estimates for a nonlinear parabolic equation (including the fast diffusion equation and the porous medium equation) with Dirichlet boundary condition on Riemannian manifolds with compact boundary when the Ricci curvature and the mean curvature of boundary are bounded below. As an application, we obtain Liouville type results for certain ancient solutions to the nonlinear parabolic equation on Riemannian manifolds with compact boundary.

Policy ensemble gradient for continuous control problems in deep reinforcement learning

Policy gradient algorithms for reinforcement learning (RL) have successfully tackled a broad range of high-dimensional continuous RL problems, including many challenging robotic control problems. These algorithms can be largely divided into two categories, i.e., on-policy algorithms and off-policy algorithms. Off-policy deep RL (DRL) algorithms enjoy better sample efficiency than and often outperform on-policy algorithms. However, cutting-edge off-policy algorithms still suffer from the low-quality estimation of policy gradients, resulting in compromised learning performance and high sensitivity to hyper-parameter settings. To address this issue, we propose a new concept of robust policy gradient (RPG). Driven by RPG, this paper further develops a new policy ensemble gradient (PEG) algorithm for DRL, inspired by the recent success of several ensemble DRL algorithms. PEG efficiently and effectively estimates RPG by using multiple policy gradients obtained respectively from several off-policy base learners in an ensemble. The estimated RPG is then utilized for training all base learners simultaneously. Comprehensive experiments have been performed on six Mujoco benchmark problems. Compared to four state-of-the-art off-policy algorithms and four cutting-edge ensemble policy gradient algorithms, our new PEG algorithm achieved highly competitive stability, performance and sample efficiency. Further analysis shows that PEG is insensitive to varied hyper-parameter settings, confirming the positive role of RPG in building reliable and effective off-policy DRL algorithms.

Full‐Waveform Tomography of the African Plate Using Dynamic Mini‐Batches

AbstractWe present a seismic model of the Africa Plate, constructed with the technique of full‐waveform inversion. The purpose of our model is to serve as a foundation for quantitative geodynamic and geochemical interpretation, earthquake‐induced ground motion predictions, and earthquake source inversion. Starting from the first‐generation Collaborative Seismic Earth Model, we invert seismograms filtered to a minimum period of 35 s and compute gradients of the misfit function with respect to the model parameters using the adjoint state method. We use dynamically changing mini‐batches of the complete data set to compute approximate gradients at each iteration. This approach has three significant advantages: (a) it reduces computational costs for model updates and the inversion, (b) it enables the use of larger datasets without increasing iteration costs, and (c) it makes it trivial to assimilate new data since we can extend the data set without changing the misfit function. We invert data from 397 unique earthquakes and 184,356 unique source‐receiver pairs. We clearly image tectonic features such as the Afar triple junction and low‐velocity zones below the Hoggar, Aïr, and Tibesti Mountains, pronounced more than in earlier works. Finally, we introduce a new strategy to assess model uncertainty. We deliberately perturb the final model, perform additional mini‐batch iterations, and compare the result with the original final model. This test uses actual seismic data instead of artificially generated synthetic data and requires no assumptions about the linearity of the inverse problem.

Harnack Estimation for Nonlinear, Weighted, Heat-Type Equation along Geometric Flow and Applications

The method of gradient estimation for the heat-type equation using the Harnack quantity is a classical approach used for understanding the nature of the solution of these heat-type equations. Most of the studies in this field involve the Laplace–Beltrami operator, but in our case, we studied the weighted heat equation that involves weighted Laplacian. This produces a number of terms involving the weight function. Thus, in this article, we derive the Harnack estimate for a positive solution of a weighted nonlinear parabolic heat equation on a weighted Riemannian manifold evolving under a geometric flow. Applying this estimation, we derive the Li–Yau-type gradient estimation and Harnack-type inequality for the positive solution. A monotonicity formula for the entropy functional regarding the estimation is derived. We specify our results for various different flows. Our results generalize some works.

Liquid temperature gradient estimation for screen channel liquid acquisition device bubble point tests in liquid nitrogen

The LAD screen utilizes surface tension forces in microgravity environments to allow liquid to flow through the screen while blocking vapor ingestion up to a certain pressure differential across the screen. The limiting pressure differential of LAD screen operation prior to bubble breakthrough is known as the bubble point pressure (ΔPBP). Previous cryogenic ΔPBP studies have identified how noncondensable pressurization decreases the interface temperature within the pores of a LAD screen via evaporative cooling and how autogenous pressurization increases the interface temperature within a LAD screen via condensation heating. The temperature gradient in the liquid side boundary layer near the screen caused by these effects has not been previously quantified. This study provides an order of magnitude analysis to show how certain variables can influence the liquid side temperature gradient during ΔPBP tests in liquid nitrogen (LN2) using noncondensable pressurization. Furthermore, cryogenic LN2ΔPBP data is taken for a 325x2300 Dutch Twill and 200x600 Dutch Twill screen with liquid side temperature measurements collected at varying distances away from the screen using dewar pressures ranging from ∼0-8psig (∼101.3-156.5kPa absolute). It has been shown that measuring the liquid screen side temperature 1/8″ (3.2 mm) away from the screen can result in a liquid side surface tension value that is ∼10-12.5% lower than the actual surface tension value at the interface for GHe tests in LN2. An empirical correlation is provided based on experimental temperature gradient data to correct historical and future ΔPBP data for the liquid screen side diode placement.

Adversarial example‐based test case generation for black‐box speech recognition systems

AbstractTest case generation techniques based on adversarial examples are commonly used to enhance the reliability and robustness of image‐based and text‐based machine learning applications. However, efficient techniques for speech recognition systems are still absent. This paper proposes a family of methods that generate targeted adversarial examples for speech recognition systems. All are based on the firefly algorithm (F), and are enhanced with gauss mutations and / or gradient estimation (F‐GM, F‐GE, F‐GMGE) to fit the specific problem of targeted adversarial test case generation. We conduct an experimental evaluation on three different types of speech datasets, including Google Command, Common Voice and LibriSpeech. In addition, we recruit volunteers to evaluate the performance of the adversarial examples. The experimental results show that, compared with existing approaches, these approaches can effectively improve the success rate of the targeted adversarial example generation. The code is publicly available at https://github.com/HanboCai/FGMGE.

A point cloud denoising network based on manifold in an unknown noisy environment

Acquiring original point cloud data from 3D sensors is easily affected by the environment and inherent limitations of the sensor, which inevitably contain noise and outliers. The irregularity and disorder of point clouds make denoising challenging. Traditional point cloud denoising methods heavily rely on geometric or noise priors. Recently, a deep learning denoising method based on gradient estimation has been proposed. Although it achieves the best denoising performance under Gaussian noise, the gradient can fluctuate under unknown noise, leading to decreased denoising performance. To address the problem that the above two algorithms have reduced denoising ability when facing unknown scenes, this paper proposes a denoising network based on manifold features of point clouds for unknown scenes. Specifically, the neural network consists of two key components: one is the manifold feature extraction module, which extracts manifold features and ensures accurate estimation of noise offset the other is the manifold denoising module, which can predict the noise offset of different disturbance points in the manifold by taking manifold features as input. Experiments show that our proposed method outperforms existing denoising methods on various noise model datasets and real outdoor datasets.

Gradient estimates for the Schrödinger potentials: convergence to the Brenier map and quantitative stability

We show convergence of the gradients of the Schrödinger potentials to the (uniquely determined) gradient of Kantorovich potentials in the small-time limit under general assumptions on the marginals, which allow for unbounded densities and supports. Furthermore, we provide novel quantitative stability estimates for the optimal values and optimal couplings for the Schrödinger problem (SP), that we express in terms of a negative order weighted homogeneous Sobolev norm. The latter encodes the linearized behavior of the 2-Wasserstein distance between the marginals. The proofs of both results highlight for the first time the relevance of gradient bounds for Schrödinger potentials, that we establish here in full generality, in the analysis of the short-time behavior of Schrödinger bridges. Finally, we discuss how our results translate into the framework of quadratic Entropic Optimal Transport, that is a version of SP more suitable for applications in machine learning and data science.

Souplet–Zhang and Hamilton‐type gradient estimates for non‐linear elliptic equations on smooth metric measure spaces

AbstractIn this article, we present new gradient estimates for positive solutions to a class of non‐linear elliptic equations involving the f‐Laplacian on a smooth metric measure space. The gradient estimates of interest are of Souplet–Zhang and Hamilton types, respectively, and are established under natural lower bounds on the generalised Bakry–Émery Ricci curvature tensor. From these estimates, we derive amongst other things Harnack inequalities and general global constancy and Liouville‐type theorems. The results and approach undertaken here provide a unified treatment and extend and improve various existing results in the literature. Some implications and applications are presented and discussed.

Gradient estimates and the fundamental solution for higher-order elliptic systems with lower-order terms

Abstract We establish the Caccioppoli inequality, a reverse Hölder inequality in the spirit of the classic estimate of Meyers, and construct the fundamental solution for linear elliptic differential equations of order 2 m 2m with certain lower order terms.