Partial Differential Equation Constraints Research Articles

On generalized Nash equilibrium problems in infinite-dimensional spaces using Nikaido–Isoda type functionals

We present an analysis of generalized Nash equilibrium problems in infinite-dimensional spaces with possibly non-convex objective functions of the players. Such settings arise, for instance, in games that involve nonlinear partial differential equation constraints. Due to non-convexity, we work with equilibrium concepts that build on first order optimality conditions, especially Quasi-Nash Equilibria (QNE), i.e. first-order optimality conditions for (Generalized) Nash Equilibria, and Variational Equilibria (VE), i.e. first-order optimality conditions for Normalized Nash Equilibria. We prove existence of these types of equilibria and study characterizations of them via regularized (and localized) Nikaido-Isoda merit functions. We also develop continuity and (continuous) differentiability results for these merit functions under quite weak assumptions, using a generalization of Danskin's theorem. They provide a theoretical foundation for, e.g. using globalized descent methods for computing QNE or VE.

TCAS-PINN: Physics-informed neural networks with a novel temporal causality-based adaptive sampling method

Physics-informed neural networks (PINNs) have become an attractive machine learning framework for obtaining solutions to partial differential equations (PDEs). PINNs embed initial, boundary, and PDE constraints into the loss function. The performance of PINNs is generally affected by both training and sampling. Specifically, training methods focus on how to overcome the training difficulties caused by the special PDE residual loss of PINNs, and sampling methods are concerned with the location and distribution of the sampling points upon which evaluations of PDE residual loss are accomplished. However, a common problem among these original PINNs is that they omit special temporal information utilization during the training or sampling stages when dealing with an important PDE category, namely, time-dependent PDEs, where temporal information plays a key role in the algorithms used. There is one method, called Causal PINN, that considers temporal causality at the training level but not special temporal utilization at the sampling level. Incorporating temporal knowledge into sampling remains to be studied. To fill this gap, we propose a novel temporal causality-based adaptive sampling method that dynamically determines the sampling ratio according to both PDE residual and temporal causality. By designing a sampling ratio determined by both residual loss and temporal causality to control the number and location of sampled points in each temporal sub-domain, we provide a practical solution by incorporating temporal information into sampling. Numerical experiments of several nonlinear time-dependent PDEs, including the Cahn–Hilliard, Korteweg–de Vries, Allen–Cahn and wave equations, show that our proposed sampling method can improve the performance. We demonstrate that using such a relatively simple sampling method can improve prediction performance by up to two orders of magnitude compared with the results from other methods, especially when points are limited.

Physics-informed kernel function neural networks for solving partial differential equations

This paper proposes an improved version of physics-informed neural networks (PINNs), the physics-informed kernel function neural networks (PIKFNNs), to solve various linear and some specific nonlinear partial differential equations (PDEs). It can also be considered as a novel radial basis function neural network (RBFNN). In the proposed PIKFNNs, it employs one-hidden-layer shallow neural network with the physics-informed kernel functions (PIKFs) as the customized activation functions. The PIKFs fully or partially contain PDE information, which can be chosen as fundamental solutions, green's functions, T-complete functions, harmonic functions, radial Trefftz functions, probability density functions and even the solutions of some linear simplified PDEs and so on. The main difference between the PINNs and the proposed PIKFNNs is that the PINNs add PDE constraints to the loss function, and the proposed PIKFNNs embed PDE information into the activation functions of the neural network. The feasibility and accuracy of the proposed PIKFNNs are validated by some benchmark examples.

Deep learning methods for blood flow reconstruction in a vessel with contrast enhanced x-ray computed tomography.

The reconstruction of blood velocity in a vessel from contrast enhanced x-ray computed tomography projections is a complex inverse problem. It can be formulated as reconstruction problem with a partial differential equation constraint. A solution can be estimated with the a variational adjoint method and proper orthogonal decomposition (POD) basis. In this work, we investigate new inversion approaches based on PODs coupled with deep learning methods. The effectiveness of the reconstruction methods is shown with simulated realistic stationary blood flows in a vessel. The methods outperform the reduced adjoint method and show large speed-up at the online stage.

Function-space optimization to coordinate multi-energy storage across the integrated electricity and natural gas system

The synergy between electric power systems and natural gas systems brings significant energy storage potentials yet challenges the modeling and solution techniques of partial-differential-equation (PDE) constrained optimization. This paper proposes a function-space optimization to coordinate multi-energy storage across the integrated electricity and natural gas system. Compared to general algebraic-space optimization, the proposed function-space optimization (FSO) uses continuous spatial–temporal functions as variables, for which time and space can be explicitly expressed and the PDE constraints of gas dynamics can be directly embedded into optimization. Considering the bidirectional energy conversion enabled by gas-fired units and power-to-gas, linepack gas storage and battery energy storage are coordinated through FSO to improve the operational economy. For tractable calculation, the Bernstein polynomial spline-based solution space transformation is adopted to transform FSO into mixed-integer linear programming. Case studies on a single-pipe example and two practical test systems are conducted and the simulation results validate the effectiveness of the proposed FSO method.

A New Stopping Criterion for Krylov Solvers Applied in Interior Point Methods

When an iterative method is applied to solve the linear equation system in interior point methods (IPMs), the attention is usually placed on accelerating their convergence by designing appropriate preconditioners, but the linear solver is applied as a black box with a standard termination criterion which asks for a sufficient reduction of the residual in the linear system. Such an approach often leads to an unnecessary 'oversolving' of linear equations. In this paper, an IPM that relies on an inner termination criterion not based on the residual of the linear system is introduced and analyzed. Moreover, new indicators for the early termination of the inner iterations are derived from a deep understanding of IPM needs. The new technique has been adapted to the Conjugate Gradient (CG) and to the Minimum Residual method (MINRES) applied in the IPM context. The new criterion has been tested on a set of quadratic optimization problems including compressed sensing, image processing and instances with partial differential equation constraints, and it has been compared to standard residual tests with variable tolerance. Evidence gathered from these computational experiments shows that the new technique delivers significant improvements in terms of inner (linear) iterations and those translate into significant savings of the IPM solution time.

Knowledge-informed generative adversarial network for functional calibration of computer models

Functional model calibration considers the input dependency of model parameters by explicitly considering them as functional calibration parameters. In this study, a novel knowledge-informed generative adversarial network (KI-GAN) is proposed for the functional calibration of computer models under uncertainty. The proposed KI-GAN leverages the superior distribution learning ability of generative models for uncertainty quantification of model parameters and uses prior knowledge to inform generative model training by formulating a knowledge-based loss function that can easily incorporate different types of physical and expert knowledge of the computer model and its parameters. Two experiments are presented to demonstrate the effectiveness of the proposed KI-GAN for calibrating the functional model parameters under various scenarios. The results indicate that the proposed KI-GAN can effectively encode various types of prior knowledge on the computer model and its parameters, such as partial differential equation constraints and derivative constraints, which can significantly improve the parameter estimation accuracy and reduce estimation uncertainty. Furthermore, with appropriate constraints applied, the proposed KI-GAN can estimate model parameters with reasonable accuracy even under sparse and noisy observations and quantify the underlying uncertainties. Finally, the trained generators can accurately infer the probabilistic distributions of unobserved model outputs without running the computer model.

Using Legendre Spectral Element Method with Split Bregman Method for Solving Optimal Control Problem with Elliptic Partial Differential Equation Constraint

Using Legendre Spectral Element Method with Split Bregman Method for Solving Optimal Control Problem with Elliptic Partial Differential Equation Constraint

Learning-PDE-Based Approximate Optimal Control for an MHD System With Uncertainty Quantification

Handling uncertainty is one of the most important challenges in real physical systems. In this article, we study an approximate optimal magnetic control strategy for a one-dimensional (1-D) magnetohydrodynamic (MHD) system with uncertainty quantification within the learning framework of the underlying MHD model. First, the MHD flow system is modeled by coupled partial differential equations (PDEs) wherein the Reynolds number is not deterministic but random. Then, the optimal magnetic control problem is formulated and reduced to a parameter selection problem with stochastic PDE constraints by means of the control parameterization method. Significantly different from the conventional sensitivity analysis and adjoint methods, a polynomial chaos expansion (PCE) based on a multifidelity model is developed to construct the underlying PDEs model by polynomial functions. Thus, the relationship between the objective function and the control sequence is derived explicitly, which, in turn, transforms the optimal parameter selection problem with stochastic PDE constraints into a typical algebraic optimization problem that can be easily solved by the existing nonlinear programming algorithm. Numerical simulations are illustrated to demonstrate the high performance of our proposed method.

Algorithmic Differentiation for Automated Modeling of Machine Learned Force Fields.

Reconstructing force fields (FFs) from atomistic simulation data is a challenge since accurate data can be highly expensive. Here, machine learning (ML) models can help to be data economic as they can be successfully constrained using the underlying symmetry and conservation laws of physics. However, so far, every descriptor newly proposed for an ML model has required a cumbersome and mathematically tedious remodeling. We therefore propose using modern techniques from algorithmic differentiation within the ML modeling process, effectively enabling the usage of novel descriptors or models fully automatically at an order of magnitude higher computational efficiency. This paradigmatic approach enables not only a versatile usage of novel representations and the efficient computation of larger systems─all of high value to the FF community─but also the simple inclusion of further physical knowledge, such as higher-order information (e.g., Hessians, more complex partial differential equations constraints etc.), even beyond the presented FF domain.

Automated computation of topological derivatives with application to nonlinear elasticity and reaction–diffusion problems

While topological derivatives have proven useful in applications of topology optimization and inverse problems, their mathematically rigorous derivation remains an ongoing research topic, in particular in the context of nonlinear partial differential equation (PDE) constraints. We present a systematic yet formal approach for the numerical computation of topological derivatives of a large class of PDE-constrained topology optimization problems with respect to arbitrary inclusion shapes. Scalar and vector-valued as well as linear and nonlinear elliptic PDE constraints are considered in two and three space dimensions including a nonlinear elasticity model and nonlinear reaction–diffusion problems. The systematic procedure follows a Lagrangian approach for computing topological derivatives. For problems where the exact formula is known, the numerically computed values show good coincidence. Moreover, by inserting the computed values into the topological asymptotic expansion, we verify that the obtained values satisfy the expected behavior also for other, previously unknown problems, indicating the correctness of the procedure.

Challenges in Optimization with Complex PDE-Systems

The workshop concentrated on various aspects of optimization problems with systems of nonlinear partial differential equations (PDEs) or variational inequalities (VIs) as constraints. In particular, discussions around several keynote presentations in the areas of optimal control of nonlinear or non-smooth systems, optimization problems with functional and discrete or switching variables leading to mixed integer nonlinear PDE constrained optimization, shape and topology optimization, feedback control and stabilization, multi-criteria problems and multiple optimization problems with equilibrium constraints as well as versions of these problems under uncertainty or stochastic influences, and the respectively associated numerical analysis as well as design and analysis of solution algorithms were promoted. Moreover, aspects of optimal control of data-driven PDE constraints (e.g. related to machine learning) were addressed.

Physics constrained learning for data-driven inverse modeling from sparse observations

Deep neural networks (DNN) can model nonlinear relations between physical quantities. Those DNNs are embedded in physical systems described by partial differential equations (PDE) and trained by minimizing a loss function that measures the discrepancy between predictions and observations in some chosen norm. This loss function often includes the PDE constraints as a penalty term when only sparse observations are available. As a result, the PDE is only satisfied approximately by the solution. However, the penalty term typically slows down the convergence of the optimizer for stiff problems. We present a new approach that trains the embedded DNNs while numerically satisfying the PDE constraints. We developed an algorithm that enables differentiating both explicit and implicit numerical solvers in reverse-mode automatic differentiation. Our method allows the gradients of the DNNs and the PDE solvers to be computed in a unified framework. We demonstrate that our approach enjoys faster convergence and better stability in relatively stiff problems compared to the penalty method. Our approach could solve and accelerate a wide range of data-driven inverse modeling, where the physical constraints are described by PDEs and need to be satisfied accurately.

Randomized Source Sketching for Full Waveform Inversion

Partial differential equation (PDE) constrained optimization problems such as seismic full waveform inversion (FWI) frequently arise in the geoscience and related fields. For such problems, many observations are usually gathered by multiple sources, which form the right-hand-sides of the PDE constraint. Solving the inverse problem with such massive data sets is computationally demanding, in particular when dealing with large number of model parameters. This paper proposes a novel randomized source sketching method for the efficient resolution of multisource PDE constrained optimization problems. We first formulate the different source-encoding strategies used in seismic imaging into a unified framework based on a randomized sketching. To this end, the source dimension of the problem is projected in a smaller domain by a suitably defined projection matrix that gathers the physical sources in super-sources through a weighted summation. This reduction in the number of physical sources decreases significantly the number of PDE solves while suitable sparsity-promoting regularization can efficiently mitigate the footprint of the cross-talk noise to maintain the convergence speed of the algorithm. We implement the randomized sketching method in an extended search-space formulation of frequency-domain FWI, which relies on the alternating direction method of multipliers (ADMM). Numerical examples carried out with a series of well documented 2D benchmarks demonstrate that the randomized sketching algorithm reduces the cost of large-scale problems by at least one order of magnitude compared to the original deterministic algorithm.

On PDE Characterization of Smooth Hierarchical Functions Computed by Neural Networks.

Neural networks are versatile tools for computation, having the ability to approximate a broad range of functions. An important problem in the theory of deep neural networks is expressivity; that is, we want to understand the functions that are computable by a given network. We study real, infinitely differentiable (smooth) hierarchical functions implemented by feedforward neural networks via composing simpler functions in two cases: (1) each constituent function of the composition has fewer inputs than the resulting function and (2) constituent functions are in the more specific yet prevalent form of a nonlinear univariate function (e.g., tanh) applied to a linear multivariate function. We establish that in each of these regimes, there exist nontrivial algebraic partial differential equations (PDEs) that are satisfied by the computed functions. These PDEs are purely in terms of the partial derivatives and are dependent only on the topology of the network. Conversely, we conjecture that such PDE constraints, once accompanied by appropriate nonsingularity conditions and perhaps certain inequalities involving partial derivatives, guarantee that the smooth function under consideration can be represented by the network. The conjecture is verified in numerous examples, including the case of tree architectures, which are of neuroscientific interest. Our approach is a step toward formulating an algebraic description of functional spaces associated with specific neural networks, and may provide useful new tools for constructing neural networks.

Cluster‐based gradient method for stochastic optimal control problems with elliptic partial differential equation constraint

In this article, we establish a cluster-based gradient method (CGM) by combining K-means clustering algorithm and stochastic gradient descent (SGD). By clustering the sampling solutions, we use the cluster centroids to represent sampling data and give an estimate to the full gradient. It is well known that the full gradient descent (FGD) can provide the steepest descent direction for finding a local minimum of the desired stochastic control problems. However, the huge computational requirements, which is proportional to the product of sample size and the numerical cost for each sample, often makes FGD cost prohibitive for large scale optimization problems. To reduce the formidable cost and the risks of getting stuck in a local minimum, SGD is proposed and can be regarded as a stochastic approximation of FGD. This, however, would result in a slow convergence due to the incorrect approximation of the iteratively update parameters. Our study shows that CGM could provide a good stochastic approximation to the full gradient with small sample size while has a more stable and faster convergence than SGD. To verify our algorithm, a stochastic elliptic control problem is selected and tested. The numerical results validate our method as a reliable gradient descent method with great potential applications in optimization problems.

Optimization with Partial Differential Equation Constraints for Bio-imaging Applications

Molecular transport and interaction are of funda- mental importance in biology and medicine. The spatiotemporal diffusion map can reflect the regulation of molecular interactions and their intracellular functions. To construct subcellular diffusion maps based on bio-imaging data, we explore a general optimization framework with diffusion equation constraints (OPT-PDE). For the solution of the spatially piecewise constant and anisotropic diffusion.

Full-space approach to aerodynamic shape optimization

Aerodynamic shape optimization (ASO) involves finding an optimal surface while constraining a set of nonlinear partial differential equations (PDE). The conventional approaches use quasi-Newton methods operating in the reduced-space, where the PDE constraints are eliminated at each design step by decoupling the flow solver from the optimizer. Conversely, the full-space Lagrange-Newton-Krylov-Schur (LNKS) approach couples the design and flow iteration by simultaneously minimizing the objective function and improving feasibility of the PDE constraints, which requires fewer iterations of the forward problem. Additionally, the use of second-order information leads to a number of design cycles independent of the number of control variables. We discuss the necessary ingredients to build an efficient LNKS ASO framework as well as the intricacies of their implementation. The LNKS approach is then compared to reduced-space approaches on a benchmark two-dimensional test case using a high-order discontinuous Galerkin method to discretize the PDE constraint.

Stochastic proximal gradient methods for nonconvex problems in Hilbert spaces

For finite-dimensional problems, stochastic approximation methods have long been used to solve stochastic optimization problems. Their application to infinite-dimensional problems is less understood, particularly for nonconvex objectives. This paper presents convergence results for the stochastic proximal gradient method applied to Hilbert spaces, motivated by optimization problems with partial differential equation (PDE) constraints with random inputs and coefficients. We study stochastic algorithms for nonconvex and nonsmooth problems, where the nonsmooth part is convex and the nonconvex part is the expectation, which is assumed to have a Lipschitz continuous gradient. The optimization variable is an element of a Hilbert space. We show almost sure convergence of strong limit points of the random sequence generated by the algorithm to stationary points. We demonstrate the stochastic proximal gradient algorithm on a tracking-type functional with a L^1-penalty term constrained by a semilinear PDE and box constraints, where input terms and coefficients are subject to uncertainty. We verify conditions for ensuring convergence of the algorithm and show a simulation.

Finite Element and Split Bregman Methods for Solving a Family of Optimal Control Problem with Partial Differential Equation Constraint

Finite Element and Split Bregman Methods for Solving a Family of Optimal Control Problem with Partial Differential Equation Constraint