Neural Network Solver Research Articles

Exactly conservative physics-informed neural networks and deep operator networks for dynamical systems

We introduce a method for training exactly conservative physics-informed neural networks and physics-informed deep operator networks for dynamical systems, that is, for ordinary differential equations. The method employs a projection-based technique that maps a candidate solution learned by the neural network solver for any given dynamical system possessing at least one first integral onto an invariant manifold. We illustrate that exactly conservative physics-informed neural network solvers and physics-informed deep operator networks for dynamical systems vastly outperform their non-conservative counterparts for several real-world problems from the mathematical sciences.

Deep-Learning-Based Simulation and Inversion of Transient Electromagnetic Sounding Signals in Permafrost Monitoring Problem

Abstract —This article presents a novel approach to addressing the challenges in permafrost monitoring through the integration of deep-learning techniques with conventional electromagnetic sounding methods. Our methodology comprises a forward finite element method (FEM) solver, augmented with the Sumudu transform, and an artificial neural network (ANN) solver trained on FEM-generated data. Remarkably, the ANN solver demonstrates similar accuracy to the FEM solver but operates at a superior speed that is nearly 10,000 times faster. Furthermore, we introduce an inverse problem solution drawing on the PARS algorithm. In addition, we present an ANN-based inverse solver, where the input and output roles are inverted. The ANN inverse solver is trained on the same data, thereby offering an alternative approach to solving the inverse problem. In a computational experiment, we compare the numerical inversion results using the PARS algorithm with those obtained from the ANN forward solver, ANN inversion, and a linear combination of these solutions. This comprehensive analysis sheds light on the effectiveness of our deep-learning-based approach in permafrost monitoring, providing insights for future applications in geophysics and environmental science.

A robust radial point interpolation method empowered with neural network solvers (RPIM-NNS) for nonlinear solid mechanics

In this work, we proposed a robust radial point interpolation method empowered with neural network solvers (RPIM-NNS) for solving highly nonlinear solid mechanics problems. It is enabled by neural network solvers via minimizing an energy-based functional loss. The RPIM-NNS has the following key ingredients: (1) It uses radial basis functions (RBFs) for displacement interpolation at arbitrary points in the problem domain, permitting irregular node distributions. (2) Nodes are placed also beyond the domain boundary, allowing the convenient implementation of boundary conditions of both Dirichlet and Neumann types. (3) It uses strain energy in an integral form as a part of the loss function, ensuring solution stability. (4) A well-developed gradient descendant algorithm in machine learning is employed to find the optimal solution, enabling robustness and ease in handling material and geometrical nonlinearities. (5) The proposed RPIM-NNS is compatible with parallel computing schemes. The performance of this method is tested using nonlinear problems including Cook's membrane and 3D twisting rubber problems, demonstrating its remarkable stability and robustness. This work, which seamlessly integrates the neural network solvers with mechanics governing equations and computational mechanics techniques, offers an excellent alternative for nonlinear solid mechanics problems. MATLAB codes are made available at https://github.com/JinshuaiBai/RPIM_NNS for free downloading.

HNS: An efficient hermite neural solver for solving time-fractional partial differential equations

Neural network solvers represent an innovative and promising approach for tackling time-fractional partial differential equations by utilizing deep learning techniques. L1 interpolation approximation serves as the standard method for addressing time-fractional derivatives within neural network solvers. However, we have discovered that neural network solvers based on L1 interpolation approximation are unable to fully exploit the benefits of neural networks, and the accuracy of these models is constrained to interpolation errors. In this paper, we present the high-precision Hermite Neural Solver (HNS) for solving time-fractional partial differential equations. Specifically, we first construct a high-order explicit approximation scheme for fractional derivatives using Hermite interpolation techniques, and rigorously analyze its approximation accuracy. Afterward, taking into account the infinitely differentiable properties of deep neural networks, we integrate the high-order Hermite interpolation explicit approximation scheme with deep neural networks to propose the HNS. The experimental results show that HNS achieves higher accuracy than methods based on the L1 scheme for both forward and inverse problems, as well as in high-dimensional scenarios. This indicates that HNS has significantly improved accuracy and flexibility compared to existing L1-based methods, and has overcome the limitations of explicit finite difference approximation methods that are often constrained to function value interpolation. As a result, the HNS is not a simple combination of numerical computing methods and neural networks, but rather achieves a complementary and mutually reinforcing advantages of both approaches. The data and code can be found at https://github.com/hsbhc/HNS.

Physics-informed ConvNet: Learning physical field from a shallow neural network

We introduce a novel methodology for solving nonlinear partial differential equation (PDE) on regular or irregular domains using physics-informed ConvNet, which we call the PICN. The network structure consists of three parts: 1) a convolutional neural network for physical field generation, 2) a pre-trained convolutional layer corresponding to the finite-difference filters to estimate differential fields of the generated physical field, and 3) an interpolation network for loss analysis in irregular geometry domains. From a CNN perspective, the physical field is generated by a deconvolution layer and a convolution layer. Unlike the standard Physics-informed Neural Network (PINN) approach, the convolutions corresponding to the finite-difference filters estimate the spatial gradients forming the physical operator and then construct the PDE residual in a PINN-like loss function. The total loss function involving boundary conditions and the physical constraints in irregular geometry domains can be calculated from an efficient linear interpolation network. The theoretical analysis of PICN convergence is performed on a simplified case for solving a one-dimensional physical field, and several examples of nonlinear PDE of solutions with multifrequency characteristics are executed. The theory and examples confirm the effective learning capability of PICN for the physical field solution with high-frequency components, compared to the standard PINN. A series of numerical cases are performed to validate the current PICN, including the solving (and estimation) of nonlinear physical operator equations and recovering physical information from noisy observations. First, the ability of PICN to solve nonlinear PDE has been verified by executing three nonlinear problems including ODE with sine nonlinearity, PDE involving nonlinear sine-square operators, and Schrödinger equation. The proposed PICN has been assessed by solving some nonlinear PDE on irregular domains such as star-shaped domain, bird-like domain, and starfish domain. Moreover, PICN is applied to identify the thermal diffusivity parameters in an anisotropic heat transfer problem from noisy data, and a denoising display of the temperature field from strong noisy data with standard deviations ranging from 0.1 to 0.4. The numerical results demonstrate the high accuracy approximation and fast convergence performance of PICN. The potential advantage in approximating complex physical field with multi-frequency components indicates that PICN may become an alternative efficient neural network solver in physics-informed machine learning. This paper is adapted from the work originally posted on arXiv.com by the same authors (arXiv:2201.10967, Jan 26, 2022). The data and code accompanying this paper are publicly available at https://github.com/zengzhi2015/PICN.

The deep neural network solver for B-spline approximation

This paper introduces a novel unsupervised deep learning approach to address the knot placement problem in the field of B-spline approximation, called deep neural network solvers (DNN-Solvers). Given discrete points, the DNN acts as a solver for calculating knot positions in the case of a fixed knot number. The input can be any initial knots and the output provides the desirable knots. The loss function is based on the approximation error. The DNN-Solver converts the lower-dimensional knot placement problem, characterized as a nonconvex nonlinear optimization problem, into a search for suitable network parameters within a high-dimensional space. Owing to the over-parameterization nature, DNN-Solvers are less likely to be trapped in local minima and robust against initial knots. Moreover, the unsupervised learning paradigm of DNN-Solvers liberates us from constructing high-quality synthetic datasets with labels. Numerical experiments demonstrate that DNN-Solvers are excellent in both approximation results and efficiency under the premise of an appropriate number of knots.

Solution of conservative-form transport equations with physics-informed neural network

Physics-informed neural network (PINN) has recently gained significant attention in the field of scientific computing. This paper proposes a PINN framework for solving conservative-form transport equations (PINNforCTE). The PINNforCTE features an accelerated back-propagation in the neural network (NN) training due to low-order derivatives in the governing equations. We demonstrate the application of PINNforCTE in simulating a classical flow and heat transfer problem, i.e., the lid-driven cavity flow and heat transfer under three types of boundary conditions, and the simulation results are compared with those obtained from the traditional PINN and the finite element CFD method. The comparisons indicate that PINNforCTE can significantly accelerate the NN training and potentially shorten the solution time. For all cases considered, the training time of PINNforCTE is reduced by approximately ∼50% compared with the conventional PINN. Furthermore, PINNforCTE can accelerate the Adam optimization algorithm (L-BFGS-B optimization algorithm) by ∼100% (∼80%) compared to the traditional PINN. Additionally, PINNforCTE shows greater convenience in applying the Neumann and Robin boundary conditions directly and higher calculation accuracy than the conventional PINN. Finally, the ability of PINNforCTE to solve the inverse problems is confirmed with an example of 2D steady lid-driven cavity flow and heat transfer. PINNforCTE is an efficient NN solver for conservative-form transport equations (CTEs) that are ubiquitous in science and engineering.

DOSnet as a non-black-box PDE solver: When deep learning meets operator splitting

Deep neural networks (DNNs) recently emerged as a promising tool for analyzing and solving complex differential equations arising in science and engineering applications. Alternative to traditional numerical schemes, learning-based solvers utilize the representation power of DNNs to approximate the input-output relations in an automated manner. However, the lack of physics-in-the-loop often makes it difficult to construct a neural network solver that simultaneously achieves high accuracy, low computational burden, and interpretability. In this work, focusing on a class of evolutionary PDEs characterized by decomposable operators, we show that the classical “operator splitting” technique can be adapted to design neural network architectures. This gives rise to a learning-based PDE solver, which we name Deep Operator-Splitting Network (DOSnet). Such non-black-box network design is constructed from the physical rules and operators governing the underlying dynamics, and is more efficient and flexible than the classical numerical schemes and standard DNNs. To demonstrate the advantages of our new AI-enhanced PDE solver, we train and validate it on several types of operator-decomposable differential equations. We also apply DOSnet to nonlinear Schrödinger equations which have important applications in the signal processing for modern optical fiber transmission systems, and experimental results show that our model has better accuracy and lower computational complexity than numerical schemes and the baseline DNNs.

Kinematics-Based Motion-Force Control for Redundant Manipulators With Quaternion Control

Motion-force control of redundant manipulators is universally regarded as a pivotal issue in industrial manufacturing, especially for the processing of precision instruments. This paper proposes a kinematics-based motion-force control (KBMFC) scheme for redundant manipulators, which is driven by joint velocity commands and different from the dynamics-based methods. Specifically, the force and motion are modeled and decoupled in the end-effector frame with the help of a stiffness coefficient. To control the orientation of the force, a quaternion control equation is designed by combining the rotation matrix and neural dynamics method. Different from traditional motion-force control methods, the proposed scheme is constructed as quadratic programming with the corresponding recurrent neural network (RNN) solver derived, which considers the kinematic optimization index and joint constraints. According to the generated control signals, a redundant manipulator is able to accurately fulfill the hybrid control of motion and force with the desired quaternion, which is intuitively confirmed by simulations and experiments. <i>Note to Practitioners</i>&#x2014;This paper is motivated by the deficiencies that restrict the real-world applications of the motion-force control of redundant manipulators. On the one hand, most existing motion-force control schemes are implemented under the framework of dynamics, which inevitably leads to some kinematics-related defects. On the other hand, the latest kinematics-based techniques introduce an admittance control to achieve motion-force control. However, they model the force in the Z-axis of the base coordinate while the motion planning is limited in the X-Y plane, which dramatically reduces real-world applications. In this paper, the deformation force is designed in the end-effector frame, and a quaternion control technology of the end-effector is developed. Such a scheme can realize the real-time control of the orientation and magnitude of the force while ensuring trajectory tracking. In addition, the introduction of optimization indexes and joint constraints dramatically improves the functionality of the proposed scheme. Finally, the contributions of this paper are verified through simulations, experiments and comparisons. This work proposes a feasible framework for the motion-force control and orientation control of redundant manipulators.

Artificial neural network solver for time-dependent Fokker–Planck equations

Stochastic differential equations (SDEs) play a crucial role in various applications for modeling systems that have either random perturbations or chaotic dynamics at faster time scales. The time evolution of the probability distribution of a stochastic differential equation is described by the Fokker–Planck equation, which is a second order parabolic partial differential equation (PDE). Previous work combined artificial neural networks and Monte Carlo data to solve stationary Fokker–Planck equations. This paper extends this approach to time dependent Fokker–Planck equations. The main focus is on the investigation of algorithms for training a neural network that has multi-scale loss functions. Additionally, a new approach for collocation point sampling is proposed. A few 1D and 2D numerical examples are demonstrated.

NN-LCS: Neural Network and Linear Coordinate Solver Fusion Method for UWB Localization in Car Keyless Entry System.

Nowadays, ultra-wideband (UWB) technology is becoming a new approach to localize keyfobs in the car keyless entry system (KES), because it provides precise localization and secure communication. However, for vehicles the distance ranging suffers from great errors because of none-line-of-sight (NLOS) which is raised by the car. Regarding the NLOS problem, efforts have been made to mitigate the point-to-point ranging error or to estimate the tag coordinate by neural networks. However, it still suffers from some problems such as low accuracy, overfitting, or a large number of parameters. In order to address these problems, we propose a fusion method of a neural network and linear coordinate solver (NN-LCS). We use two FC layers to extract the distance feature and received signal strength (RSS) feature, respectively, and a multi-layer perceptron (MLP) to estimate the distances with the fusion of these two features. We prove that the least square method which supports error loss backpropagation in the neural network is feasible for distance correcting learning. Therefore, our model is end-to-end and directly outputs the localization results. The results show that the proposed method is high-accuracy and with small model size which could be easily deployed on embedded devices with low computing ability.

Ground-penetrating radar wavefield simulation via physics-informed neural network solver

The traditional methods for ground-penetrating radar (GPR) wavefield simulation suffer from numerical dispersion phenomena and difficulties for models with complex geometry. We propose to use the physics-informed neural networks (PINNs) to solve the GPR wave equation and model the wavefield propagation. Deep fully connected networks are used to approximate the solution to the equations. Automatic differentiation with back propagation is taken to calculate the partial derivatives. The loss function combining the wave equation, boundary condition, and temporal constraints is constructed and minimized to train the PINNs. As the solver is mesh-free, this simulation method avoids numerical dispersion artifacts and is very flexible in implementation for inversion. The numerical examples indicate that the PINNs solver for GPR wavefield simulation has high accuracy and efficiency. We also explore the possibility of inverting the electrical parameters from the observed wavefields. The results demonstrate the promising potential of the proposed approach in GPR forward modeling.

A physics-informed neural network based on mixed data sampling for solving modified diffusion equations

We developed a physics-informed neural network based on a mixture of Cartesian grid sampling and Latin hypercube sampling to solve forward and backward modified diffusion equations. We optimized the parameters in the neural networks and the mixed data sampling by considering the squeeze boundary condition and the mixture coefficient, respectively. Then, we used a given modified diffusion equation as an example to demonstrate the efficiency of the neural network solver for forward and backward problems. The neural network results were compared with the numerical solutions, and good agreement with high accuracy was observed. This neural network solver can be generalized to other partial differential equations.

A Chebyshev Polynomial Neural Network Solver for Boundary Value Problems of Elliptic Equations

A Chebyshev Polynomial Neural Network Solver for Boundary Value Problems of Elliptic Equations

Polynomial color compensation model with neural network solver for color-coded fringe patterns in phase-measuring deflectometry

The total image display/projection and image acquisition time in a projector-based phase-measuring deflectometry (PMD) measurement is reduced when color-coded fringe patterns are used since only a single image of the fringe pattern is sufficient to extract phase information using the phase-stepping method. This benefit is, however, offset by the difficulty in resolving various sources of color intensity errors of the fringe patterns in the camera view, such as uneven illumination, color imbalance, color crosstalk, and deviation of color intensity response of the camera and the projector from the calibrated color targets. Thus, a polynomial color compensation model is proposed in this study to compensate for color intensity errors. The model coefficients were estimated using the relationship between the captured and targeted color intensities, which are sequences of monochromatic 8-bit red, green, and blue (RGB) images for projection. The model was applied to modify the color intensities of the generated fringe patterns before projection. The modified color fringe patterns were then used for PMD measurement on spherical concave and convex mirrors. Based on the experimental results, the root-mean-squared (RMS) color intensity errors in R, G, and B channels were reduced by 33.3%, 58.3%, and 36.4%, respectively, after conducting the polynomial color compensation. The RMS phase errors in the fringe patterns were reduced by 71.9%. The RMS height errors and the radius percentage errors of the reconstructed concave and convex heightmaps were 1.87 μm (0.12%) and 1.55 μm (0.17%), respectively. The performance of the proposed polynomial color compensation model in reducing color intensity errors of the captured fringe patterns for accurate PMD measurement was demonstrated successfully.

Neural Eikonal solver: Improving accuracy of physics-informed neural networks for solving eikonal equation in case of caustics

The concept of physics-informed neural networks has become a useful tool for solving differential equations due to its flexibility. A few approaches use this concept to solve the eikonal equation that describes the first-arrival traveltimes of waves propagating in smooth heterogeneous velocity models. However, the challenge of the eikonal is exacerbated by the velocity models producing caustics, resulting in instabilities and deterioration of accuracy due to the non-smooth solution behavior. In this paper, we revisit the problem of solving the eikonal equation using neural networks to tackle caustic pathologies. We introduce the novel Neural Eikonal Solver (NES) for solving the isotropic eikonal equation in two formulations: the one-point problem is for a fixed source location; the two-point problem is for an arbitrary source-receiver pair. We present several techniques which provide stability in the case of caustics: improved factorization; non-symmetric loss function based on Hamiltonian; gaussian activation; symmetrization. In our tests, NES showed the relative mean-absolute error of 0.2-0.4% from the second-order factored Fast Marching Method with a similar inference time, and outperformed existing neural-network solvers giving 10-60 times lower errors and 2-30 times faster training. The one-point NES provides the most accurate solution, while the two-point NES gives slightly lower accuracy but an extremely compact representation with all spatial derivatives. It can be useful in many seismic problems: massive computations of traveltimes for millions of source-receiver pairs in Kirchhoff migration; modeling of ray amplitudes using spatial derivatives; traveltime tomography; earthquake localization; ray multipathing analysis.

Portfolio Insurance through Error-Correction Neural Networks

Minimum-cost portfolio insurance (MCPI) is a well-known investment strategy that tries to limit the losses a portfolio may incur as stocks decrease in price without requiring the portfolio manager to sell those stocks. In this research, we define and study the time-varying MCPI problem as a time-varying linear programming problem. More precisely, using real-world datasets, three different error-correction neural networks are employed to address this financial time-varying linear programming problem in continuous-time. These neural network solvers are the zeroing neural network (ZNN), the linear-variational-inequality primal-dual neural network (LVI-PDNN), and the simplified LVI-PDNN (S-LVI-PDNN). The neural network solvers are tested using real-world data on portfolios of up to 20 stocks, and the results show that they are capable of solving the financial problem efficiently, in some cases more than five times faster than traditional methods, though their accuracy declines as the size of the portfolio increases. This demonstrates the speed and accuracy of neural network solvers, showing their superiority over traditional methods in moderate-size portfolios. To promote and contend the outcomes of this research, we created two MATLAB repositories, for the interested user, that are publicly accessible on GitHub.

Neural Network Solver for Small Quantum Clusters

Machine learning approaches have recently been applied to the study of various problems in physics. Most of these studies are focused on interpreting the data generated by conventional numerical methods or the data on an existing experimental database. An interesting question is whether it is possible to use a machine learning approach, in particular a neural network, for solving the many-body problem. In this paper, we present a neural network solver for the single impurity Anderson model, the paradigm of an interacting quantum problem in small clusters. We demonstrate that the neural-network-based solver provides quantitative accurate results for the spectral function as compared to the exact diagonalization method. This opens the possibility of utilizing the neural network approach as an impurity solver for other many-body numerical approaches, such as the dynamical mean field theory.

Exploiting Mean-Variance Portfolio Optimization Problems through Zeroing Neural Networks

In this research, three different time-varying mean-variance portfolio optimization (MVPO) problems are addressed using the zeroing neural network (ZNN) approach. The first two MVPO problems are defined as time-varying quadratic programming (TVQP) problems, while the third MVPO problem is defined as a time-varying nonlinear programming (TVNLP) problem. Then, utilizing real-world datasets, the time-varying MVPO problems are addressed by this alternative neural network (NN) solver and conventional MATLAB solvers, and their performances are compared in three various portfolio configurations. The results of the experiments show that the ZNN approach is a magnificent alternative to the conventional methods. To publicize and explore the findings of this study, a MATLAB repository has been established and is freely available on GitHub for any user who is interested.

Linear Function Approximation as a Resource Efficient Method to Solve the Travelling Salesman Problem

This paper presents an approach to combinatorial optimization problems using linear function approximation (LFA) to solve the Travelling Salesman Problem (TSP). We create a reinforcement learning model in which we parameterize our policy using linear function approximation instead of the more commonly used neural networks. We then evaluated our models based on two factors: training time and optimality. When we compared our results with a state-of-the-art neural network solver, we found that our model was able to solve the TSP accurately while using drastically less computational resources and time to train than the neural network algorithm (Kool et al., 2019).