Neural Solver Research Articles

Neural differentiable modeling with diffusion-based super-resolution for two-dimensional spatiotemporal turbulence

Simulating spatiotemporal turbulence with high fidelity remains a cornerstone challenge in computational fluid dynamics (CFD) due to its intricate multiscale nature and prohibitive computational demands. Traditional approaches typically employ closure models, which attempt to represent small-scale features in an unresolved manner. However, these methods often sacrifice accuracy and lose high-frequency/wavenumber information, especially in scenarios involving complex flow physics. In this paper, we introduce an innovative neural differentiable modeling framework designed to enhance the predictability and efficiency of spatiotemporal turbulence simulations. Our approach features differentiable hybrid modeling techniques that seamlessly integrate deep neural networks with numerical PDE solvers within a differentiable programming framework, synergizing deep learning with physics-based CFD modeling. Specifically, a hybrid differentiable neural solver is constructed on a coarser grid to capture large-scale turbulent phenomena, followed by the application of a Bayesian conditional diffusion model that generates small-scale turbulence conditioned on large-scale flow predictions. Two innovative hybrid architecture designs are studied, and their performance is evaluated through comparative analysis against conventional large eddy simulation techniques with physics-based subgrid-scale closures and purely data-driven neural solvers. The findings underscore the potential of the neural differentiable modeling framework to significantly enhance the accuracy and computational efficiency of turbulence simulations. This study not only demonstrates the efficacy of merging deep learning with physics-based numerical solvers but also sets a new precedent for advanced CFD modeling techniques, highlighting the transformative impact of differentiable programming in scientific computing.

Large-scale flood modeling and forecasting with FloodCast

Large-scale hydrodynamic models generally rely on fixed-resolution spatial grids and model parameters as well as incurring a high computational cost. This limits their ability to accurately forecast flood crests and issue time-critical hazard warnings. In this work, we build a fast, stable, accurate, resolution-invariant, and geometry-adaptive flood modeling and forecasting framework that can perform at large scales, namely FloodCast. The framework comprises two main modules: multi-satellite observation and hydrodynamic modeling. In the multi-satellite observation module, a real-time unsupervised change detection method and a rainfall processing and analysis tool are proposed to harness the full potential of multi-satellite observations in large-scale flood prediction. In the hydrodynamic modeling module, a geometry-adaptive physics-informed neural solver (GeoPINS) is introduced, benefiting from the absence of a requirement for training data in physics-informed neural networks (PINNs) and featuring a fast, accurate, and resolution-invariant architecture with Fourier neural operators. To adapt to complex river geometries, we reformulate PINNs in a geometry-adaptive space. GeoPINS demonstrates impressive performance on popular partial differential equations across regular and irregular domains. Building upon GeoPINS, we propose a sequence-to-sequence GeoPINS model to handle long-term temporal series and extensive spatial domains in large-scale flood modeling. This model employs sequence-to-sequence learning and hard-encoding of boundary conditions. Next, we establish a benchmark dataset in the 2022 Pakistan flood using a widely accepted finite difference numerical solution to assess various flood simulation methods. Finally, we validate the model in three dimensions - flood inundation range, depth, and transferability of spatiotemporal downscaling - utilizing SAR-based flood data, traditional hydrodynamic benchmarks, and concurrent optical remote sensing images. Traditional hydrodynamics and sequence-to-sequence GeoPINS exhibit exceptional agreement during high water levels, while comparative assessments with SAR-based flood depth data show that sequence-to-sequence GeoPINS outperforms traditional hydrodynamics, with smaller simulation errors. The experimental results for the 2022 Pakistan flood demonstrate that the proposed method enables high-precision, large-scale flood modeling with an average MAPE of 14.93 % and an average Mean Absolute Error (MAE) of 0.0610 m for 14-day water depth simulations while facilitating real-time flood hazard forecasting using reliable precipitation data.

ASP: Learn a Universal Neural Solver!

Applying machine learning to combinatorial optimization problems has the potential to improve both efficiency and accuracy. However, existing learning-based solvers often struggle with generalization when faced with changes in problem distributions and scales. In this paper, we propose a new approach called ASP: Adaptive Staircase Policy Space Response Oracle to address these generalization issues and learn a universal neural solver. ASP consists of two components: Distributional Exploration, which enhances the solver's ability to handle unknown distributions using Policy Space Response Oracles, and Persistent Scale Adaption, which improves scalability through curriculum learning. We have tested ASP on several challenging COPs, including the traveling salesman problem, the vehicle routing problem, and the prize collecting TSP, as well as the real-world instances from TSPLib and CVRPLib. Our results show that even with the same model size and weak training signal, ASP can help neural solvers explore and adapt to unseen distributions and varying scales, achieving superior performance. In particular, compared with the same neural solvers under a standard training pipeline, ASP produces a remarkable decrease in terms of the optimality gap with 90.9% and 47.43% on generated instances and real-world instances for TSP, and a decrease of 19% and 45.57% for CVRP.

Graph neural network based human detection in videos during occlusion environments

One of the most difficult perceptual problems for many applications is accurately recognizing the human object in a variety of circumstances. This can be difficult due to obstructions, weather, complex backdrops, cast shadows, and occlusions. Occlusion is a challenging open problem where a detector can only perceive a portion of the target human because of obstacles in the surrounding. In this research, an experimental investigation was conducted using the multi object tracking (MOT17) datasets to construct a graph neural network-based solution for the detection of humans in videos while considering the possibility of occlusion. Graph neural network (GNN) is used for the construction of neural solver model for detecting human object in occlusion scenario. The results obtained shows that this proposed method offers a considerable improvement in efficiency in comparison to the ways that have been used in the past. The values obtained for the standard performance metrics are higher than the state-of-the-art methods.

Comparative study of typical neural solvers in solving math word problems

In recent years, there has been a significant increase in the design of neural network models for solving math word problems (MWPs). These neural solvers have been designed with various architectures and evaluated on diverse datasets, posing challenges in fair and effective performance evaluation. This paper presents a comparative study of representative neural solvers, aiming to elucidate their technical features and performance variations in solving different types of MWPs. Firstly, an in-depth technical analysis is conducted from the initial deep neural solver DNS to the state-of-the-art GPT-4. To enhance the technical analysis, a unified framework is introduced, which comprises highly reusable modules decoupled from existing MWP solvers. Subsequently, a testbed is established to conveniently reproduce existing solvers and develop new solvers by combing these reusable modules, and finely regrouped datasets are provided to facilitate the comparative evaluation of the designed solvers. Then, comprehensive testing is conducted and detailed results for eight representative MWP solvers on five finely regrouped datasets are reported. The comparative analysis yields several key findings: (1) Pre-trained language model-based solvers demonstrate significant accuracy advantages across nearly all datasets, although they suffer from limitations in math equation calculation. (2) Models integrated with tree decoders exhibit strong performance in generating complex math equations. (3) Identifying and appropriately representing implicit knowledge hidden in problem texts is crucial for improving the accuracy of math equation generation. Finally, the paper also discusses the major technical challenges and potential research directions in this field. The insights gained from this analysis offer valuable guidance for future research, model development, and performance optimization in the field of math word problem solving.

GLOP: Learning Global Partition and Local Construction for Solving Large-Scale Routing Problems in Real-Time

The recent end-to-end neural solvers have shown promise for small-scale routing problems but suffered from limited real-time scaling-up performance. This paper proposes GLOP (Global and Local Optimization Policies), a unified hierarchical framework that efficiently scales toward large-scale routing problems. GLOP hierarchically partitions large routing problems into Travelling Salesman Problems (TSPs) and TSPs into Shortest Hamiltonian Path Problems. For the first time, we hybridize non-autoregressive neural heuristics for coarse-grained problem partitions and autoregressive neural heuristics for fine-grained route constructions, leveraging the scalability of the former and the meticulousness of the latter. Experimental results show that GLOP achieves competitive and state-of-the-art real-time performance on large-scale routing problems, including TSP, ATSP, CVRP, and PCTSP. Our code is available at: https://github.com/henry-yeh/GLOP.

A pre-defined finite time neural solver for the time-variant matrix equation [formula omitted

During the process of the general practical applications and scientific researches, time varying issues are an inescapable challenge to be tackled. In this article, a pre-defined finite-time zeroing neural networks (PDFT-ZNN) is devoted to solve the linear time-variant matrix equation (TVME) E(t)X(t)G(t)=D(t) within a finite time (FT), which contrasts with the general ZNN (G-ZNN) models with relatively long global convergence time. Moreover, the proposed PDFT-ZNN model’s convergence time could be calculated in advance by adhering to the designed system parameters; this has nothing to do with the model’s initial state. Additionally, after the convergence analysis, if the solution error is relative small, the simple and effective method is to introduce a linear item to accelerate the convergence speed, in comparison with only the power-type ZNN models. Theory-based analysis and simulation-based results further validate that, the neural state solved by the presented FT-ZNN model can reach the theory-based solution of E(t)X(t)G(t)=D(t) within the pre-defined finite time.

HNS: An efficient hermite neural solver for solving time-fractional partial differential equations

Neural network solvers represent an innovative and promising approach for tackling time-fractional partial differential equations by utilizing deep learning techniques. L1 interpolation approximation serves as the standard method for addressing time-fractional derivatives within neural network solvers. However, we have discovered that neural network solvers based on L1 interpolation approximation are unable to fully exploit the benefits of neural networks, and the accuracy of these models is constrained to interpolation errors. In this paper, we present the high-precision Hermite Neural Solver (HNS) for solving time-fractional partial differential equations. Specifically, we first construct a high-order explicit approximation scheme for fractional derivatives using Hermite interpolation techniques, and rigorously analyze its approximation accuracy. Afterward, taking into account the infinitely differentiable properties of deep neural networks, we integrate the high-order Hermite interpolation explicit approximation scheme with deep neural networks to propose the HNS. The experimental results show that HNS achieves higher accuracy than methods based on the L1 scheme for both forward and inverse problems, as well as in high-dimensional scenarios. This indicates that HNS has significantly improved accuracy and flexibility compared to existing L1-based methods, and has overcome the limitations of explicit finite difference approximation methods that are often constrained to function value interpolation. As a result, the HNS is not a simple combination of numerical computing methods and neural networks, but rather achieves a complementary and mutually reinforcing advantages of both approaches. The data and code can be found at https://github.com/hsbhc/HNS.

Math Word Problem Solver using Machine Learning

This paper presents a deep neural solver to automatically solve math word problems. In contrast to previous statistical learning approaches, we directly translate math word problems to equation templates using a recurrent neural network (RNN) model, without sophisticated feature engineering. We further design a hybrid model that combines the RNN model and a similarity-based retrieval model to achieve additional performance improvement. Experiments conducted on a large dataset show that the RNN model and the hybrid model significantly outperform state of-the-art statistical learning methods for math word problem solving.

Qualia Role-Based Quantity Relation Extraction for Solving Algebra Story Problems

A qualia role-based entity-dependency graph (EDG) is proposed to represent and extract quantity relations for solving algebra story problems stated in Chinese. Traditional neural solvers use end-to-end models to translate problem texts into math expressions, which lack quantity relation acquisition in sophisticated scenarios. To address the problem, the proposed method leverages EDG to represent quantity relations hidden in qualia roles of math objects. Algorithms were designed for EDG generation and quantity relation extraction for solving algebra story problems. Experimental result shows that the proposed method achieved an average accuracy of 82.2% on quantity relation extraction compared to 74.5% of baseline method. Another prompt learning result shows a 5% increase obtained in problem solving by injecting the extracted quantity relations into the baseline neural solvers.

An adaptive gradient-descent-based neural networks for the on-line solution of linear time variant equations and its applications

It is well-known that, the classical gradient-descent-based neural network (CGNN) model is used widely for the time-invariant problem solving. However, it is an extremely common problem for the time varying cases in the practical engineering applications, while the CGNN effective neural solver for the time-variant problems. For this reason, in this article, an adaptive GNN (AGNN) is presented for the linear time variant matrix equation (LTVME) on-line solving based on Lyapunov theory. Theoretical analysis already verified that the presented AGNN model could achieve the correct state solution effectively. The simulated experiment results further validate that the state solution of the AGNN model could be convergent to the correct solution of the solved time variant problems in theory.

Fourier Neural Solver for Large Sparse Linear Algebraic Systems

Large sparse linear algebraic systems can be found in a variety of scientific and engineering fields and many scientists strive to solve them in an efficient and robust manner. In this paper, we propose an interpretable neural solver, the Fourier neural solver (FNS), to address them. FNS is based on deep learning and a fast Fourier transform. Because the error between the iterative solution and the ground truth involves a wide range of frequency modes, the FNS combines a stationary iterative method and frequency space correction to eliminate different components of the error. Local Fourier analysis shows that the FNS can pick up on the error components in frequency space that are challenging to eliminate with stationary methods. Numerical experiments on the anisotropic diffusion equation, convection–diffusion equation, and Helmholtz equation show that the FNS is more efficient and more robust than the state-of-the-art neural solver.

Geometry-Aware Neural Solver for Fast Bayesian Calibration of Brain Tumor Models.

Modeling of brain tumor dynamics has the potential to advance therapeutic planning. Current modeling approaches resort to numerical solvers that simulate the tumor progression according to a given differential equation. Using highly-efficient numerical solvers, a single forward simulation takes up to a few minutes of compute. At the same time, clinical applications of tumor modeling often imply solving an inverse problem, requiring up to tens of thousands of forward model evaluations when used for a Bayesian model personalization via sampling. This results in a total inference time prohibitively expensive for clinical translation. While recent data-driven approaches become capable of emulating physics simulation, they tend to fail in generalizing over the variability of the boundary conditions imposed by the patient-specific anatomy. In this paper, we propose a learnable surrogate for simulating tumor growth which maps the biophysical model parameters directly to simulation outputs, i.e. the local tumor cell densities, whilst respecting patient geometry. We test the neural solver in a Bayesian model personalization task for a cohort of glioma patients. Bayesian inference using the proposed surrogate yields estimates analogous to those obtained by solving the forward model with a regular numerical solver. The near real-time computation cost renders the proposed method suitable for clinical settings. The code is available at https://github.com/IvanEz/tumor-surrogate.

A NEURAL SOLVER FOR VARIATIONAL PROBLEMS ON CAD GEOMETRIES WITH APPLICATION TO ELECTRIC MACHINE SIMULATION

This work presents a deep learning-based framework for the solution of partial differential equations on complex computational domains described with computer-aided design tools. To account for the underlying distribution of the training data caused by spline-based projections from the reference to the physical domain, a variational neural solver equipped with an importance sampling scheme is developed, such that the loss function based on the discretized energy functional obtained after the weak formulation is modified according to the sample distribution. To tackle multi-patch domains possibly leading to solution discontinuities, the variational neural solver is additionally combined with a domain decomposition approach based on the Discontinuous Galerkin formulation. The proposed neural solver is verified on a toy problem and then applied to a real-world engineering test case, namely that of electric machine simulation. The numerical results show clearly that the neural solver produces physics-conforming solutions of significantly improved accuracy.

Comparison and Analysis of Neural Solver Methods for Differential Equations in Physical Systems

Differential equations are ubiquitous in many fields of study, yet not all equations, whether ordinary or partial, can be solved analytically. Traditional numerical methods such as time-stepping schemes have been devised to approximate these solutions. With the advent of modern deep learning, neural networks have become a viable alternative to traditional numerical methods. By reformulating the problem as an optimisation task, neural networks can be trained in a semi-supervised learning fashion to approximate nonlinear solutions. In this paper, neural solvers are implemented in TensorFlow for a variety of differential equations, namely: linear and nonlinear ordinary differential equations of the first and second order; Poisson’s equation, the heat equation, and the inviscid Burgers’ equation. Different methods, such as the naive and ansatz formulations, are contrasted, and their overall performance is analysed. Experimental data is also used to validate the neural solutions on test cases, specifically: the spring-mass system and Gauss’s law for electric fields. The errors of the neural solvers against exact solutions are investigated and found to surpass traditional schemes in certain cases. Although neural solvers will not replace the computational speed offered by traditional schemes in the near future, they remain a feasible, easy-to-implement substitute when all else fails.

MoCap-solver

In a conventional optical motion capture (MoCap) workflow, two processes are needed to turn captured raw marker sequences into correct skeletal animation sequences. Firstly, various tracking errors present in the markers must be fixed ( cleaning or refining ). Secondly, an agent skeletal mesh must be prepared for the actor/actress, and used to determine skeleton information from the markers ( re-targeting or solving ). The whole process, normally referred to as solving MoCap data, is extremely time-consuming, labor-intensive, and usually the most costly part of animation production. Hence, there is a great demand for automated tools in industry. In this work, we present MoCap-Solver, a production-ready neural solver for optical MoCap data. It can directly produce skeleton sequences and clean marker sequences from raw MoCap markers, without any tedious manual operations. To achieve this goal, our key idea is to make use of neural encoders concerning three key intrinsic components: the template skeleton, marker configuration and motion, and to learn to predict these latent vectors from imperfect marker sequences containing noise and errors. By decoding these components from latent vectors, sequences of clean markers and skeletons can be directly recovered. Moreover, we also provide a novel normalization strategy based on learning a pose-dependent marker reliability function, which greatly improves system robustness. Experimental results demonstrate that our algorithm consistently outperforms the state-of-the-art on both synthetic and real-world datasets.

SMART: A Situation Model for Algebra Story Problems via Attributed Grammar

Solving algebra story problems remains a challenging task in artificial intelligence, which requires a detailed understanding of real-world situations and a strong mathematical reasoning capability. Previous neural solvers of math word problems directly translate problem texts into equations, lacking an explicit interpretation of the situations, and often fail to handle more sophisticated situations. To address such limits of neural solvers, we introduce the concept of a situation model, which originates from psychology studies to represent the mental states of humans in problem-solving, and propose SMART, which adopts attributed grammar as the representation of situation models for algebra story problems. Specifically, we first train an information extraction module to extract nodes, attributes and relations from problem texts and then generate a parse graph based on a pre-defined attributed grammar. An iterative learning strategy is also proposed to further improve the performance of SMART. To study this task more rigorously, we carefully curate a new dataset named ASP6.6k. Experimental results on ASP6.6k show that the proposed model outperforms all previous neural solvers by a large margin, while preserving much better interpretability. To test these models' generalization capability, we also design an out-of-distribution (OOD) evaluation, in which problems are more complex than those in the training set. Our model exceeds state-of-the-art models by 17% in the OOD evaluation, demonstrating its superior generalization ability.

Learning by Fixing: Solving Math Word Problems with Weak Supervision

Previous neural solvers of math word problems (MWPs) are learned with full supervision and fail to generate diverse solutions. In this paper, we address this issue by introducing a weakly-supervised paradigm for learning MWPs. Our method only requires the annotations of the final answers and can generate various solutions for a single problem. To boost weakly-supervised learning, we propose a novel learning-by-fixing (LBF) framework, which corrects the misperceptions of the neural network via symbolic reasoning. Specifically, for an incorrect solution tree generated by the neural network, the fixing mechanism propagates the error from the root node to the leaf nodes and infers the most probable fix that can be executed to get the desired answer. To generate more diverse solutions, tree regularization is applied to guide the efficient shrinkage and exploration of the solution space, and a memory buffer is designed to track and save the discovered various fixes for each problem. Experimental results on the Math23K dataset show the proposed LBF framework significantly outperforms reinforcement learning baselines in weakly-supervised learning. Furthermore, it achieves comparable top-1 and much better top-3/5 answer accuracies than fully-supervised methods, demonstrating its strength in producing diverse solutions.

Neural networks for power flow: Graph neural solver

Recent trends in power systems and those envisioned for the next few decades push Transmission System Operators to develop probabilistic approaches to risk estimation. However, current methods to solve AC power flows are too slow to fully attain this objective. Thus we propose a novel artificial neural network architecture that achieves a more suitable balance between computational speed and accuracy in this context. Improving on our previous work on Graph Neural Solver for Power System [1], our architecture is based on Graph Neural Networks and allows for fast and parallel computations. It learns to perform a power flow computation by directly minimizing the violation of Kirchhoff’s law at each bus during training. Unlike previous approaches, our graph neural solver learns by itself and does not try to imitate the output of a Newton-Raphson solver. It is robust to variations of injections, power grid topology, and line characteristics. We experimentally demonstrate the viability of our approach on standard IEEE power grids (case9, case14, case30 and case118) both in terms of accuracy and computational time.

Robustness Analysis and Robotic Application of Combined Function Activated RNN for Time-Varying Matrix Pseudo Inversion

As a typical representative of the recurrent neural network (RNN), Zhang neural network (ZNN) has been proved as a powerful parallel-processing neural solver for time-varying matrix problems. Recent studies have shown that, in the absence of noise, the ZNN model activated by a combined activation function (CAF), which is the linear combination of sign-bi-power (SBP) and linear functions (termed CAF-ZNN), can achieve much better finite-time convergence compared with other ZNN models for time-varying matrix pseudoinversion. This paper investigates for the first time the influence of noises on such a CAF-ZNN model. We not only present the upper bound of steady-state solution error synthesized by the noise-polluted CAF-ZNN model for time-varying matrix pseudoinversion but also derive the finite convergence time for the solution error reaching the upper bound. Both the theoretical analyses and simulation observation reveal that the solution error can be made arbitrarily small by choosing some design parameters of CAF-ZNN. At last, a robotic application is successfully performed to further illustrate the feasibility of the CAF-ZNN model to kinematic control of redundant robot manipulator.