Efficient Gradient Computation Research Articles

TaNSR:Efficient 3D Reconstruction with Tetrahedral Difference and Feature Aggregation

AbstractNeural surface reconstruction methods have demonstrated their ability to recover 3D surfaces from multiple images. However, current approaches struggle to rapidly achieve high‐fidelity surface reconstructions. In this work, we propose TaNSR, which inherits the speed advantages of multi‐resolution hash encodings and extends its representation capabilities. To reduce training time, we propose an efficient numerical gradient computation method that significantly reduces additional memory access overhead. To further improve reconstruction quality and expedite training, we propose a feature aggregation strategy in volume rendering. Building on this, we introduce an adaptively weighted aggregation function to ensure the network can accurately reconstruct the surface of objects and recover more geometric details. Experiments on multiple datasets indicate that TaNSR significantly reduces training time while achieving better reconstruction accuracy compared to state‐of‐the‐art nerual implicit methods.

Learning Constitutive Relations From Soil Moisture Data via Physically Constrained Neural Networks

AbstractThe constitutive relations of the Richardson‐Richards equation encode the macroscopic properties of soil water retention and conductivity. These soil hydraulic functions are commonly represented by models with a handful of parameters. The limited degrees of freedom of such soil hydraulic models constrain our ability to extract soil hydraulic properties from soil moisture data via inverse modeling. We present a new free‐form approach to learning the constitutive relations using physically constrained neural networks. We implemented the inverse modeling framework in a differentiable modeling framework, JAX, to ensure scalability and extensibility. For efficient gradient computations, we implemented implicit differentiation through a nonlinear solver for the Richardson‐Richards equation. We tested the framework against synthetic noisy data and demonstrated its robustness against varying magnitudes of noise and degrees of freedom of the neural networks. We applied the framework to soil moisture data from an upward infiltration experiment and demonstrated that the neural network‐based approach was better fitted to the experimental data than a parametric model and that the framework can learn the constitutive relations.

Topology optimization for magnetic circuits with continuous adjoint method in 3D

PurposeThe purpose of this paper is to present a new approach to optimizing the design of 3D magnetic circuits. This approach is based on topology optimization, where derivative calculations are performed using the continuous adjoint method. Thus, the continuous adjoint method for magnetostatics has to be developed in 3D and has to be combined with penalization, filtering and homotopy approaches to provide an efficient optimization code.Design/methodology/approachTo provide this new topology optimization code, this study starts from 2D magnetostatic results to perform the sensitivity analysis, and this approach is extended to 3D. From this sensitivity analysis, the continuous adjoint method is derived to compute the gradient of an objective function of a 3D topological optimization design problem. From this result, this design problem is discretized and can then be solved by finite element software. Thus, by adding the solid isotropic material with penalization (SIMP) penalization approach and developing a homotopy-based optimization algorithm, an interesting means for designing 3D magnetic circuits is provided.FindingsIn this paper, the 3D continuous adjoint method for magnetostatic problems involving an objective least-squares function is presented. Based on 2D results, new theoretical results for developing sensitivity analysis in 3D taking into account different parameters including the ferromagnetic material, the current density and the magnetization are provided. Then, by discretizing, filtering and penalizing using SIMP approaches, a topology optimization code has been derived to address only the ferromagnetic material parameters. Based on this efficient gradient computation method, a homotopy-based optimization algorithm for solving large-scale 3D design problems is developed.Originality/valueIn this paper, an approach based on topology optimization to solve 3D magnetostatic design problems when an objective least-squares function is involved is proposed. This approach is based on the continuous adjoint method derived for 3D magnetostatic design problems. The effectiveness of this topology optimization code is demonstrated by solving the design of a 3D magnetic circuit with up to 100,000 design variables.

Differentiable and accelerated spherical harmonic and Wigner transforms

Many areas of science and engineering encounter data defined on spherical manifolds. Modelling and analysis of spherical data often necessitates spherical harmonic transforms, at high degrees, and increasingly requires efficient computation of gradients for machine learning or other differentiable programming tasks. We develop novel algorithmic structures for accelerated and differentiable computation of generalised Fourier transforms on the sphere S2 and rotation group SO(3), i.e. spherical harmonic and Wigner transforms, respectively. We present a recursive algorithm for the calculation of Wigner d-functions that is both stable to high harmonic degrees and extremely parallelisable. By tightly coupling this with separable spherical transforms, we obtain algorithms that exhibit an extremely parallelisable structure that is well-suited for the high throughput computing of modern hardware accelerators (e.g. GPUs). We also develop a hybrid automatic and manual differentiation approach so that gradients can be computed efficiently. Our algorithms are implemented within the JAX differentiable programming framework in the S2FFT▪ software code. Numerous samplings of the sphere are supported, including equiangular and HEALPix sampling. Computational errors are at the order of machine precision for spherical samplings that admit a sampling theorem. When benchmarked against alternative C codes we observe up to a 400-fold acceleration. Furthermore, when distributing over multiple GPUs we achieve very close to optimal linear scaling with increasing number of GPUs due to the highly parallelised and balanced nature of our algorithms. Provided access to sufficiently many GPUs our transforms thus exhibit an unprecedented effective linear time complexity.

Dynamic-mode-decomposition-based gradient prediction for adjoint-based aerodynamic shape optimization

Accurate and efficient gradient computation is the key to aerodynamic shape optimization. In this paper, dynamic mode decomposition (DMD) is employed to analyze the dynamic characteristics of the early pseudo-time marching of adjoint equations and to predict the gradient. Besides the first-order zero-frequency mode, other zero-frequency modes also contribute to the pseudo iterations of the adjoint equations in the early iterations. Hence, different from existing methods, all zero-frequency modes are retained to reconstruct adjoint fields for gradient prediction. Moreover, to further improve the modeling accuracy, an improved DMD (IDMD) is proposed by omitting the initial snapshots in early iterations. The effect of pseudo-time step on modeling accuracy is also studied. By solving the adjoint equations of the transonic and subsonic flows, the accuracy of the proposed method is verified. Results indicate that the proposed method still works despite the conventional solution process diverges. Through aerodynamic shape optimization examples of transonic flow over an airfoil, the number of adjoint pseudo-time steps is remarkably reduced by 83%, which indicates the proposed IDMD-based gradient prediction method has great potential for improving the efficiency of aerodynamic shape optimization.

On the use of neural networks for full waveform inversion

Neural networks have recently gained attention in the context of solving inverse problems. Physics-Informed Neural Networks (PINNs) are a prominent methodology for the task of solving both forward and inverse problems. In the paper at hand, full waveform inversion is the inverse problem under consideration. The performance of PINNs is compared against classical adjoint optimization. The comparison focuses on three key aspects: the forward-solver, the representation of the material, and the sensitivity computation for the gradient-based minimization. Starting from PINNs, each of these key aspects is investigated and adapted individually until the classical adjoint optimization emerges. It is shown that it is beneficial to use the neural network only for the discretization of the unknown spatially varying material field. Here the neural network produces reconstructions without oscillatory artifacts as typically encountered in classical full waveform inversion approaches. Due to this finding, a hybrid method is proposed. It exploits both the efficient gradient computation with the continuous adjoint method as well as the neural network ansatz for the unknown material field. This new hybrid method outperforms Physics-Informed Neural Networks and the classical adjoint optimization in two-dimensional and three-dimensional settings.

Timed hazard networks: Incorporating temporal difference for oncogenetic analysis.

Oncogenetic graphical models are crucial for understanding cancer progression by analyzing the accumulation of genetic events. These models are used to identify statistical dependencies and temporal order of genetic events, which helps design targeted therapies. However, existing algorithms do not account for temporal differences between samples in oncogenetic analysis. This paper introduces Timed Hazard Networks (TimedHN), a new statistical model that uses temporal differences to improve accuracy and reliability. TimedHN models the accumulation process as a continuous-time Markov chain and includes an efficient gradient computation algorithm for optimization. Our simulation experiments demonstrate that TimedHN outperforms current state-of-the-art graph reconstruction methods. We also compare TimedHN with existing methods on a luminal breast cancer dataset, highlighting its potential utility. The Matlab implementation and data are available at https://github.com/puar-playground/TimedHN.

Towards understanding and synthesis of contact-rich anthropomorphic motions through interactive cyber-physical human.

This article presents perspective on the research challenge of understanding and synthesizing anthropomorphic whole-body contact motions through a platform called "interactive cyber-physical human (iCPH)" for data collection and augmentation. The iCPH platform combines humanoid robots as "physical twins" of human and "digital twins" that simulates humans and robots in cyber-space. Several critical research topics are introduced to address this challenge by leveraging the advanced model-based analysis together with data-driven learning to exploit collected data from the integrated platform of iCPH. Definition of general description is identified as the first topic as a common basis of contact motions compatible to both humans and humanoids. Then, we set continual learning of a feasible contact motion network as the second challenge by benefiting from model-based approach and machine learning bridged by the efficient analytical gradient computation developed by the author and his collaborators. The final target is to establish a high-level symbolic system allowing automatic understanding and generation of contact motions in unexperienced environments. The proposed approaches are still under investigation, and the author expects that this article triggers discussions and further collaborations from different research communities, including robotics, artificial intelligence, neuroscience, and biomechanics.

A Model for Characteristic X-Ray Emission in Electron Probe Microanalysis Based on the (Filtered) Spherical Harmonic () Method for Electron Transport

Classical$k$-ratio models, for example, ZAF and$\phi ( \rho z)$, used in electron probe microanalysis (EPMA) assume a homogeneous or multilayered material structure, which essentially limits the spatial resolution of EPMA to the size of the interaction volume where characteristic X-rays are produced. We present a new model for characteristic X-ray emission that avoids assumptions on the material structure to not restrict the resolution of EPMAa priori. Our model bases on the spherical harmonic ($P_{\rm N}$) approximation of the Boltzmann equation for electron transport in continuous slowing down approximation.$P_{\rm N}$models have a simple structure, are hierarchical in accuracy and well-suited for efficient adjoint-based gradient computation, which makes our model a promising alternative to classical models in terms of improving the resolution of EPMA in the future. We present results of various test cases including a comparison of the$P_{\rm N}$model to a minimum entropy moment model as well as Monte-Carlo (MC) trajectory sampling, a comparison of$P_{\rm N}$-based$k$-ratios to$k$-ratios obtained with MC, a comparison with experimental data of electron backscattering yields as well as a comparison of$P_{\rm N}$and MC based on characteristic X-ray generation in a three-dimensional material probe with fine structures.

Alignment-free comparison of metagenomics sequences via approximate string matching.

Quantifying pairwise sequence similarities is a key step in metagenomics studies. Alignment-free methods provide a computationally efficient alternative to alignment-based methods for large-scale sequence analysis. Several neural network-based methods have recently been developed for this purpose. However, existing methods do not perform well on sequences of varying lengths and are sensitive to the presence of insertions and deletions. In this article, we describe the development of a new method, referred to as AsMac that addresses the aforementioned issues. We proposed a novel neural network structure for approximate string matching for the extraction of pertinent information from biological sequences and developed an efficient gradient computation algorithm for training the constructed neural network. We performed a large-scale benchmark study using real-world data that demonstrated the effectiveness and potential utility of the proposed method. The open-source software for the proposed method and trained neural-network models for some commonly used metagenomics marker genes were developed and are freely available at www.acsu.buffalo.edu/~yijunsun/lab/AsMac.html. Supplementary data are available at Bioinformatics online.

Third-Order Many-Body Expansion of OSV-MP2 Wave Function for Low-Order Scaling Analytical Gradient Computation.

We present a many-body expansion (MBE) formulation and implementation for efficient computation of analytical energy gradients from the orbital-specific-virtual second-order Møllet-Plesset perturbation theory (OSV-MP2) based on our earlier work (Zhou et al. J. Chem. Theory Comput. 2020, 16, 196-210). The third-order MBE(3) expansion of OSV-MP2 amplitudes and density matrices was developed to adopt the orbital-specific clustering and long-range termination schemes, which avoids term-by-term differentiations of the MBE energy bodies. We achieve better efficiency by exploiting the algorithmic sparsity that allows us to prune out insignificant fitting integrals and OSV relaxations. With these approximations, the present implementation is benchmarked on a range of molecules that show an economic scaling in the linear and quadratic regimes for computing MBE(3)-OSV-MP2 amplitude and gradient equations, respectively, and yields normal accuracy comparable to the original OSV-MP2 results. The MPI-3-based parallelism through shared memory one-sided communication is further developed for improving parallel scalability and memory accessibility by sorting the MBE(3) orbital clusters into independent tasks that are distributed on multiple processes across many nodes, supporting both global and local data locations in which selected MBE(3)-OSV-MP2 intermediates of different sizes are distinguished and accordingly placed. The accuracy and efficiency level of our MBE(3)-OSV-MP2 analytical gradient implementation is finally illustrated in two applications: we show that the subtle coordination structure differences of mechanically interlocked Cu-catenane complexes can be distinguished when tuning ligand lengths; and the porphycene molecular dynamics reveals the emergence of the vibrational signature arising from softened N-H stretching associated with hydrogen transfer, using an MP2 level of electron correlation and classical nuclei for the first time.

Efficient snapshot-free reverse time migration and computation of multiparameter gradients in full-waveform inversion

Computing images in reverse time migration and model parameter gradients from adjoint wavefields in full-waveform inversion (FWI) requires the correlation of a forward-propagated wavefield with another reverse-propagated wavefield. Although in theory only two wavefield propagations are required, one forward propagation and one reverse propagation, it requires storing the forward-propagated wavefield as a function of time to carry out the correlations, which is associated with significant input/output (I/O) cost. Alternatively, three wavefield propagations can be carried out to reverse propagate the forward-propagated wavefield in tandem with the reverse-propagated wavefield. We have determined how highly accurate reverse time migrated images and FWI model parameter gradients for anisotropic elastic FWI can be efficiently computed without significant disk I/O using two wavefield propagations by means of the principle of superposition.

Fast control parameterization optimal control with improved Polak–Ribière–Polyak conjugate gradient implementation for industrial dynamic processes

This paper proposes a fast control parameterization optimal control algorithm for industrial dynamic process with constraints. Derived from the frame of control variable parameterization (CVP) technique, the proposed method combines an efficient gradient computation strategy with an improved nonlinear optimization computation approach to overcome the challenge of computation efficiency caused by gradients and bounds in optimal control problems. Firstly, a fast gradient computation method based on the costate system of Hamiltonian function is developed to decrease the computational expense of gradients by employing approximate treatments and numerical integration strategy. Then, a trigonometric function transformation scheme is presented to tackle the boundary constraints so that the original optimal control problem is further converted into an unconstrained one. On this basis, an improved restricted Polak–Ribière–Polyak (PRP) conjugate gradient approach is introduced to solve the nonlinear optimization problem by using conjugate gradient iterations and strong Wolfe line search. Meanwhile, to enhance the convergence, a restricting condition is imposed in strong Wolfe line search to create iteration step-length. Finally, the proposed algorithm is implemented on three dynamic processes. The detailed comparison among the classical CVP method, literature results and the proposed method are carried out. Simulation studies show that the proposed fast approach averagely saves more than 90% computation time in contrast to the classical CVP method, demonstrating the effectiveness of the proposed fast optimal control approach.

A new scalable algorithm for computational optimal control under uncertainty

We address the design of optimal control strategies for high-dimensional stochastic dynamical systems. Such systems may be deterministic nonlinear systems evolving from random initial states, or systems driven by random parameters or random noise. The objective is to provide a validated new computational capability for optimal control which will be achieved more efficiently than current state-of-the-art methods. The new framework utilizes direct single or multi-shooting discretization, and is based on efficient vectorized gradient computation with adaptable memory management. The algorithm is demonstrated to be scalable to high-dimensional nonlinear control systems with random initial condition and unknown parameters. Numerical applications are presented and discussed for stochastic path planning problems involving models of unmanned aerial and ground vehicles, and for distributed control of a nonlinear advection-reaction-diffusion equation.

Elastodynamic full waveform inversion on GPUs with time-space tiling and wavefield reconstruction

Full waveform inversion (FWI) is a procedure used to determine the elastic parameters of the Earth by reducing the misfit between observed elastodynamic wavefields and their numerically modelled counterparts. The numerical solution of the elastodynamic wave equation is computationally expensive, and its performance is typically bandwidth bound. Computing the gradient of the FWI misfit functional adds further complexity as it involves computing the zero-lag cross-correlation of two wavefields propagating in opposite temporal directions. In this paper, we utilize graphics processing units (GPUs) for their high memory bandwidth and combine two principal optimizations in order to compute FWI gradients on large models and for long simulation times. Wavefield reconstruction methods allow efficient gradient computations with minimal memory requirements and interconnection transfers. Time-space tiling techniques permit us to transcend the limited amount of GPU memory while avoiding dramatic slowdowns due to the low interconnection bandwidth. The implementation considers a task-oriented, hybrid usage of explicitly managed and Unified Memory in order to satisfy the requirements. Benchmarks demonstrate that the proposed approach is able to preserve 78–90% of the original performance, when oversubscribing the amount of physical memory available on GPUs. Comparison with existing methods highlights the benefits of the method.

Inverse Design of Photonic Crystals through Automatic Differentiation

Gradient-based inverse design in photonics has already achieved remarkable results in designing small-footprint, high-performance optical devices. The adjoint variable method, which allows for the efficient computation of gradients, has played a major role in this success. However, gradient-based optimization has not yet been applied to the mode-expansion methods that are the most common approach to studying periodic optical structures like photonic crystals. This is because, in such simulations, the adjoint variable method cannot be defined as explicitly as in standard finite-difference or finite-element time- or frequency-domain methods. Here, we overcome this through the use of automatic differentiation, which is a generalization of the adjoint variable method to arbitrary computational graphs. We implement the plane-wave expansion and the guided-mode expansion methods using an automatic differentiation library, and show that the gradient of any simulation output can be computed efficiently and in parallel with respect to all input parameters. We then use this implementation to optimize the dispersion of a photonic crystal waveguide, and the quality factor of an ultra-small cavity in a lithium niobate slab. This extends photonic inverse design to a whole new class of simulations, and more broadly highlights the importance that automatic differentiation could play in the future for tracking and optimizing complicated physical models.

A Fast Algorithm for Maximum Likelihood Estimation of Mixture Proportions Using Sequential Quadratic Programming

Maximum likelihood estimation of mixture proportions has a long history, and continues to play an important role in modern statistics, including in development of nonparametric empirical Bayes methods. Maximum likelihood of mixture proportions has traditionally been solved using the expectation maximization (EM) algorithm, but recent work by Koenker and Mizera shows that modern convex optimization techniques—in particular, interior point methods—are substantially faster and more accurate than EM. Here, we develop a new solution based on sequential quadratic programming (SQP). It is substantially faster than the interior point method, and just as accurate. Our approach combines several ideas: first, it solves a reformulation of the original problem; second, it uses an SQP approach to make the best use of the expensive gradient and Hessian computations; third, the SQP iterations are implemented using an active set method to exploit the sparse nature of the quadratic subproblems; fourth, it uses accurate low-rank approximations for more efficient gradient and Hessian computations. We illustrate the benefits of the SQP approach in experiments on synthetic datasets and a large genetic association dataset. In large datasets ( observations, mixture components), our implementation achieves at least 100-fold reduction in runtime compared with a state-of-the-art interior point solver. Our methods are implemented in Julia and in an R package available on CRAN (https://CRAN.R-project.org/package=mixsqp). Supplementary materials for this article are available online.

Multilayer Topology Optimization of Wideband SIW-to-Waveguide Transitions

This article utilizes a topology optimization approach to design planar multilayer transitions between substrate integrated waveguides (SIWs) and rectangular waveguides (RWGs). The optimization problem is formulated based on the modal field analyses and Maxwell’s equations in the time domain solved by the finite-difference time-domain (FDTD) method. We present a time-domain boundary condition based on the Klein–Gordon equation to split traveling waves at homogeneous waveguide ports. We employ the boundary condition to compute portal quantities and to devise an adjoint-field system that enabled an efficient computation of the objective function gradient. We solve design problems that include more than 105 000 design variables by using less than 400 solutions of Maxwell’s equations. Moreover, a new formulation that effectively combats the development of in-band resonances in the design is presented. The transition configuration allows the direct mount of conventional RWG sections on the circuit board and aims to cover the entire K-band. The guiding structure of the optimized transition requires blind vias, which is realized by a simple and cost-efficient technique. In addition, the transition is optimized for three different setups that can be used to provide different field polarizations. The proposed transitions show less than 1-dB insertion loss and around 15-dB return loss over the frequency interval 18–28 GHz. Several prototypes are fabricated with an excellent match between the simulation and measurement results.

Gradient based biobjective shape optimization to improve reliability and cost of ceramic components

We consider the simultaneous optimization of the reliability and the cost of a ceramic component in a biobjective PDE constrained shape optimization problem. A probabilistic Weibull-type model is used to assess the probability of failure of the component under tensile load, while the cost is assumed to be proportional to the volume of the component. Two different gradient-based optimization methods are suggested and compared at 2D test cases. The numerical implementation is based on a first discretize then optimize strategy and benefits from efficient gradient computations using adjoint equations. The resulting approximations of the Pareto front nicely exhibit the trade-off between reliability and cost and give rise to innovative shapes that compromise between these conflicting objectives.

Exploiting GPUs for Efficient Gradient Boosting Decision Tree Training

In this paper, we present a novel parallel implementation for training Gradient Boosting Decision Trees (GBDTs) on Graphics Processing Units (GPUs). Thanks to the excellent results on classification/regression and the open sourced libraries such as XGBoost, GBDTs have become very popular in recent years and won many awards in machine learning and data mining competitions. Although GPUs have demonstrated their success in accelerating many machine learning applications, it is challenging to develop an efficient GPU-based GBDT algorithm. The key challenges include irregular memory accesses, many sorting operations with small inputs and varying data parallel granularities in tree construction. To tackle these challenges on GPUs, we propose various novel techniques including (i) Run-length Encoding compression and thread/block workload dynamic allocation, (ii) data partitioning based on stable sort, and fast and memory efficient attribute ID lookup in node splitting, (iii) finding approximate split points using two-stage histogram building, (iv) building histograms with the aware of sparsity and exploiting histogram subtraction to reduce histogram building workload, (v) reusing intermediate training results for efficient gradient computation, and (vi) exploiting multiple GPUs to handle larger data sets efficiently. Our experimental results show that our algorithm named ThunderGBM can be 10x times faster than the state-of-the-art libraries (i.e., XGBoost, LightGBM and CatBoost) running on a relatively high-end workstation of 20 CPU cores. In comparison with the libraries on GPUs, ThunderGBM can handle higher dimensional problems which the libraries become extremely slow or simply fail. For the data sets the existing libraries on GPUs can handle, ThunderGBM achieves up to 10 times speedup on the same hardware, which demonstrates the significance of our GPU optimizations. Moreover, the models trained by ThunderGBM are identical to those trained by XGBoost, and have similar quality as those trained by LightGBM and CatBoost.