Numerical Gradient Research Articles

Density Matrix Emulation of Quantum Recurrent Neural Networks for Multivariate Time Series Prediction

Abstract Quantum Recurrent Neural Networks (QRNNs) are robust candidates for modelling and predicting future values in multivariate time series. However, the effective implementation of some QRNN models is limited by the need for mid-circuit measurements. Those increase the requirements for quantum hardware, which in the current NISQ era does not allow reliable computations. Emulation arises as the main near-term alternative to explore the potential of QRNNs, but existing quantum emulators are not dedicated to circuits with multiple intermediate measurements. In this context, we design a specific emulation method that relies on density matrix formalism. Using a compact tensor notation, we provide the mathematical formulation of the operator-sum representation involved. This allows us to show how the present and past information from a time series is transmitted through the circuit, and how to reduce the computational cost in every time step of the emulated network. In addition, we derive the analytical gradient and the Hessian of the network outputs with respect to its trainable parameters, which are needed when the outputs have stochastic noise due to hardware errors and a finite number of circuit shots (sampling). We finally test the presented methods using a hardware-efficient ansatz and four diverse datasets that include univariate and multivariate time series, with and without sampling noise. In addition, we compare the model with other existing quantum and classical approaches. Our results show how QRNNs can be trained with numerical and analytical gradients to make accurate predictions of future values by capturing non-trivial patterns of input series with different complexities.

TaNSR:Efficient 3D Reconstruction with Tetrahedral Difference and Feature Aggregation

AbstractNeural surface reconstruction methods have demonstrated their ability to recover 3D surfaces from multiple images. However, current approaches struggle to rapidly achieve high‐fidelity surface reconstructions. In this work, we propose TaNSR, which inherits the speed advantages of multi‐resolution hash encodings and extends its representation capabilities. To reduce training time, we propose an efficient numerical gradient computation method that significantly reduces additional memory access overhead. To further improve reconstruction quality and expedite training, we propose a feature aggregation strategy in volume rendering. Building on this, we introduce an adaptively weighted aggregation function to ensure the network can accurately reconstruct the surface of objects and recover more geometric details. Experiments on multiple datasets indicate that TaNSR significantly reduces training time while achieving better reconstruction accuracy compared to state‐of‐the‐art nerual implicit methods.

Beyond Time-Homogeneity for Continuous-Time Multistate Markov Models

–Multistate Markov models are a canonical parametric approach for data modeling of observed or latent stochastic processes supported on a finite state space. Continuous-time Markov processes describe data that are observed irregularly over time, as is often the case in longitudinal medical data, for example. Assuming that a continuous-time Markov process is time-homogeneous, a closed-form likelihood function can be derived from the Kolmogorov forward equations—a system of differential equations with a well-known matrix-exponential solution. Unfortunately, however, the forward equations do not admit an analytical solution for continuous-time, time- inhomogeneous Markov processes, and so researchers and practitioners often make the simplifying assumption that the process is piecewise time-homogeneous. In this article, we provide intuitions and illustrations of the potential biases for parameter estimation that may ensue in the more realistic scenario that the piecewise-homogeneous assumption is violated, and we advocate for a solution for likelihood computation in a truly time-inhomogeneous fashion. Particular focus is afforded to the context of multistate Markov models that allow for state label misclassifications, which applies more broadly to hidden Markov models (HMMs), and Bayesian computations bypass the necessity for computationally demanding numerical gradient approximations for obtaining maximum likelihood estimates (MLEs). Supplemental materials are available online.

Fundamental Invariant Neural Network (FI-NN) Potential Energy Surface for the OH + CH3OH Reaction with Analytical Forces.

The hydrogen abstraction reaction of OH + CH3OH plays a great role in combustion and atmospheric and interstellar chemistry and has been extensively studied theoretically and experimentally. Theoretically, the numerical gradients with respect to the Cartesian coordinates of atoms in molecular simulations on our recent potential energy surface (PES) for the title reaction trained using the permutationally invariant polynomial neural network (PIP-NN) approach hinder the extensive calculation because of the unaffordable computation cost. To address this issue, we in this work report a new full-dimensional accurate analytical PES for the title reaction using the fundamental invariant neural network (FI-NN) approach based on 140,192 points of the quality UCCSD(T)-F12a/AVTZ. Besides, the spin-orbit (SO) corrections of OH in the entrance channel were determined at the level of complete active space self-consistent field with the AVTZ basis set. As a compromise between computational cost and efficiency, the Δ-machine learning approach was employed to construct the SO-corrected PES. Based on this new FI-NN PES with analytical forces, thermal rate coefficients and various dynamic properties, including the integral cross sections, the differential cross sections, and the product energy partitioning, were determined by running a total of 5.5 million trajectories. The use of analytical gradients of the FI-NN PES accelerated simulations and about 99% of computation cost was saved, compared to that for the PIP-NN PES with numerical gradients. Such a significant acceleration is achieved mainly by replacing PIPs with FIs.

Brain functional gradient and structure features in adolescent and adult autism spectrum disorders.

Understanding how function and structure are organized and their coupling with clinical traits in individuals with autism spectrum disorder (ASD) is a primary goal in network neuroscience research for ASD. Atypical brain functional networks and structures in individuals with ASD have been reported, but whether these associations show heterogeneous hierarchy modeling in adolescents and adults with ASD remains to be clarified. In this study, 176 adolescent and 74 adult participants with ASD without medication or comorbidities and sex, age matched healthy controls (HCs) from 19 research groups from the openly shared Autism Brain Imaging Data Exchange II database were included. To investigate the relationship between the functional gradient, structural changes, and clinical symptoms of brain networks in adolescents and adults with ASD, functional gradient and voxel-based morphometry (VBM) analyses based on 1000 parcels defined by Schaefer mapped to Yeo's seven-network atlas were performed. Pearson's correlation was calculated between the gradient scores, gray volume and density, and clinical traits. The subsystem-level analysis showed that the second gradient scores of the default mode networks and frontoparietal network in patients with ASD were relatively compressed compared to adolescent HCs. Adult patients with ASD showed an overall compression gradient of 1 in the ventral attention networks. In addition, the gray density and volumes of the subnetworks showed no significant differences between the ASD and HC groups at the adolescent stage. However, adults with ASD showed decreased gray density in the limbic network. Moreover, numerous functional gradient parameters, but not VBM parameters, in adolescents with ASD were considerably correlated with clinical traits in contrast to those in adults with ASD. Our findings proved that the atypical changes in adolescent ASD mainly involve the brain functional network, while in adult ASD, the changes are more related to brain structure, including gray density and volume. These changes in functional gradients or structures are markedly correlated with clinical traits in patients with ASD. Our study provides a novel understanding of the pathophysiology of the structure-function hierarchy in ASD.

Optimization of Quantum Nuclei Positions with the Adaptive Nuclear-Electronic Orbital Approach.

The use of multicomponent methods has become increasingly popular over the last years. Under this framework, nuclei (commonly protons) are treated quantum mechanically on the same footing as the electronic structure problem. Under the use of atomic-centered orbitals, this can lead to some complications as the ideal location of the nuclear basis centers must be optimized. In this contribution, we propose a straightforward approach to determine the position of such centers within the self-consistent cycle of a multicomponent calculation, making use of individual proton charge centroids. We test the method on model systems including the water dimer, a protonated water tetramer, and a porphine system. Comparing to numerical gradient calculations, the adaptive nuclear-electronic orbital (NEO) procedure is able to converge the basis centers to within a few cents of an Ångström and with less than 0.1 kcal/mol differences in absolute energies. This is achieved in one single calculation and with a small added computational effort of up to 80% compared to a regular NEO- self-consistent field run. An example application for the human transketolase proton wire is also provided.

Entropy-Weighted Numerical Gradient Optimization Spiking Neural System for Biped Robot Control.

The optimization of robot controller parameters is a crucial task for enhancing robot performance, yet it often presents challenges due to the complexity of multi-objective, multi-dimensional multi-parameter optimization. This paper introduces a novel approach aimed at efficiently optimizing robot controller parameters to enhance its motion performance. While spiking neural P systems have shown great potential in addressing optimization problems, there has been limited research and validation concerning their application in continuous numerical, multi-objective, and multi-dimensional multi-parameter contexts. To address this research gap, our paper proposes the Entropy-Weighted Numerical Gradient Optimization Spiking Neural P System, which combines the strengths of entropy weighting and spiking neural P systems. First, the introduction of entropy weighting eliminates the subjectivity of weight selection, enhancing the objectivity and reproducibility of the optimization process. Second, our approach employs parallel gradient descent to achieve efficient multi-dimensional multi-parameter optimization searches. In conclusion, validation results on a biped robot simulation model show that our method markedly enhances walking performance compared to traditional approaches and other optimization algorithms. We achieved a velocity mean absolute error at least 35% lower than other methods, with a displacement error two orders of magnitude smaller. This research provides an effective new avenue for performance optimization in the field of robotics.

Neural Physical Simulation with Multi-Resolution Hash Grid Encoding

We explore the generalization of the implicit representation in the physical simulation task. Traditional time-dependent partial differential equations (PDEs) solvers for physical simulation often adopt the grid or mesh for spatial discretization, which is memory-consuming for high resolution and lack of adaptivity. Many implicit representations like local extreme machine or Siren are proposed but they are still too compact to suffer from limited accuracy in handling local details and a long time of convergence. We contribute a neural simulation framework based on multi-resolution hash grid representation to introduce hierarchical consideration of global and local information, simultaneously. Furthermore, we propose two key strategies: 1) a numerical gradient method for computing high-order derivatives with boundary conditions; 2) a range analysis sample method for fast neural geometry boundary sampling with dynamic topologies. Our method shows much higher accuracy and strong flexibility for various simulation problems: e.g., large elastic deformations, complex fluid dynamics, and multi-scale phenomena which remain challenging for existing neural physical solvers.

Optimizing Retention Bunkers in Copper Mines with Numerical Methods and Gradient Descent

This study examines the optimization of ore receiving bins in underground copper mines, targeting the reduction of rapid wear and tear on bin components. The investigation identifies the primary wear contributors as the force exerted by the accumulated ore and the velocity at which ore particles move. By altering design and operational parameters, the objective is to decrease wear at key points such as transfer areas, thereby improving the efficiency and service life of retention bunkers. A Discrete Element Method (DEM) model of the bin was created and validated against actual mining conditions to study the impact of material flow on wear. The optimization approach used a constrained gradient descent algorithm to minimize factors like particle velocity and pressure force, while maintaining the efficiency of the bin. The findings provide valuable insights for the future design enhancements, potentially improving the operational performance of retention bunkers in the mining industry.

Physics‐Aware Analytic‐Gradient Training of Photonic Neural Networks

AbstractPhotonic neural networks (PNNs) have emerged as promising alternatives to traditional electronic neural networks. However, the training of PNNs, especially the chip implementation of analytic gradient descent algorithms that are recognized as highly efficient in traditional practice, remains a major challenge because physical systems are not differentiable. Although training methods such as gradient‐free and numerical gradient methods are proposed, they suffer from excessive measurements and limited scalability. State‐of‐the‐art in situ training method is also cost‐challenged, requiring expensive in‐line monitors and frequent optical I/O switching. Here, a physics‐aware analytic‐gradient training (PAGT) method is proposed that calculates the analytic gradient in a divide‐and‐conquer strategy, overcoming the difficulty induced by chip non‐differentiability in the training of PNNs. Multiple training cases, especially a generative adversarial network, are implemented on‐chip, achieving a significant reduction in time consumption (from 31 h to 62 min) and a fourfold reduction in energy consumption, compared to the in situ method. The results provide low‐cost, practical, and accelerated solutions for training hybrid photonic‐digital electronic neural networks.

Vertical Parking Trajectory Planning With the Combination of Numerical Optimization Method and Gradient Lifting Decision Tree

Vertical Parking Trajectory Planning With the Combination of Numerical Optimization Method and Gradient Lifting Decision Tree

Enhancing virtual flow metering on offshore oil platforms through parallel computing and data reconciliation

Technological advances in recent years have facilitated data accessibility and increased computational resources; consequently, more remarkable numerical performances in data analysis and computational tests have become possible. In this context, parallel computing techniques are considered relevant tools in applications necessitating reduced computational time, such as online and real-time monitoring of industrial processes. For these reasons, the present work develops and implements online and real-time monitoring of offshore oil and gas production, using parallel computing to improve the performance of the data reconciliation methodology implemented, which will estimate the virtual flow metering with more reliability and in real time. To achieve this, three optimization methods (DA, L-BFGS-B, and SLSQP) and two approaches of parallelization schemes were considered: (1) parallelization of the simulations of each production well; (2) parallelization of the numerical gradient, performed by the deterministic optimization method. Parallel implementations allowed the reduction of up to 80% (type 1) and up to 89% (type 2) of the simulation time compared to their serial counterparts. Hence, the parallel execution of type 2 exhibited more promising results, while type 1 demonstrated greater adaptability to optimization methods. Finally, the results indicate that parallel computing techniques facilitate data reconciliation in real-time, enabling online monitoring in a much shorter time and more accurately than achieved with similar sequential procedures. Moreover, the implemented methodology has proven effective for virtual flow metering of each well’s flow rates of oil, water, and gas. In this context, parallelism techniques present the potential for developing the global monitoring of the virtual flow meters for real-time oil, water, and gas flow rates in each well.

A general strategy for improving the performance of PINNs -- Analytical gradients and advanced optimizers in the NeuralSchrödinger framework

In this work, the previously introduced NeuralSchrödinger PINN is extended towards the use of analytical gradient expressions of the loss function. It is shown that the analytical gradients derived in this work increase the convergence properties for both the BFGS and ADAM optimizers compared to the previously employed numerical gradient implementation. In addition, the use of parallelised GPU computations via CUDA greatly increased the computational performance over the previous implementation using single-core CPU computations. As a consequence, an extension of the NeuralSchrödinger PINN towards two-dimensional quantum systems became feasible as also demonstrated in this work.

Hierarchical Fe-Co@TiO2 with Incoherent Heterointerfaces and Gradient Magnetic Domains for Electromagnetic Wave Absorption.

Induced polarization response and integrated magnetic resonance show prosperous advantages in boosting electromagnetic wave absorption but still face huge challenges in revealing the intrinsic mechanism. In this work, we propose a self-confined strategy to construct hierarchical Fe-Co@TiO2 microrods with numerous incoherent heterointerfaces and gradient magnetic domains. The results demonstrate that the use of polyvinylpyrrolidone (PVP) coating is crucial for the subsequent deposition of Co-zeolitic imidazolate frameworks (ZIF-67), the distance of ordered arranged metal ions manipulates the size of magnetic domains, and the pyrolysis of PVP layers restricts the eutectic process of Fe-Co alloys to some extent. As a result, these introduced lattice defects, oxygen vacancies, and incoherent heterointerfaces inevitably generate a strong polarization response, and the regulated gradient magnetic domains realize integrated magnetic resonance, including macroscopic magnetic coupling, long-range magnetic diffraction, and nanoscale magnetic bridge connection, and both of the intrinsic mechanisms in dissipating electromagnetic energy are quantitatively clarified by Lorentz off-axis electron holography. Owing to the cooperative merits, the Fe-Co@TiO2 absorbents exhibit enhanced absorption intensity and strong absorption bandwidth. This study inspires us to develop a generalized strategy for manipulating the size of magnetic domains, and the integrated magnetic resonance theory provides a versatile methodology in clarifying magnetic loss mechanism.

A superconvergent CDG finite element for the Poisson equation on polytopal meshes

AbstractA conforming discontinuous Galerkin (CDG) finite element is constructed for solving second order elliptic equations on polygonal and polyhedral meshes. The numerical trace on the edge between two elements is no longer the average of two discontinuous Pk functions on the two sides, but a lifted function from four Pk functions. When the numerical gradient space is the subspace of piecewise polynomials on subtriangles/subtehrahedra of a polygon/polyhedron which have a one‐piece polynomial divergence on , this CDG method has a superconvergence of order two above the optimal order. Due to the superconvergence, we define a post‐process which lifts a Pk CDG solution to a quasi‐optimal solution on each element. Numerical examples in 2D and 3D are computed and the results confirm the theory.

Comparative analysis and computational optimization of potential-based multiphase lattice Boltzmann models

The potential-based multiphase lattice Boltzmann models are widely used because they root in thermodynamics and evade the interface tracking or integrating. This paper investigates several potential-based models with the common equations of state (EOS) by the theoretical analyses and numerical computations of the thermodynamic consistency and spurious currents. Surprisingly, the Shan–Chen model presents a superior accuracy compared to the Zhang–Chen models, although they are mathematically equivalent. We find that the great improvement is attributed to the square root form of the pseudopotential model, which significantly lessens the error of numerical gradient calculation. Inspired by the improvement, a general formula φ′=n−1φ1−n∂x(φn) is introduced for calculating the gradient, and the coefficient n=0.1 yields better results than n=0.5, which is equal to the pseudopotential model. This scheme is further applied to optimize the evaluation of the chemical potential model. The improved chemical potential model displays lower numerical errors in the liquid–gas transition region and smaller spurious currents near the curved phase interface than the pseudopotential model. Additionally, the improved model is confirmed to meet the Young–Laplace law and Galilean invariance.

Development of a New Numerical Conjugate Gradient Technique for Image Processing

We present a new iterative conjugate gradient technique for image processing. The technique is based on a new derivation of the conjugacy coefficient and develops a variant of the classical Fletcher-Reeves conjugate gradient method. The derivation exploits a quadratic function model. The new method is intended to minimize the presence of noise by utilizing the adaptive median filter (AMF) to reduce salt-and-pepper noise, while the adaptive center-weighted median filter (ACWMF) is used to reduce random-valued noise. The theoretical convergence properties of the method are proven and then tested on a basic set of images using MATLAB. The results show that the proposed algorithm is more efficient than the classical Fletcher-Reeves (FR) method, as measured by the signal-to-noise ratio (PSNR). The number of iterations and the number of function evaluations are also lower for the proposed method. The favorable performance of the new algorithm provides promise for deriving similar techniques that enhance the speed and efficiency of image-processing libraries.

Compressive Reconstruction Based on Sparse Autoencoder Network Prior for Single-Pixel Imaging

The combination of single-pixel imaging and single photon-counting technology enables ultra-high-sensitivity photon-counting imaging. In order to shorten the reconstruction time of single-photon counting, the algorithm of compressed sensing is used to reconstruct the underdetermined image. Compressed sensing theory based on prior constraints provides a solution that can achieve stable and high-quality reconstruction, while the prior information generated by the network may overfit the feature extraction and increase the burden of the system. In this paper, we propose a novel sparse autoencoder network prior for the reconstruction of the single-pixel imaging, and we also propose the idea of multi-channel prior, using the fully connected layer to construct the sparse autoencoder network. Then, take the network training results as prior information and use the numerical gradient descent method to solve underdetermined linear equations. The experimental results indicate that this sparse autoencoder network prior for the single-photon counting compressed images reconstruction has the ability to outperform the traditional one-norm prior, effectively improving the reconstruction quality.

Hybrid Polymer-Alloy-Fluoride Interphase Enabling Fast Ion Transport Kinetics for Low-Temperature Lithium Metal Batteries.

Low-temperature lithium metal batteries are of vital importance for cold-climate condition applications. Their realization, however, is plagued by the extremely sluggish Li+ transport kinetics in the vicinity of Li metal anode at low temperatures. Different from the widely adopted electrolyte engineering, a functional interphase design concept is proposed in this work to efficiently improve the low-temperature electrochemical reaction kinetics of Li metal anodes. As a proof of concept, we design a hybrid polymer-alloy-fluoride (PAF) interphase featuring numerous gradient fluorinated solid-solution alloy composite nanoparticles embedded in a polymerized dioxolane matrix. Systematic experimental and theoretical investigations demonstrate that the hybrid PAF interphase not only exhibits superior lithiophilicity but also provides abundant ionic conductive pathways for homogeneous and fast Li+ transport at the Li-electrolyte interface. With enhanced interfacial dynamics of Li-ion migration, the as-designed PAF-Li anode works stably for 720 h with low voltage hysteresis and dendrite-free electrode morphology in symmetric cell configurations at -40 °C. The full cells with PAF-Li anode display a commercial-grade capacity of 4.26 mAh cm-2 and high capacity retention of 74.7% after 150 cycles at -20 °C. The rational functional interphase design for accelerating ion-transfer kinetics sheds innovative insights for developing high-areal-capacity and long-lifespan lithium metal batteries at low temperatures.

A temporally and spatially local spike-based backpropagation algorithm to enable training in hardware

Spiking neural networks (SNNs) have emerged as a hardware efficient architecture for classification tasks. The challenge of spike-based encoding has been the lack of a universal training mechanism performed entirely using spikes. There have been several attempts to adopt the powerful backpropagation (BP) technique used in non-spiking artificial neural networks (ANNs): (1) SNNs can be trained by externally computed numerical gradients. (2) A major advancement towards native spike-based learning has been the use of approximate BP using spike-time dependent plasticity with phased forward/backward passes. However, the transfer of information between such phases for gradient and weight update calculation necessitates external memory and computational access. This is a challenge for standard neuromorphic hardware implementations. In this paper, we propose a stochastic SNN based back-prop (SSNN-BP) algorithm that utilizes a composite neuron to simultaneously compute the forward pass activations and backward pass gradients explicitly with spikes. Although signed gradient values are a challenge for spike-based representation, we tackle this by splitting the gradient signal into positive and negative streams. The composite neuron encodes information in the form of stochastic spike-trains and converts BP weight updates into temporally and spatially local spike coincidence updates compatible with hardware-friendly resistive processing units. Furthermore, we characterize the quantization effect of discrete spike-based weight update to show that our method approaches BP ANN baseline with sufficiently long spike-trains. Finally, we show that the well-performing softmax cross-entropy loss function can be implemented through inhibitory lateral connections enforcing a winner take all rule. Our SNN with a two-layer network shows excellent generalization through comparable performance to ANNs with equivalent architecture and regularization parameters on static image datasets like MNIST, Fashion-MNIST, Extended MNIST, and temporally encoded image datasets like Neuromorphic MNIST datasets. Thus, SSNN-BP enables BP compatible with purely spike-based neuromorphic hardware.