Hessian Matrix Research Articles

Unified finite-volume physics informed neural networks to solve the heterogeneous partial differential equations

The emerging physics informed neural network (PINN) has been recently applied to a wide range of mathematical problems. It is promising to precisely solve the partial differential equations (PDEs) in a fast, flexible manner. Whereas, PINN struggles with poor accuracy and costly computation in case of heterogeneous PDE coefficients. To mitigate these issues, a new PINN, which is known as the unified finite volume PINN (UFV-PINN), is proposed to unify the sub-domain decomposition, finite volume discretization, and conventional numerical solvers. The output by neural network (NN) over the boundaries of agglomerated sub-domains functions as boundary conditions (BCs) for UFV-PINN training. In this connection, the customized differentiable conventional numerical solver further solves the PDEs. The discrepancy between NN prediction and the conventional numerical solution within the sub-domains is taken as the novel training loss, enforcing the conservation law of PDE. For illustration, the Poisson and advection–diffusion equations (ADE) are solved, which are classical but still challenging to PINN in the presence of heterogeneity. Numerical experiments are conducted to compare the performance of the proposed UFV-PINN and the standard PINN, both in terms of accuracy and efficiency. Results indicate that UFV-PINN attains remarkable accuracy improvement with less computation time. Hessian spectrum analysis indicates that the loss Hessian matrix of UFV-PINN is more inclined to be positive definite, highlighting its superior performance over the standard PINN. It is the first time that the FV numerical solver is seamlessly embedded into the PINN training for performance improvement.

Boosting sharpness-aware training with dynamic neighborhood

Learning algorithms motivated by minimizing the sharpness of loss surface is a hot research topic in improving generalization. The existing methods usually solve a constrained min–max problem to minimize sharpness and find flat minima. However, most constraints (i.e., the neighborhood of the sharpness) are inappropriate, leading to sub-optimal results. This paper theoretically explores the optimal neighborhood from the view of Probably Approximately Correct-Bayesian (PAC-Bayesian) framework. A closed form of the optimal neighborhood is provided. This neighborhood is determined by the Hessian matrix and the scales of parameters. Then a generalization bound is derived that serves as a guiding principle in the design of the sharpness minimization algorithm. The Dynamic neighborhood-based Sharpness-Aware Minimization algorithm is proposed, which can adaptively adjust the neighborhood during the training process to gain better performance. Also, the algorithm is proved can convergent at the rate O(logT/T). Experimental results demonstrate that the proposed algorithm outperforms the other methods (e.g., accuracy ＋2.86% over baseline on CIFAR-100 for VGG-16).

Strain field reconstruction from helical-winding fiber distributed acoustic sensing and its application in anisotropic elastic reverse time migration

Optical fiber-based distributed acoustic sensing (DAS) technology has been a popular seismic acquisition tool due to its easy deployment, wide bandwidth, and dense sampling. However, the sensitivity of straight optical fiber to only single-axis strain presents challenges in fully characterizing multicomponent seismic wavefields, making it difficult to use these data in elastic reverse time migration (ERTM). The helical-winding fiber receives projecting signals projected onto the fiber from all seismic strain field components and has the potential to reconstruct those strain components for ERTM imaging. Here, we give detailed mathematical principles of helical fiber-based DAS with crucial parameters such as pitch angle, gauge length, and rotating angle. At least six points of DAS responses are required in one or several winding periods to rebuild the strain fields within the seismic wavelength. The projecting matrix of conventional regular helical-winding fiber is singular and ill conditioned, which results in computation challenges for the inverse of the Hessian matrix for strain component reconstruction. To tackle this problem, we develop a nonregular variant pitch-angle winding configuration for helical fiber. Our winding design is validated using the rank and condition number of the projecting matrix, which is proven to be an important tool in the reconstruction of the original seismic strains. The recovered strain components from the DAS response are then used to backward propagate the receiver wavefields in ERTM with an efficient P/S decoupled approach. To summarize, we develop a novel winding design of helical fiber to recover the strain fields and then develop an efficient 3D anisotropic P/S wave-mode decomposition method for generating vector P and S wavefields during their propagation. Both methods are applied to build an anisotropic DAS-ERTM workflow for producing PP and PS images. Two synthetic examples demonstrate the effectiveness of our approach.

The modified fractional‐order quasi‐reversibility method for a class of direct and inverse problems governed by time‐fractional heat equations with involution perturbation

This study aims to explore two classes of ill‐posed problems governed by a nonclassical fractional heat equation with an involution perturbation. To achieve a stable solution, we introduce a modified pseudo‐parabolic regularization method that involves a correction term through a mixed fractional derivation operator, resulting in a sequence of well‐posed problems that depend on a regularization parameter . We rigorously demonstrate that the resulting approximate problems are well‐posed and present convergence results for this regularization approach.

Differential characterization of quadratic surfaces

AbstractLet $$f\in W^{3,1}_{\textrm{loc}}(\Omega )$$ f ∈ W loc 3 , 1 ( Ω ) be a function defined on a connected open subset $$\Omega \subseteq {\mathbb {R}}^2$$ Ω ⊆ R 2 . We will show that its graph is contained in a quadratic surface if and only if f is a weak solution to a certain system of $${3}{\hbox {rd}}$$ 3 rd order partial differential equations unless the Hessian determinant of f is non-positive on the whole $$\Omega $$ Ω . Moreover, we will prove that the system is in some sense the simplest possible in a wide class of differential equations, which will lead to the classification of all polynomial partial differential equations satisfied by parametrizations of generic quadratic surfaces. Although we will mainly use the tools of linear and commutative algebra, the theorem itself is also somehow related to holomorphic functions.

Sparse subnetwork inference for neural network epistemic uncertainty estimation with improved Hessian approximation

Despite significant advances in deep neural networks across diverse domains, challenges persist in safety-critical contexts, including domain shift sensitivity and unreliable uncertainty estimation. To address these issues, this study investigates Bayesian learning for uncertainty handling in modern neural networks. However, the high-dimensional, non-convex nature of the posterior distribution poses practical limitations for epistemic uncertainty estimation. The Laplace approximation, as a cost-efficient Bayesian method, offers a practical solution by approximating the posterior as a multivariate normal distribution but faces computational bottlenecks in precise covariance matrix computation and storage. This research employs subnetwork inference, utilizing only a subset of the parameter space for Bayesian inference. In addition, a Kronecker-factored and low-rank representation is explored to reduce space complexity and computational costs. Several corrections are introduced to converge the approximated curvature to the exact Hessian matrix. Numerical results demonstrate the effectiveness and competitiveness of this method, whereas qualitative experiments highlight the impact of Hessian approximation granularity and parameter space utilization in Bayesian inference on mitigating overconfidence in predictions and obtaining high-quality uncertainty estimates.

An extended Gauss-Newton method for full-waveform inversion

Full-waveform inversion (FWI) is a large-scale nonlinear ill-posed problem for which computationally expensive Newton-type methods can become trapped in undesirable local minima, particularly when the initial model lacks a low-wavenumber component and the recorded data lack low-frequency content. A modification to the Gauss-Newton (GN) method is developed to address these issues. The standard GN system for multisource multireceiver FWI is reformulated into an equivalent matrix equation form, with the solution becoming a diagonal matrix rather than a vector as in the standard system. The search direction is transformed from a vector to a matrix by relaxing the diagonality constraint, effectively adding a degree of freedom to the subsurface offset axis. The relaxed system can be explicitly solved with only the inversion of two small matrices that deblur the data residual matrix along the source and receiver dimensions, which simplifies the inversion of the Hessian matrix. When used to solve the extended-source FWI objective function, the extended GN (EGN) method integrates the benefits of model and source extension. The EGN method effectively combines the computational effectiveness of the reduced FWI method with the robustness characteristics of extended formulations and offers a promising solution for addressing the challenges of FWI. It bridges the gap between these extended formulations and the reduced FWI method, enhancing inversion robustness while maintaining computational efficiency. The robustness and stability of the EGN algorithm for waveform inversion are demonstrated numerically.

Half-Sandwich Zirconium and Hafnium Amidoborane Complexes: Precursors of Hydride Derivatives.

Half-sandwich zirconium(IV) and hafnium(IV) complexes with amidoborane and hydride ligands have been isolated in the stoichiometric reactions of mono(pentamethylcyclopentadienyl)metal alkyl and amido derivatives with the amine-boranes NHR2BH3 (R2 = H2, Me2, HtBu). Treatment of the tris(trimethylsilylmethyl) complexes [M(η5-C5Me5)(CH2SiMe3)3] with NH3BH3 (3 equiv) gives the seven-coordinate species [M(η5-C5Me5)(NH2BH3)3] (M = Zr (1), Hf (2)) with three κ2N,H-NH2BH3 ligands. The tris(neophyl) [M(η5-C5Me5)(CH2CMe2Ph)3] or tris(dimethylamido) [M(η5-C5Me5)(NMe2)3] derivatives react with NHMe2BH3 (≥3 equiv) to afford bis(dimethylamidoborane) hydride complexes [M(η5-C5Me5)H(NMe2BH3)2] (M = Zr (3), Hf (4)) via thermally unstable [M(η5-C5Me5)(NMe2BH3)3] species. The reaction of [M(η5-C5Me5)(NMe2)3] and NH2tBuBH3 (≥4 equiv) affords analogous mixed amidoborane hydride derivatives [M(η5-C5Me5)H(NHtBuBH3)(NMe2BH3)] (M = Zr (5), Hf (6)) with κ2N,H-NHtBuBH3 and κ3N,H,H-NMe2BH3 ligands. The addition of NHR2BH3 (≥1 equiv) on the mono(dimethylamido) complexes [M(η5-C5Me5)Cl2(NMe2)] in hexane leads to the precipitation of the ionic compounds [(NHR2)2BH2][{M(η5-C5Me5)Cl2}2(μ-H)3] (R2 = Me2, M = Zr (7), Hf (8); R2 = HtBu, M = Zr (9), Hf (10)). Molecular hydride species [Cl2(η5-C5Me5)M(μ-Cl)(μ-H)2M(η5-C5Me5)Cl(NH2tBu)] (M = Zr (11), Hf (12)) could be isolated from mixtures of complexes [M(η5-C5Me5)Cl2(NMe2)] and lower ratios of NH2tBuBH3. The zirconium complex 11 decomposes in solution to give the mononuclear tert-butylamido derivative [Zr(η5-C5Me5)Cl2(NHtBu)] (13) along with other byproducts.

Schwartz Symmetry Condition for Coherent Diffraction Imaging Patterns

We demonstrate a symmetry condition for the mixed partial derivatives of measured data when performing a coherent diffraction imaging (CDI) experiment for differentiable samples under scientific investigation. The proposed condition can be used as a physical restriction to improve real data measurements and has been used within the most celebrated phase-retrieval inversion algorithms as an ad hoc constraint without proof. The symmetry relies on John’s ultrahyperbolic equation for the X-ray transform, which is also demonstrated to be valid in the imaging regime for CDI. The obtained conditions are easy to implement and can be used as a constraint by computational imaging methods.

Small spheres with prescribed nonconstant mean curvature in Riemannian manifolds

Given a function f on a smooth Riemannian manifold without boundary, we prove that if p∈M is a non-degenerate critical point of f, then a neighborhood of p contains a foliation by spheres with mean curvature proportional to f. This foliation is essentially unique. The nondegeneracy assumption can be substantially relaxed, at the expense of losing the property that the family of spheres with prescribed mean curvature defines a foliation.

FedNS: A Fast Sketching Newton-Type Algorithm for Federated Learning

Recent Newton-type federated learning algorithms have demonstrated linear convergence with respect to the communication rounds. However, communicating Hessian matrices is often unfeasible due to their quadratic communication complexity. In this paper, we introduce a novel approach to tackle this issue while still achieving fast convergence rates. Our proposed method, named as Federated Newton Sketch methods (FedNS), approximates the centralized Newton's method by communicating the sketched square-root Hessian instead of the exact Hessian. To enhance communication efficiency, we reduce the sketch size to match the effective dimension of the Hessian matrix. We provide convergence analysis based on statistical learning for the federated Newton sketch approaches. Specifically, our approaches reach super-linear convergence rates w.r.t. the communication rounds for the first time. We validate the effectiveness of our algorithms through various experiments, which coincide with our theoretical findings.

Communication Efficient Distributed Newton Method over Unreliable Networks

Distributed optimization in resource constrained devices demands both communication efficiency and fast convergence rates. Newton-type methods are getting preferable due to their superior convergence rates compared to the first-order methods. In this paper, we study a new problem in regard to the second-order distributed optimization over unreliable networks. The working devices are power-limited or operate in unfavorable wireless channels, experiencing packet losses during their uplink transmission to the server. Our scenario is very common in real-world and leads to instability of classical distributed optimization methods especially the second-order methods because of their sensitivity to the imprecision of local Hessian matrices. To achieve robustness to high packet loss, communication efficiency and fast convergence rates, we propose a novel distributed second-order method, called RED-New (Packet loss Resilient Distributed Approximate Newton). Each iteration of RED-New comprises two rounds of light-weight and lossy transmissions, in which the server aggregates the local information with a new developed scaling strategy. We prove the linear-quadratic convergence rate of RED-New. Experimental results demonstrate its advantage over first-order and second-order baselines, and its tolerance to packet loss rate ranging from 5% to 40%.

Clarifying the Behavior and the Difficulty of Adversarial Training

Adversarial training is usually difficult to optimize. This paper provides conceptual and analytic insights into the difficulty of adversarial training via a simple theoretical study, where we derive an approximate dynamics of a recursive multi-step attack in a simple setting. Despite the simplicity of our theory, it still reveals verifiable predictions about various phenomena in adversarial training under real-world settings. First, compared to vanilla training, adversarial training is more likely to boost the influence of input samples with large gradient norms in an exponential manner. Besides, adversarial training also strengthens the influence of the Hessian matrix of the loss w.r.t. network parameters, which is more likely to make network parameters oscillate and boosts the difficulty of adversarial training.

Lagrangian of extended multiconfigurational self-consistent field second-order quasidegenerate perturbation theory combined with reference interaction site model self-consistent field constraint spatial electron density.

Lagrangians of the state-averaged multiconfigurational self-consistent field (SA-MCSCF) and multistate extended second-order quasidegenerate perturbation theory (MS-XMCQDPT2) coupled with the reference interaction site model self-consistent field constraint spatial electron density are defined. In addition, variational equations were derived to calculate the excitation energies of the target molecules dissolved in various solvents. The theory was applied to a phenol molecule in various solutions, and the gradients and Hessian matrices were calculated to evaluate the absorption spectral lines, including the broadening bandwidth. Numerical calculations revealed fine structures in any solvent surroundings. The main intramolecular vibrational modes related to such fine structures were stretching vibrations of the aromatic ring and the oxygen atom of the phenol molecule. The present theory plays an important role in predicting the structure of potential energy surfaces, such as Hessian matrices for various solvent types, during the photoexcitation process.

Online stochastic Newton methods for estimating the geometric median and applications

In the context of large samples, a small number of individuals might spoil basic statistical indicators like the mean. It is difficult to detect automatically these atypical individuals, and an alternative strategy is using robust approaches. This paper focuses on estimating the geometric median of a random variable, which is a robust indicator of central tendency. In order to deal with large samples of data arriving sequentially, online stochastic Newton algorithms for estimating the geometric median are introduced and we give their rates of convergence. Since estimates of the median and those of the Hessian matrix can be recursively updated, we also determine confidences intervals of the median in any designated direction and perform online statistical tests.

Quasi-P-Wave Reverse Time Migration in TTI Media with a Generalized Fractional Convolution Stencil

In seismic modeling and reverse time migration (RTM), incorporating anisotropy is crucial for accurate wavefield modeling and high-quality images. Due to the trade-off between computational cost and simulation accuracy, the pure quasi-P-wave equation has good accuracy to describe wave propagation in tilted transverse isotropic (TTI) media. However, it involves a fractional pseudo-differential operator that depends on the anisotropy parameters, making it unsuitable for resolution using conventional solvers for fractional operators. To address this issue, we propose a novel pure quasi-P-wave equation with a generalized fractional convolution operator in TTI media. First, we decompose the conventional pure quasi-P-wave equation into an elliptical anisotropy equation and a fractional pseudo-differential correction term. Then, we use a generalized fractional convolution stencil to approximate the spatial-domain pseudo-differential term through the solution of an inverse problem. The proposed approximation method is accurate, and the wavefield modeling method based on it also accurately describes quasi-P-wave propagation in TTI media. Moreover, it only increases the computational cost for calculating mixed partial derivatives compared to those in vertical transverse isotropic (VTI) media. Finally, the proposed wavefield modeling method is utilized in RTM to correct the anisotropic effects in seismic imaging. Numerical RTM experiments demonstrate the flexibility and viability of the proposed method.

Nonlinear optimization for compact representation of orientation distributions based on generalized spherical harmonics

An orientation distribution is a necessary input in any crystal plasticity simulation. The computational time involved in crystal plasticity simulations scales linearly with the number of crystal orientations in the input distributions. Reducing the number of crystal orientations in representing the input orientation distributions quantitatively is a critical and necessary requirement for performing computationally efficient crystal plasticity simulations of deformation processes. A procedure for the compaction of orientation distribution functions (ODFs) relying on a spectral representation using series of generalized spherical harmonics (GSH) basis functions was recently developed. Linear fitting of the spectral representation of an ODF containing a compact set of weighted orientations towards a full-size ODF containing many crystal orientations was in the core of the procedure. This paper advances the compaction procedure by replacing the linear with a nonlinear optimization in Matlab for which a suitably defined error, gradient, and Hessian matrix are derived to allow for more efficient, accurate, and greater compactions. The utility of the new procedure is to allow for not only fitting the weights of a compacted set of orientations but to allow for optimizing weighted orientations or a combination of optimizing weights and orientations. The new compaction procedure has been successfully applied to compactions of large ODFs of cubic, hexagonal, and orthorhombic polycrystalline metals. In doing so, the evolution of texture, twinning, and stress-strain are predicted at large plastic strains with compact ODFs to agree with the corresponding full size ODFs using crystal plasticity models. In closing, guidance for effective texture compaction trading off the accuracy and computational gains are provided.

Convergence rate of entropy-regularized multi-marginal optimal transport costs

Abstract We investigate the convergence rate of multi-marginal optimal transport costs that are regularized with the Boltzmann–Shannon entropy, as the noise parameter $\varepsilon $ tends to $0$ . We establish lower and upper bounds on the difference with the unregularized cost of the form $C\varepsilon \log (1/\varepsilon )+O(\varepsilon )$ for some explicit dimensional constants C depending on the marginals and on the ground cost, but not on the optimal transport plans themselves. Upper bounds are obtained for Lipschitz costs or locally semiconcave costs for a finer estimate, and lower bounds for $\mathscr {C}^2$ costs satisfying some signature condition on the mixed second derivatives that may include degenerate costs, thus generalizing results previously in the two marginals case and for nondegenerate costs. We obtain in particular matching bounds in some typical situations where the optimal plan is deterministic.

Structural, topological, vibrational, and electronic analysis, and ADMET study of methyl-2-(4-isobutylphenyl)propanoate

In this work, the methyl ibuprofen ester, named methyl-2-(4-isobutylphenyl)propanoate (C14H20O2, hereinafter MET-IBU).was synthesized and characterized by 1D- and 2D-NMR, ATR-FTIR, FT-Raman, and UV–Vis spectroscopy, while their structural, QTAIM topological and spectroscopic properties were investigated using DFT calculations. Through the analysis of the topological properties of the electron density computed at bonding critical points (BCP's) for MET-IBU molecule were determined the electron density, the Laplacian of electron density, the eigenvalues of Hessian matrix, the ellipticity of electron density, the potential energy density, the kinetic energy density in the Lagrangian and Hamiltonian forms, the total energy density, the Lagrangian density, and the electron delocalization index. Density Functional Theory (DFT) calculations were also used to determine the vibrational normal modes, the frontier molecular orbitals (FMOs), the dipole moment, and the molecular electrostatic potential (MEP) of MET-IBU. Theoretical UV absorption spectrum was also evaluated from time-dependent density functional theory (TD-DFT) providing the electronic transitions, which are expected for this compound. The molecular descriptors of the properties of absorption, distribution, metabolism and excretion, and toxicity (ADMET) were obtained. Empirical models for estimating ADME attributes showed that the MET-IBU has a distribution volume that allows permeability and activity in the Central Nervous System - CNS, with structural contributions from the side chain of isobutyl and methoxy that favor liver metabolism and excretion with a low toxic response. Furthermore, the MET-IBU does not have ionizable centers compared to its analog Ibuprofen constituting a more lipophilic chemical product.

A Second-Order Finite-Difference Method for Derivative-Free Optimization

In this paper, a second-order finite-difference method is proposed for finding the second-order stationary point of derivative-free nonconvex unconstrained optimization problems. The forward-difference or the central-difference technique is used to approximate the gradient and Hessian matrix of objective function, respectively. The traditional trust-region framework is used, and we minimize the approximation trust region subproblem to obtain the search direction. The global convergence of the algorithm is given without the fully quadratic assumption. Numerical results show the effectiveness of the algorithm using the forward-difference and central-difference approximations.