Approximate Newton Method Research Articles

A modular framework for implicit 3D–0D coupling in cardiac mechanics

In numerical simulations of cardiac mechanics, coupling the heart to a model of the circulatory system is essential for capturing physiological cardiac behavior. A popular and efficient technique is to use an electrical circuit analogy, known as a lumped parameter network or zero-dimensional (0D) fluid model, to represent blood flow throughout the cardiovascular system. Due to the strong physical interaction between the heart and the blood circulation, developing accurate and efficient numerical coupling methods remains an active area of research. In this work, we present a modular framework for implicitly coupling three-dimensional (3D) finite element simulations of cardiac mechanics to 0D models of blood circulation. The framework is modular in that the circulation model can be modified independently of the 3D finite element solver, and vice versa. The numerical scheme builds upon a previous work that combines 3D blood flow models with 0D circulation models (3D fluid–0D fluid). Here, we extend it to couple 3D cardiac tissue mechanics models with 0D circulation models (3D structure–0D fluid), showing that both mathematical problems can be solved within a unified coupling scheme. The effectiveness, temporal convergence, and computational cost of the algorithm are assessed through multiple examples relevant to the cardiovascular modeling community. Importantly, in an idealized left ventricle example, we show that the coupled model yields physiological pressure–volume loops and naturally recapitulates the isovolumic contraction and relaxation phases of the cardiac cycle without any additional numerical techniques. Furthermore, we provide a new derivation of the scheme inspired by the Approximate Newton Method of Chan (1985), explaining how the proposed numerical scheme combines the stability of monolithic approaches with the modularity and flexibility of partitioned approaches.

Accelerated Distributed Approximate Newton Method.

Distributed second-order optimization, as an effective strategy for training large-scale machine learning systems, has been widely investigated due to its low communication complexity. However, the existing distributed second-order optimization algorithms, including distributed approximate Newton (DANE), accelerated inexact DANE (AIDE), and statistically preconditioned accelerated gradient (SPAG), are all required to precisely solve an expensive subproblem up to the target precision. Therefore, this causes these algorithms to suffer from high computation costs and this hinders their development. In this article, we design a novel distributed second-order algorithm called the accelerated distributed approximate Newton (ADAN) method to overcome the high computation costs of the existing ones. Compared with DANE, AIDE, and SPAG, which are constructed based on the relative smooth theory, ADAN's theoretical foundation is built upon the inexact Newton theory. The different theoretical foundations lead to handle the expensive subproblem efficiently, and steps required to solve the subproblem are independent of the target precision. At the same time, ADAN resorts to the acceleration and can effectively exploit the objective function's curvature information, making ADAN to achieve a low communication complexity. Thus, ADAN can achieve both the communication and computation efficiencies, while DANE, AIDE, and SPAG can achieve only the communication efficiency. Our empirical study also validates the advantages of ADAN over extant distributed second-order algorithms.

Dissipation and time step scaling strategies for low and high Mach number flows

Aircraft aerodynamic design often involves evaluating flow conditions that span low subsonic to transonic or even supersonic Mach numbers. Compressible flow solvers are a natural choice for such design problems, but these solvers suffer from reduced accuracy and efficiency at low Mach numbers. In addition, simulations with supersonic conditions can be challenging to converge because of large gradients in the flow field. This paper presents three contributions to address these issues in the context of an approximate Newton–Krylov solver for the Reynolds-averaged Navier–Stokes equations. First, we propose a method for scaling the artificial dissipation terms in the Jameson–Schmidt–Turkel scheme to improve its accuracy at low Mach numbers while retaining the simplicity of the original scalar dissipation formulation. Second, we show that characteristic time-stepping combined with an approximate Newton method can accelerate convergence for low Mach number flows by reducing the stiffness in the linear system for each Newton iteration. Third, we introduce a dissipation-based continuation method for flows with shocks that improves robustness and accelerates convergence without sacrificing accuracy. These methods can make compressible flow solvers more accurate and efficient across low and high Mach number regimes.

A globally convergent approximate Newton method for non-convex sparse learning

Newton-type greedy pursuit methods have been shown to work favorably for cardinality-constrained sparse learning problems. The appealing sparsity recovery performance of the existing Newton-type greedy pursuit methods, however, is typically guaranteed within a local neighborhood around the target solution. To address this limitation, we present in this paper a novel approximate Newton pursuit method for sparse learning with linear models. The computation procedure of our method iterates between constructing an inexact Newton-type quadratic majorization to the global empirical risk and solving the quadratic approximation via iterative hard thresholding. Provable global guarantees on mean squared prediction error, which is less understood for prior methods, are provided for our method. Numerical evidence is provided to show the advantages of our approach over the prior methods.

Second-Order Approximation of Minimum Discrimination Information in Independent Component Analysis

Independent Component Analysis (ICA) is intended to recover the mutually independent sources from their linear mixtures, and <inline-formula><tex-math notation="LaTeX">$FastICA$</tex-math></inline-formula> is one of the most successful ICA algorithms. Although it seems reasonable to improve the performance of <inline-formula><tex-math notation="LaTeX">$FastICA$</tex-math></inline-formula> by introducing more nonlinear functions to the negentropy estimation, the original fixed-point method (approximate Newton method) in <inline-formula><tex-math notation="LaTeX">$FastICA$</tex-math></inline-formula> degenerates under this circumstance. To alleviate this problem, we propose a novel method based on the second-order approximation of minimum discrimination information (MDI). The joint maximization in our method is consisted of minimizing single weighted least squares and seeking unmixing matrix by the fixed-point method. Experimental results validate its efficiency compared with other popular ICA algorithms.

Decentralized Approximate Newton Methods for Convex Optimization on Networked Systems

In this article, a class of decentralized approximate Newton (DEAN) methods for addressing convex optimization on a networked system is developed, where nodes in the networked system seek a consensus that minimizes the sum of their individual objective functions through local interactions only. The proposed DEAN algorithms allow each node to repeatedly perform a local approximate Newton update, which leverages tracking the global Newton direction and dissipating the discrepancies among the nodes. Under less restrictive problem assumptions in comparison with most existing second-order methods, the DEAN algorithms enable the nodes to reach a consensus that can be arbitrarily close to the optimum. Moreover, for a particular DEAN algorithm, the nodes linearly converge to a common suboptimal solution with an explicit error bound. Finally, simulations demonstrate the competitive performance of DEAN in convergence speed, accuracy, and efficiency.

A Riemannian optimization approach to the radial distribution network load flow problem

The Load Flow (LF) equations in power networks are the foundation of several applications on active and reactive power flow control, distributed and real-time control and optimization. In this paper, we formulate the LF problem in radial electricity distribution networks as an unconstrained Riemannian optimization problem, consisting of two manifolds, and we consider alternative retractions and initialization options. We introduce a Riemannian approximate Newton method tailored to the LF problem, as an exact solution method guaranteeing monotonic descent and local superlinear convergence rate. Extensive numerical comparisons on several test networks illustrate that the proposed method outperforms standard Riemannian optimization methods (Gradient Descent, Newton’s), and achieves comparable performance with the traditional Newton–Raphson method (in Euclidean coordinates), albeit besting it by a guarantee to convergence. We also consider an approximate LF solution obtained by the first iteration of the proposed method, and we show that it significantly outperforms other approximants in the LF literature. Lastly, we derive an interesting analogy with the well-known Backward–Forward Sweep (BFS) method showing that BFS iterations move on the power flow manifold, and highlighting the advantage of our method in converging under high constant impedance loading conditions, whereas BFS may diverge.

Optimal design of acoustic metamaterial cloaks under uncertainty

In this work, we consider the problem of optimal design of an acoustic cloak under uncertainty and develop scalable approximation and optimization methods to solve this problem. The design variable is taken as an infinite-dimensional spatially-varying field that represents the material property, while an additive infinite-dimensional random field represents, e.g., the variability of the material property or the manufacturing error. Discretization of this optimal design problem results in high-dimensional design variables and uncertain parameters. To solve this problem, we develop a computational approach based on a Taylor approximation and an approximate Newton method for optimization, which is based on a Hessian derived at the mean of the random field. We show our approach is scalable with respect to the dimension of both the design variables and uncertain parameters, in the sense that the necessary number of acoustic wave propagations is essentially independent of these dimensions, for numerical experiments with up to one million design variables and half a million uncertain parameters. We demonstrate that, using our computational approach, an optimal design of the acoustic cloak that is robust to material uncertainty is achieved in a tractable manner. The optimal design under uncertainty problem is posed and solved for the classical circular obstacle surrounded by a ring-shaped cloaking region, subjected to both a single-direction single-frequency incident wave and multiple-direction multiple-frequency incident waves. Finally, we apply the method to a deterministic large-scale optimal cloaking problem with complex geometry, to demonstrate that the approximate Newton method's Hessian computation is viable for large, complex problems.

Two-Phase Switching Optimization Strategy in Deep Neural Networks.

Optimization in a deep neural network is always challenging due to the vanishing gradient problem and intensive fine-tuning of network hyperparameters. Inspired by multistage decision control systems, the stochastic diagonal approximate greatest descent (SDAGD) algorithm is proposed in this article to seek for optimal learning weights using a two-phase switching optimization strategy. The proposed optimizer controls the relative step length derived based on the long-term optimal trajectory and adopts the diagonal approximated Hessian for efficient weight update. In Phase-I, it computes the greatest step length at the boundary of each local spherical search region and, subsequently, descends rapidly toward the direction of an optimal solution. In Phase-II, it switches to an approximate Newton method automatically once it is closer to the optimal solution to achieve fast convergence. The experiments show that SDAGD produces steeper learning curves and achieves lower misclassification rates compared with other optimization techniques. Implementation of the proposed optimizer to deeper networks is also investigated in this article to study the vanishing gradient problem.

Fast Distributed Voltage Control for PV Generation Clusters Based on Approximate Newton Method

Solar power generation is exhibiting a distributed development trend, and has changed conventional characteristics of power supply and consumption. The integration of large-scale distributed photovoltaic (PV) generation forms high-penetration PV clusters in distribution networks, which aims to organize and control geographically scattered resources. However, due to the inherent uncertainty, high-penetration PV clusters are suffering from random disturbances and serious voltage risks. Besides, integrating system via power electronic interfaces, PV generation usually tracks system frequency without any backup and regulation capacity, which poses significant challenges to system operation, and requires greater flexibility of voltage control. In fact, by using appropriate methods, PV inverters can autonomously regulate reactive power output in a distributed manner to improve voltage profiles in clusters. In this article, a distributed voltage control method for PV generation clusters is presented to realize decentralized coordination of PV inverters. Based on matrix splitting and approximate Newton method, it can fast respond to reactive power mismatch and realize voltage profiles optimization. Exhibiting a more efficient, reliable and flexible performance than existing decentralized methods in case studies, the proposed method is demonstrated to effectively organize and control PV cluster to provide corresponding reactive power support and exert friendly effect on system operation.

ECT Image Reconstruction Based on Alternating Direction Approximate Newton Algorithm

To solve the underdetermined problem and the mismatches of position and size of the objects caused by the ill-posed sensitivity matrix in electrical capacitance tomography (ECT) image reconstruction, an alternating direction approximate newton (ADAN) method is developed. Based on the alternating direction method of multipliers (ADMMs), this algorithm utilizes the Bregman operator splitting (BOS) algorithm to segment the total variation (TV) regularization model. An approximation to Newton's method, in which a term in Newton's hessian is replaced by a Barzilai-Borwein (BB) approximation, is used to solve the subproblems after segmentation, and the complete convergence proof and the stability analysis of the algorithm are provided. The quantitative analysis of reconstruction results of the ADAN, Landweber, modified Landweber, Newton, and ${l}_{1}$ regularization methods shows that the average relative error (RE) of ADAN in the reconstruction of ten flow images is 0.2199, which is lower than that of reconstructed by the other four algorithms. The average correlation coefficient (CC) of the reconstructed images of ADAN is higher than that reconstructed by the ${l}_{1}$ regularization, Newton, modified Landweber, and Landweber methods, which are 0.8539, 0.6792, 0.6251, 0.6545, 0.6665, and 0.6614, separately. Moreover, under the same iteration stopping condition, the iteration time of ADAN is the shortest.

Distributed Approximate Newton's Method Robust to Byzantine Attackers

There is a recent surge of interest in the design of the first-order and the second-order distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified information to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers' choice. Some recent works have proposed algorithms that can defend against Byzantine attackers for the first-order methods. In this paper, we design two algorithms that can deal with Byzantine attackers for the second-order methods. The main idea of the first algorithm, named median-based approximate Newton's method (MNM), is to ask the parameter server to aggregate gradient information and approximate Newton's direction from all workers by geometric median. We show that MNM can converge when up to half of the workers are Byzantine attackers. To deal with the case with an arbitrary number of attackers, we then propose a comparison-based approximate Newton's method (CNM). The main idea of CNM is to ask the server to randomly select a small clean dataset and compute noisy gradient and Newton's direction using this small dataset. These noisy information will then be used as an approximation of the ground truth to filter out bad information from Byzantine attackers. We show that CNM can converge to the neighborhood of the population minimizer even when more than half of the workers are Byzantine workers. We further provide numerical examples to illustrate the performance of the proposed algorithms.

Discrete Calabi Flow: A Unified Conformal Parameterization Method

AbstractConformal parameterization for surfaces into various parameter domains is a fundamental task in computer graphics. Prior research on discrete Ricci flow provided us with promising inspirations from methods derived via Riemannian geometry, which is rigorous in theory and effective inpractice. In this paper, we propose a unified conformal parameterization approachfor turning triangle meshes into planar and spherical domains using discrete Calabi flow onpiecewise linear metric. We incorporate edge‐flipping surgery to guarantee convergence as well as other significant improvements including approximate Newton's method, optimal step‐lengths, priority embedding and boundary customizing, which achieve better performance and functionality with robustness and accuracy.

Convergent iteration in Sobolev space for time dependent closed quantum systems

Time dependent quantum systems have become indispensable in science and its applications, particularly at the atomic and molecular levels. Here, we discuss the approximation of closed time dependent quantum systems on bounded domains, via iterative methods in Sobolev space based upon evolution operators. Recently, existence and uniqueness of weak solutions were demonstrated by a contractive fixed point mapping defined by the evolution operators. Convergent successive approximation is then guaranteed. This article uses the same mapping to define quadratically convergent Newton and approximate Newton methods. Estimates for the constants used in the convergence estimates are provided. The evolution operators are ideally suited to serve as the framework for this operator approximation theory, since the Hamiltonian is time-dependent. In addition, the hypotheses required to guarantee quadratic convergence of the Newton iteration build naturally upon the hypotheses used for the existence/uniqueness theory.

Parameterized Pseudo-Localization for Accurate and Efficient Moving Targets Imaging in Synthetic Aperture Radar.

Accurate and efficient moving target imaging is an important challenge for targets recognition in current synthetic aperture radar (SAR) combined with a ground moving target indication (GMTI) system. As the key but unknown parameter, the Doppler rates are estimated conventionally by searching any possible values for moving targets imaging. However, this conventional estimation method suffers from low accuracy or low efficiency due to the searching procedure. Focusing on these, we present a method to efficiently image the moving targets without the Doppler rate by Doppler delayed interferometry, and the imaged localization, which is parameterized pseudo-localization, is used to estimate the Doppler rate. In order to improve the estimation accuracy, an improved method based on the Newton method of approximation is proposed by exploiting the unused amplitude information. Compared with the conventional methods, the proposed improved method capable of high accuracy and low computation complexity simultaneously can meet the accurate and efficient requirements in the practical applications. Comparison simulations and real data processing results demonstrate the effectiveness of the proposed methods.

On the dynamics of a third order Newton's approximation method

We provide an answer to a question raised by S. Amat, S. Busquier, S. Plaza on the qualitative analysis of the dynamics of a certain third order Newton type approximation function , by proving that for functions f twice continuously differentiable and such that both f and its derivative do not have multiple roots, with at least four roots and infinite limits of opposite signs at , has periodic points of any prime period and that the set of points a at which the approximation sequence does not converge is uncountable. In addition, we observe that in their Scaling Theorem analyticity can be replaced with differentiability.

Linear filtering with adaptive adjustment of the disturbance covariation matrices in the plant and measurement noise

For filtering a nonstationary linear plant under the unknown intensities of input signals such as plant disturbances and measurement noise, a new algorithm was presented. It is based on selecting the vectors of values of these signals compatible with the observed plant output and minimizing the error variances of the last predicted measurement. The measurement prediction is determined from the Kalman filter where the input signals are assumed to be white noise and the covariance matrix coincides with the empirical covariance matrix of the selected vectors. Numerical modeling demonstrated that the so-calculated filter coefficients are close to the optimal ones constructed from the true covariance matrices of plant disturbances and measurement noise. The approximate Newton method for minimization of the prediction error variance was shown to agree with the solution of the auxiliary optimal control problem, which allows to make one or some few iterations to find the point of minimum.

An Alternating Direction Approximate Newton Algorithm for Ill-Conditioned Inverse Problems with Application to Parallel MRI

An alternating direction approximate Newton (ADAN) method is developed for solving inverse problems of the form \(\min \{\phi (Bu) +(1/2)\Vert Au-f\Vert _{2}^{2}\},\) where \(\phi \) is convex and possibly nonsmooth, and A and B are matrices. Problems of this form arise in image reconstruction where A is the matrix describing the imaging device, f is the measured data, \(\phi \) is a regularization term, and B is a derivative operator. The proposed algorithm is designed to handle applications where A is a large dense, ill-conditioned matrix. The algorithm is based on the alternating direction method of multipliers (ADMM) and an approximation to Newton’s method in which a term in Newton’s Hessian is replaced by a Barzilai–Borwein (BB) approximation. It is shown that ADAN converges to a solution of the inverse problem. Numerical results are provided using test problems from parallel magnetic resonance imaging. ADAN was faster than a proximal ADMM scheme that does not employ a BB Hessian approximation, while it was more stable and much simpler than the related Bregman operator splitting algorithm with variable stepsize algorithm which also employs a BB-based Hessian approximation.

A Brief History of the Most Remarkable Numbers $$\pi $$ π , $$g$$ g and $$\delta $$ δ in Mathematical Sciences with Applications

This paper deals with a brief history of the most remarkable numbers $$\pi , g \text {and} \delta $$ in mathematical sciences with their many examples of applications. Several series, products, continued fractions and integral representations of $$\pi $$ are discussed with examples. The celebrated Newton method of approximation of $$\pi $$ to many decimal places is included. The appearance of $$\pi $$ in many problems, formulas, elliptic integrals and in probability and statistics is presented with examples of applications including the Tchebycheff problem of prime numbers, the Buffon needle problem and the Euler quadratic polynomial. The golden number $$g$$ and its applications to algebra and geometry are briefly discussed. The Feigenbaum universal constant, $$\delta $$ is discovered in 1978 and it is found to occur in many period doubling bifurcation phenomena in the celebrated logistic map and the Lorenz differential equation system with chaotic (or aperiodic) solutions. Included is a numerically computed Lorenz attractor which resembles a butterfly or figure eight. The Lorenz attractor is a strange attractor because it has a non-integer (or fractal) dimension. The major focus of this article is to provide basic pedagogical information through historical approach to mathematics teaching and learning of the fundamental knowledge and skills required for students and teachers at all levels so that they can understand the concepts of mathematics, and mathematics education in science and technology and pursue further research.

An approximate Newton method for solving non-smooth equations with infinite max functions

In this paper, a new version of an approximate Newton method for solving non-smooth equations with infinite max function is presented. This method uses a difference approximation of the generalized Jacobian based on a weak consistently approximated Jacobian. Numerical example is reported for the generalized Newton method using two versions of approximation.