Newton Update Research Articles

Theoretical insights on the pre-image resolution in machine learning

While many nonlinear pattern recognition and data mining tasks rely on embedding the data into a latent space, one often needs to extract the patterns in the input space. Estimating the inverse of the nonlinear embedding is the so-called pre-image problem. Several strategies have been proposed to address the estimation of the pre-image; However, there are no theoretical results so far to understand the pre-image problem and its resolution. In this paper, we provide theoretical underpinnings of the resolution of the pre-image problem in Machine Learning. These theoretical results are on the gradient descent optimization, the fixed-point iteration algorithm and Newton’s method. We provide sufficient conditions on the convexity/nonconvexity of the pre-image problem. Moreover, we show that the fixed-point iteration is a Newton update and prove that it is a Majorize-Minimization (MM) algorithm where the surrogate function is a quadratic function. These theoretical results are derived for the wide classes of radial kernels and projective kernels. We also provide other insights by connecting the resolution of this problem to the gradient density estimation problem with the so-called mean shift algorithm.

Inverse Hessian by stochastic projection and application to system identification in nonlinear mechanics of solids

Parameter identification in nonlinear mechanics of solids, a class of non-convex optimization problems of significant interest, often has to balance fast convergence with enhanced exploration. Local optimization schemes based on Newton or quasi-Newton updates cannot tread this fine balance. A similar limitation also applies to many derivative-free global evolutionary searches that typically end up with slower convergence. Notwithstanding a relatively rapid convergence around a local optimum, schemes using classical gradients require that the objective function be smooth. Thus, applied to real-world scenarios, these algorithms face challenges such as non-smoothness including discontinuities in the objective function and costly evaluations of gradients or Hessians, whenever possible. Based on a novel stochastic projection to estimate inverse Hessians, this work aims at generalizing a fast Newton-type local search whilst retaining capabilities of noise-assisted exploration in non-convex optimization. In principle, such a meeting of the best of both worlds in optimization should provide for a powerful paradigm shift, offering a versatile and robust tool for solving complex identification and variational problems arising in nonlinear mechanics of solids. The approach is well-suited even for cases where the objective function is implicitly defined and not amenable to explicit evaluation. In a first implementation of our proposal, we assess its performance against a few benchmark non-convex problems. Finally, we solve a parameter estimation problem for an isotropic and incompressible hyperelastic material.

Nonlinearly preconditioned semismooth Newton algorithms for nonlinear nonsmooth systems

We aim to develop efficient and robust algorithms for nonsmooth nonlinear systems arising from complementarity problems. The semismooth Newton algorithm is popular due to its reliability and efficiency. However, it struggles with issues with imbalanced nonlinearities of the problems, leading to degraded convergence rates or failure despite help from the globalization techniques like linesearch or trust region. We introduce a right nonlinearly preconditioned semismooth Newton algorithm to address this difficulty. The critical success ingredient is that before each global Newton update, a nonlinear preconditioning step implicitly removes the so-called ‘bad components’ causing trouble via nonlinear subspace correction, inspired by Gaussian elimination but adapted nonlinearly to balance system nonlinearities. Additionally, our method integrates with a domain decomposition framework, enhancing parallelism. Numerical results on two classes of problems demonstrate significantly improved convergence over standard semismooth Newton methods.

Combined peak price tracking strategies for online portfolio selection based on the meta-algorithm

Machine learning algorithms have been widely used to establish online portfolio selection strategies. Meta-algorithm, one of the machine learning algorithms, has the advantage of combining different base expert algorithm, which can greatly reduce the risk of choosing wrong experts, especially in the case of the base expert algorithm being very sensitive to the expert selection. In this paper, we investigate the online portfolio selection problem using Online Gradient Update (OGU) and Online Newton Update (ONU) meta-algorithms to combine expert advice for price prediction, in order to avoid the prediction bias caused by using a single expert that uses a fixed parameter. In addition, we theoretically analyze the price prediction algorithms’ regret bounds, which can guarantee that the prediction algorithms perform as well as the optimal combined prediction algorithm in hindsight. More importantly, we establish an online portfolio optimization model based on the price estimator, and obtain the portfolio update formula via Lagrange method. Finally, extensive numerical experiments are conducted to test the performance of the proposed strategies in several real-world stock markets. The empirical results show that our strategies are competitive and efficient in the diverse financial environment.

Communication Efficient Curvature Aided Primal-Dual Algorithms for Decentralized Optimization

This paper presents a family of algorithms for decentralized convex composite problems. We consider the setting of a network of agents that cooperatively minimize a global objective function composed of a sum of local functions plus a regularizer. Through the use of intermediate consensus variables, we remove the need for inner communication loops between agents when computing curvature-guided updates. A general scheme is presented which unifies the analysis for a plethora of computing choices, including gradient descent, Newton updates, and BFGS updates. Our analysis establishes sublinear convergence rates under convex objective functions with Lipschitz continuous gradients, as well as linear convergence rates when the local functions are further assumed to be strongly convex. Moreover, we explicitly characterize the acceleration due to curvature information. Last but not the least, we present an asynchronous implementation for the proposed algorithms, which removes the need for a central clock, with linear convergence rates established in expectation under strongly convex objectives. We ascertain the effectiveness of the proposed methods with numerical experiments on benchmark datasets.

On the iterative regularization of non-linear illposed problems in L^{infty }

Parameter identification tasks for partial differential equations are non-linear illposed problems where the parameters are typically assumed to be in L^infty . This Banach space is non-smooth, non-reflexive and non-separable and requires therefore a more sophisticated regularization treatment than the more regular L^p-spaces with 1<p<infty . We propose a novel inexact Newton-like iterative solver where the Newton update is an approximate minimizer of a smoothed Tikhonov functional over a finite-dimensional space whose dimension increases as the iteration progresses. In this way, all iterates stay bounded in L^infty and the regularizer, delivered by a discrepancy principle, converges weakly-star to a solution when the noise level decreases to zero. Our theoretical results are demonstrated by numerical experiments based on the acoustic wave equation in one spatial dimension. This model problem satisfies all assumptions from our theoretical analysis.

SVR-Net: A Sparse Voxelized Recurrent Network for Robust Monocular SLAM with Direct TSDF Mapping.

Simultaneous localization and mapping (SLAM) plays a fundamental role in downstream tasks including navigation and planning. However, monocular visual SLAM faces challenges in robust pose estimation and map construction. This study proposes a monocular SLAM system based on a sparse voxelized recurrent network, SVR-Net. It extracts voxel features from a pair of frames for correlation and recursively matches them to estimate pose and dense map. The sparse voxelized structure is designed to reduce memory occupation of voxel features. Meanwhile, gated recurrent units are incorporated to iteratively search for optimal matches on correlation maps, thereby enhancing the robustness of the system. Additionally, Gauss-Newton updates are embedded in iterations to impose geometrical constraints, which ensure accurate pose estimation. After end-to-end training on ScanNet, SVR-Net is evaluated on TUM-RGBD and successfully estimates poses on all nine scenes, while traditional ORB-SLAM fails on most of them. Furthermore, absolute trajectory error (ATE) results demonstrate that the tracking accuracy is comparable to that of DeepV2D. Unlike most previous monocular SLAM systems, SVR-Net directly estimates dense TSDF maps suitable for downstream tasks with high efficiency of data exploitation. This study contributes to the development of robust monocular visual SLAM systems and direct TSDF mapping.

Comparison of nonlinear field-split preconditioners for two-phase flow in heterogeneous porous media

This work focuses on the development of a two-step field-split nonlinear preconditioner to accelerate the convergence of two-phase flow and transport in heterogeneous porous media. We propose a field-split algorithm named Field-Split Multiplicative Schwarz Newton (FSMSN), consisting in two steps: first, we apply a preconditioning step to update pressure and saturations nonlinearly by solving approximately two subproblems in a sequential fashion; then, we apply a global step relying on a Newton update obtained by linearizing the system at the preconditioned state. Using challenging test cases, FSMSN is compared to an existing field-split preconditioner, Multiplicative Schwarz Preconditioned for Inexact Newton (MSPIN), and to standard solution strategies such as the Sequential Fully Implicit (SFI) method or the Fully Implicit Method (FIM). The comparison highlights the impact of the upwinding scheme in the algorithmic performance of the preconditioners and the importance of the dynamic adaptation of the subproblem tolerance in the preconditioning step. Our results demonstrate that the two-step nonlinear preconditioning approach—and in particular, FSMSN—results in a faster outer-loop convergence than with the SFI and FIM methods. The impact of the preconditioners on computational performance–i.e., measured by wall-clock time–will be studied in a subsequent publication.

Some new conjugate gradient methods for solving unconstrained optimization problems

Conjugate gradient algorithms come in a wide range of flavors. Conjugate gradient techniques primarily concentrate on the coefficient conjugate. We introduce a novel conjugate gradient approach that computes the parameter by using Newton updates. In addition, we have demonstrated that our conjugate gradient algorithms are globally convergent and descent property. For the specified test issues in [1] the performance profiles revealed that the novel conjugate gradient approach is effective and efficient.

Linear and nonlinear solvers for simulating multiphase flow within large-scale engineered subsurface systems

Multiphase flow simulation is well-known to be computationally demanding, and modeling large-scale engineered subsurface systems entails significant additional numerical challenges. These challenges arise from: (a) the presence of small-scale discrete features like shafts, tunnels, waste packages, and barriers; (b) the need to accurately represent both the waste form processes at the small spatial scale of the repository and the large-scale transport processes throughout heterogeneous geological formations; (c) the strong contrast in material properties such as porosity and permeability, as well as the nonlinear constitutive relations for multiphase flow. Numerical solution entails discretization of the coupled system of nonlinear governing equations and solving a linear system of equations at each Newton–Raphson iteration. Practical problems require a very large number of unknowns that must be solved efficiently using iterative methods in parallel on high-performance computers. The unique challenges noted above can lead to an ill-conditioned Jacobian matrix and non-convergence with Newton’s method due to discontinuous nonlinearity in constitutive models. Moreover, practical applications can require numerous Monte-Carlo simulations to explore uncertainly in material properties, geological heterogeneity, failure scenarios, or other factors; governmental regulatory agencies can mandate these as part of Performance Assessments. Hence there is a need for flexible, robust, and computationally efficient methods for multiphase flow in large-scale engineered subsurface systems.We apply the open-source simulator PFLOTRAN to the practical problem of performance assessment of the US DOE Waste Isolation Pilot Plant (WIPP) site. The simulator employs a finite volume discretization and uses the PETSc parallel framework. We evaluate the performance of several preconditioners for the iterative solution of the linearized Jacobian system; these range from stabilized-biconjugate-gradient with block-Jacobi preconditioning (BCGS) to methods adopted from reservoir modeling, such as the constrained pressure residual (CPR) two-stage preconditioner and flexible generalized residual solver (FGMRES). We also implement within PETSc the general-purpose nonlinear solver, Newton trust-region dogleg Cauchy (NTRDC), which truncates the Newton update or modifies the update with a Cauchy solution that is within the quadratic model trust-region of the objective function. We demonstrate the effectiveness of each method for a series of test problems with increasing difficulty. We find that the NTRDC and FGMRES-CPR-ABF (FCA) preconditioners generally perform best for the test problem having the extreme nonlinear processes, achieving a 50x speed-up compared with BCGS. The most ill-conditioned and extreme nonlinear simulations do not converge with BCGS (as one may expect), but they do complete the simulation with NTRDC and FCA. We also investigate the strong scalability of each method and demonstrate the impact of node-packing upon parallel performance on modern processor architectures.

Implicit algorithms for eigenvector nonlinearities

We study and derive algorithms for nonlinear eigenvalue problems, where the system matrix depends on the eigenvector, or several eigenvectors (or their corresponding invariant subspace). The algorithms are derived from an implicit viewpoint. More precisely, we change the Newton update equation in a way that the next iterate does not only appear linearly in the update equation. Although the modifications of the update equation make the methods implicit, we show how corresponding iterates can be computed explicitly. Therefore, we can carry out steps of the implicit method using explicit procedures. In several cases, these procedures involve a solution of standard eigenvalue problems. We propose two modifications, one of the modifications leads directly to a well-established method (the self-consistent field iteration) whereas the other method is to our knowledge new and has several attractive properties. Convergence theory is provided along with several simulations which illustrate the properties of the algorithms.

Sequential fully implicit Newton method for compositional flow and transport

Compositional simulation for flow and transport in porous media is essential for modeling Enhanced Oil Recovery (EOR) processes. The Sequential Fully Implicit (SFI) scheme is considered as a more flexible and potentially more computationally efficient alternative to the commonly used Fully Implicit Method (FIM). SFI was proposed along with the multiscale method where the flow and transport problems are solved in sequence. SFI opens the door for flexible discretizations, specialized linear solvers, and adaptive strategies. In the SFI method, a time step entails several ‘outer’ iterations. Each iteration consists of two steps: (1) a pressure equation is formed and solved implicitly, and then (2) a set of transport equations is solved, also implicitly. The (outer) iterations are conducted in a fixed-point manner until global convergence is reached. SFI can suffer from slow convergence, or even convergence failure, in strongly coupled flow and transport problems. Examples include dominant buoyancy and complex phase behaviors which are typical in compositional processes. Wong et al. [40] proposed the Sequential Fully Implicit Newton (SFIN) method for multi-physics problems in reservoir simulation and achieved superior outer loop convergence. In this work, we develop the SFIN method for compositional flow and transport problems, and in particular, show how the main assumption of fixing the total velocity can be incorporated. The key step of the SFIN method is a global Newton update that is performed at the end of each flow-transport iteration. During this outer Newton step, the pressure and transport variables are updated together to resolve the coupling between them better. The corresponding linear system is solved by a Krylov solver (e.g., GMRES) using a Jacobian-free approach that only entails matrix-vector multiplications. The latter is constructed by reusing previously computed Jacobian matrices and their preconditioners' information in a computationally efficient way. We present several challenging two- and three-dimensional examples and show that the SFIN method can significantly reduce the number of outer and inner loop iterations compared with the original (non-accelerated) SFI method. We analyze both the nonlinear and linear convergence behaviors for the SFIN method's key components and provide potential speedup estimates with respect to the SFI method.

Decentralized Approximate Newton Methods for Convex Optimization on Networked Systems

In this article, a class of decentralized approximate Newton (DEAN) methods for addressing convex optimization on a networked system is developed, where nodes in the networked system seek a consensus that minimizes the sum of their individual objective functions through local interactions only. The proposed DEAN algorithms allow each node to repeatedly perform a local approximate Newton update, which leverages tracking the global Newton direction and dissipating the discrepancies among the nodes. Under less restrictive problem assumptions in comparison with most existing second-order methods, the DEAN algorithms enable the nodes to reach a consensus that can be arbitrarily close to the optimum. Moreover, for a particular DEAN algorithm, the nodes linearly converge to a common suboptimal solution with an explicit error bound. Finally, simulations demonstrate the competitive performance of DEAN in convergence speed, accuracy, and efficiency.

A Nonlinear Solver with Phase Boundary Detection for Compositional Reservoir Simulation

Compositional simulation is typically used to study subsurface displacement processes that involve complex physics. Such processes are highly nonlinear and they occur at the interplay of phase thermodynamics (phase stability and split) and rock/fluid interaction (relative permeability). To better characterize the convergence failures of compositional nonlinear solvers, we study the analytical and discrete forms of the component flux functions for the transport problem. We focus on isothermal, two-hydrocarbon phase models in the presence of viscous forces. Our analysis shows that the component flux function exhibits both kinks and inflection points, as described in the literature. We study the kinks due to the crossing of phase boundaries and we show their impact on residual terms. We, then, investigate the location, direction, and evolution of the inflection loci inside the two-phase region. Our findings suggest that these inflection loci may exhibit complex features, as a result of which an a priori determination of such loci is computationally prohibitive. We propose a nonlinear solver that detects phase boundaries and guides the Newton update to avoid crossing these boundaries. Our numerical results show that this phase change detection solver has superior performance in comparison with standard Newton solvers for a wide range of cases.

Solution of multiphase Rachford-Rice equations by trust region method in compositional and thermal simulations

Robust and efficient multiphase flash calculations are crucial in compositional and thermal simulations for complex fluid systems in which three or four phases may co-exist. Solution of the Rachford-Rice (RR) equations is an important operation in the multiphase flash. The Newton method generally does not converge during solution of the RR equations unless very good initial values are provided. In this paper, the solution of the RR equations is formulated as a minimization of a convex function problem. For the first time, we use a trust-region (TR) method to solve the RR equations through minimization of the convex function. The Hessian matrix of the convex function is always positive-definite, and the TR-based solver guarantees convergence. The key to successful implementation is to determine the relaxation parameter in the Newton update. We select this relaxation parameter to meet the boundary of the objective function and to ensure an adequate step length. We tested the RR solver for three and four phase RR problems in the construction of phase diagrams. The test cases are representative of complex fluid systems encountered in enhanced oil recovery, including injection of CO2 into low temperature reservoirs and steam injection into heavy oil reservoirs at elevated temperatures. The test results reveal our RR solver to be robust, efficient, insensitive to initial values, and capable of handling negative phase amounts. We also evaluated the effect of the initial values on convergence and recommend methods to estimate the initial values in our RR solver. In summary, our RR solver greatly improves the multiphase flash calculations and strengthens the coupling of phase equilibrium calculations to the governing equations in multiphase compositional and thermal simulation.

Localized nonlinear solution strategies for efficient simulation of unconventional reservoirs

Accurate and efficient numerical simulation of unconventional reservoirs is challenging. Long periods of transient flow and steep potential gradients occur due to the extreme conductivity contrast between matrix and fracture. Detailed near-well/near-fracture models are necessary to provide sufficient resolution, but they are computationally impractical for field cases with multiple hydraulic-fracture stages.Previous works in the literature of unconventional simulations mainly focus on the gridding level that adapts to wells and fractures. Limited research has been conducted on nonlinear strategies that exploit locality across timesteps and nonlinear iterations. It was reported that an individual Newton update is typically sparse and nonlinear convergence is constrained by a small portion of the model. To perform localized computations, an a-prioristrategy is essential to first determine the active subset of simulation cells for the subsequent iteration. The active set flags the cells that will be updated, and then the corresponding localized linear system is solved.The objective of this work is to develop localization methods that are readily applicable to complex fracture networks and flow physics in unconventional reservoirs. By utilizing the diffusive nature of pressure updates, an adaptive algorithm is proposed to make adequate estimates for the active domains. In addition, we further develop a localized solver based on nonlinear domain decomposition (DD). Compared to a standard DD method, domain partitions are dynamically constructed. The new solver provides effective partitioning that adapts to flow dynamics and Newton updates.We evaluate the developed methods using several complex problems with discrete fracture networks. The problems consider multi-phase and compositional fluid systems with phase changes. The results show that large degrees of solution locality present across timesteps and iterations. Compared to a standard Newton solver, the new solvers enable superior computational performance. Moreover, Newton convergence behavior is preserved, without any impact on solution accuracy.

Gradient and Newton boosting for classification and regression

Boosting algorithms are frequently used in applied data science and in research. To date, the distinction between boosting with either gradient descent or second-order Newton updates is often not made in both applied and methodological research, and it is thus implicitly assumed that the difference is irrelevant. The goal of this article is to clarify this situation. In particular, we present gradient and Newton boosting, as well as a hybrid variant of the two, in a unified framework. We compare these boosting algorithms with trees as base learners using various datasets and loss functions. Our experiments show that Newton boosting outperforms gradient and hybrid gradient-Newton boosting in terms of predictive accuracy on the majority of datasets. We also present evidence that the reason for this is not faster convergence of Newton boosting. In addition, we introduce a novel tuning parameter for tree-based Newton boosting which is interpretable and important for predictive accuracy.

Time-domain elastic Gauss–Newton full-waveform inversion: a matrix-free approach

SUMMARYWe present a time-domain matrix-free elastic Gauss–Newton full-waveform inversion (FWI) algorithm. Our algorithm consists of a Gauss–Newton update with a search direction calculated via elastic least-squares reverse time migration (LSRTM). The conjugate gradient least-squares (CGLS) method solves the LSRTM problem with forward and adjoint operators derived via the elastic Born approximation. The Hessian of the Gauss–Newton method is never explicitly formed or saved in memory. In other words, the CGLS algorithm solves for the Gauss–Newton direction via the application of implicit-form forward and adjoint operators which are equivalent to elastic Born modelling and elastic reverse time migration, respectively. We provide numerical examples to test the proposed algorithm where we invert for P- and S-wave velocities simultaneously. The proposed algorithm performs positively on mid-size problems where we report solutions of slight improvement than those computed using the conventional non-linear conjugate gradient method. In spite of the aforementioned limited gain, the theory developed in this paper contributes to a better understanding of time-domain elastic Gauss–Newton FWI.

Localized linear systems for fully implicit simulation of multiphase multicomponent flow in porous media

During the solution of fully implicit reservoir simulation time steps, it is often observed that the computed Newton updates may be very sparse, considering computer precision. This sparsity can be as high as 95% and can vary largely from one iteration to the next. In recent works, a mathematically sound framework was developed to predict the sparsity pattern before the full linear system is solved. The theory is restricted to general, scalar nonlinear advection-diffusion-reaction problems in multidimensional and heterogeneous settings. This theory had been applied to reduce the size of the linear systems that were computed during sequential implicit time steps for two-phase flow. The results confirmed that the linearization computations and the linear solution processes may be localized by as much as 95% while retaining the exact Newton convergence behavior and final solution. Inspired by the great success of that methodology, this work develops algorithmic extensions in order to devise localization algorithms for fully implicit coupled multicomponent problems. We propose, apply, and test a novel algorithm to resolve a system of hyperbolic equations obtained from an Equation of State–based compositional simulator. When applied to various fully implicit flow and multicomponent transport problems, involving six thermodynamic species, on the SPE 10 geological model, the observed reduction in computational effort ranges between six and 49-fold depending on the level of locality present in the system. We apply this algorithm to several injection and depletion scenarios with and without gravity and capillarity in order to investigate the adaptivity and robustness of the proposed method to the underlying heterogeneity and complexity. We demonstrate that the algorithm enables efficient and robust full-resolution fully implicit simulation without the errors introduced by adaptive discretization methods or the stability concerns of semi-implicit approaches.

Ionosphere Ray Tracing of Radio‐Frequency Signals and Solution Sensitivities to Model Parameters

AbstractMethods are developed to determine the refracted propagation paths of high‐frequency radio signals in the ionosphere and to determine the sensitivities of these paths to changes of the input model parameters. These techniques are being developed to support the assimilation of data from monostatic and multistatic ionosondes with the goal of improving parameterized estimates of ionosphere electron density profiles. An additional application area is that of navigation using signals from a ground‐based network of high‐frequency beacons. A nonlinear two‐point boundary value problem solver is developed using the shooting method with Newton updates. Robust convergence is achieved by seeding the algorithm with carefully designed first guesses of the wave vector's initial aiming angles and terminal group delay. Partial derivative sensitivities of the raypath solution are calculated using the adjoint of the two‐point boundary value problem. This approach speeds the calculations when the partial derivatives of many ionosphere model parameters need to be computed. The new algorithm has been applied successfully to determine the paths of O‐mode and X‐mode radio waves between known transmitter and receiver locations and to spitze reflection points. Spitze singularities pose no difficulties for the new algorithm because it uses a Hamiltonian raypath formulation that remains nonsingular at a spitze.