Newton Algorithm Research Articles

Regression and Classification With Spline-Based Separable Expansions.

We introduce a supervised learning framework for target functions that are well approximated by a sum of (few) separable terms. The framework proposes to approximate each component function by a B-spline, resulting in an approximant where the underlying coefficient tensor of the tensor product expansion has a low-rank polyadic decomposition parametrization. By exploiting the multilinear structure, as well as the sparsity pattern of the compactly supported B-spline basis terms, we demonstrate how such an approximant is well-suited for regression and classification tasks by using the Gauss–Newton algorithm to train the parameters. Various numerical examples are provided analyzing the effectiveness of the approach.

A damped Newton algorithm for generated Jacobian equations

Generated Jacobian Equations have been introduced by Trudinger (Discrete Contin Dyn Syst A 34(4):1663–1681, 2014) as a generalization of Monge–Ampère equations arising in optimal transport. In this paper, we introduce and study a damped Newton algorithm for solving these equations in the semi-discrete setting, meaning that one of the two measures involved in the problem is finitely supported and the other one is absolutely continuous. We also present a numerical application of this algorithm to the near-field parallel reflector problem arising in non-imaging problems.

Spectral CG Algorithm for Solving Fuzzy Nonlinear Equations

The nonlinear conjugate gradient method is an effective technique for solving large-scale minimizations problems, and has a wide range of applications in various fields, such as mathematics, chemistry, physics, engineering and medicine. This study presents a novel spectral conjugate gradient algorithm (non-linear conjugate gradient algorithm), which is derived based on the Hisham–Khalil (KH) and Newton algorithms. Based on pure conjugacy condition The importance of this research lies in finding an appropriate method to solve all types of linear and non-linear fuzzy equations because the Buckley and Qu method is ineffective in solving fuzzy equations. Moreover, the conjugate gradient method does not need a Hessian matrix (second partial derivatives of functions) in the solution. The descent property of the proposed method is shown provided that the step size at meets the strong Wolfe conditions. In numerous circumstances, numerical results demonstrate that the proposed technique is more efficient than the Fletcher–Reeves and KH algorithms in solving fuzzy nonlinear equations.

Scalable proximal methods for cause-specific hazard modeling with time-varying coefficients.

Survival modeling with time-varying coefficients has proven useful in analyzing time-to-event data with one or more distinct failure types. When studying the cause-specific etiology of breast and prostate cancers using the large-scale data from the Surveillance, Epidemiology, and End Results (SEER) Program, we encountered two major challenges that existing methods for estimating time-varying coefficients cannot tackle. First, these methods, dependent on expanding the original data in a repeated measurement format, result in formidable time and memory consumption as the sample size escalates to over one million. In this case, even a well-configured workstation cannot accommodate their implementations. Second, when the large-scale data under analysis include binary predictors with near-zero variance (e.g., only 0.6% of patients in our SEER prostate cancer data had tumors regional to the lymph nodes), existing methods suffer from numerical instability due to ill-conditioned second-order information. The estimation accuracy deteriorates further with multiple competing risks. To address these issues, we propose a proximal Newton algorithm with a shared-memory parallelization scheme and tests of significance and nonproportionality for the time-varying effects. A simulation study shows that our scalable approach reduces the time and memory costs by orders of magnitude and enjoys improved estimation accuracy compared with alternative approaches. Applications to the SEER cancer data demonstrate the real-world performance of the proximal Newton algorithm.

Optimal robot-world and hand-eye calibration with rotation and translation coupling

AbstractA classic hand-eye system involves hand-eye calibration and robot-world and hand-eye calibration. Insofar as hand-eye calibration can solve only hand-eye transformation, this study aims to determine the robot-world and hand-eye transformations simultaneously based on the robot-world and hand-eye equation. According to whether the rotation part and the translation part of the equation are decoupled, the methods can be divided into separable solutions and simultaneous solutions. The separable solutions solve the rotation part before solving the translation part, so the estimated errors of the rotation will be transferred to the translation. In this study, a method was proposed for calculation with rotation and translation coupling; a closed-form solution based on Kronecker product and an iterative solution based on the Gauss–Newton algorithm were involved. The feasibility was further tested using simulated data and real data, and the superiority was verified by comparison with the results obtained by the available method. Finally, we improved a method that can solve the singularity problem caused by the parameterization of the rotation matrix, which can be widely used in the robot-world and hand-eye calibration. The results show that the prediction errors of rotation and translation based on the proposed method be reduced to $0.26^\circ$ and $1.67$ mm, respectively.

Distributed adaptive Newton methods with global superlinear convergence

This paper considers the distributed optimization problem where each node of a peer-to-peer network minimizes a finite sum of objective functions by communicating with its neighboring nodes. In sharp contrast to the existing literature where the fastest distributed algorithms converge either with a global linear or a local superlinear rate, we propose a distributed adaptive Newton (DAN) algorithm with a global quadratic convergence rate. Our key idea lies in the design of a finite-time set-consensus method with Polyak’s adaptive stepsize. Moreover, we introduce a low-rank matrix approximation (LA) technique to compress the innovation of Hessian matrix so that each node only needs to transmit message of dimension O(p) (where p is the dimension of decision vectors) per iteration, which is essentially the same as that of first-order methods. Nevertheless, the resulting DAN-LA converges to an optimal solution with a global superlinear rate. Numerical experiments on logistic regression problems are conducted to validate their advantages over existing methods.

Subfields of algebraically maximal Kaplansky fields

Using the ramification theory of tame and Kaplansky fields, we show that maximal Kaplansky fields contain maximal immediate extensions of each of their subfields. Likewise, algebraically maximal Kaplansky fields contain maximal immediate algebraic extensions of each of their subfields. This study is inspired by problems that appear in henselian valued fields of rank higher than 1 when a Hensel root of a polynomial is approximated by the elements generated by a (transfinite) Newton algorithm. The main result of this article has been applied in the theory of separated (also called vector space defectless) extensions of valued fields.

A general adaptive ridge regression method for generalized linear models: an iterative re-weighting approach

This article is concerned with the problem of variable selection and estimation for high dimensional generalized linear models. In this article, we introduce a general iteratively reweighted adaptive ridge regression method (GAR). We show that the GAR estimator possesses oracle property and grouping effect. A data-driven parameter γ is introduced in the GAR method to adapt the different cases of the true model. Then, such an adaptive parameter γ is adequately taken into consideration to establish a γ-dependent sufficient condition to guarantee the oracle property and the grouping effect. Furthermore, to apply the GAR method more efficiently, a coordinate-wise Newton algorithm is employed to successfully avoid the inverse matrix operation and the numerical instability caused by iteration. Extensive numerical simulation results show that the GAR method outperforms the commonly used methods, and the GAR method is tested on the gastric cancer dataset for further illustration.

Smooth augmented Lagrangian method for twin bounded support vector machine

<p style='text-indent:20px;'>In this paper, we propose a method for solving the twin bounded support vector machine (TBSVM) for the binary classification. To do so, we use the augmented Lagrangian (AL) optimization method and smoothing technique, to obtain new unconstrained smooth minimization problems for TBSVM classifiers. At first, the augmented Lagrangian method is recruited to convert TBSVM into unconstrained minimization programming problems called as AL-TBSVM. We attempt to solve the primal programming problems of AL-TBSVM by converting them into smooth unconstrained minimization problems. Then, the smooth reformulations of AL-TBSVM, which we called AL-STBSVM, are solved by the well-known Newton's algorithm. Finally, experimental results on artificial and several University of California Irvine (UCI) benchmark data sets are provided along with the statistical analysis to show the superior performance of our method in terms of classification accuracy and learning speed.</p>

Construction of College Students' Employment Quality Evaluation Model System under the Background of Digitalization.

In the era of knowledge economy, human resources are being valued by various countries and regions. The report of the 19th National Congress of the Communist Party of China pointed out that “talent is a strategic resource for realizing national rejuvenation and winning the initiative of international competition. We must adhere to the principle of the party's management of talents, gather talents from all over the world and use them, and accelerate the construction of a strong country with talents.” College students are an important part of talents, and their employment intentions directly affect employment behavior. With the development of education in our country, the enrollment quota of most colleges and universities in the country has gradually increased, and the number of graduates has also increased. Social and economic development has different needs for different professional and technical personnel, and the employment situation in different regions is uneven. Under the increasingly complex employment environment, college students have to face greater employment pressure and compete with each other in a narrower employment field. Therefore, it is necessary to conduct better employment guidance and employment quality evaluation for college students. Based on the improved algorithm of BP network, an artificial intelligence-based employment quality evaluation model is constructed. The design model is optimized by introducing a momentum variable factor, adjusting the learning rate and quasi-Newton method, and training and recommending each optimization model through the training data. The experimental results show that the iterations of the gradient descent algorithm and the additional momentum optimization algorithm are far more than 1000 times. Second, the optimal validation errors of the two algorithms are large and the model performance is poor. The quasi-Newton ring algorithm also has faster coordination speed, stronger stability, and better overall performance. The adaptive learning rate optimization algorithm is performed in these 4 algorithms. In terms of accuracy, the accuracy of the adaptive learning rate BP optimization algorithm is 76.4%, followed by the Newton algorithm and the additional momentum algorithm, and the gradient descent algorithm is the worst.

Robust Dynamic Mode Decomposition

This paper develops a robust dynamic mode decomposition (RDMD) method endowed with statistical and numerical robustness. Statistical robustness ensures estimation efficiency at the Gaussian and non-Gaussian probability distributions, including heavy-tailed distributions. The proposed RDMD is statistically robust because the outliers in the data set are flagged via projection statistics and suppressed using a Schweppe-type Huber generalized maximum-likelihood estimator that minimizes a convex Huber cost function. The latter is solved using the iteratively reweighted least-squares algorithm that is known to exhibit an excellent convergence property and numerical stability than the Newton algorithms. Several numerical simulations using canonical models of dynamical systems demonstrate the excellent performance of the proposed RDMD method. The results reveal that it outperforms several other methods proposed in the literature.

Inverse Scattering by Perfectly Electric Conducting (PEC) Rough Surfaces: An Equivalent Model With Line Sources

This paper presents a new method for the reconstruction of the perfectly electric conducting (PEC) rough surface profiles by utilizing electromagnetic waves. The inaccessible rough surface is illuminated by a tapered plane electromagnetic wave and the scattered field data are measured on a certain number of points above the surface under test. The method for the inverse electromagnetic imaging problem is based on a special representation of the scattered field in terms of a finite number of fictitious discrete line sources located along a plane below the rough surface. The current densities of these fictitious sources are obtained through the regularized solution of an ill-posed problem. Then, it is shown that the image of the rough surface can be directly retrieved by seeking the points in the space where the tangential component of the total electric field vanishes. Alternatively, a much more rigorous iterative method based on a regularized Newton algorithm is also presented. A comprehensive numerical analysis is provided to demonstrate the feasibility of the presented approach. In this context, the quantitative successes of both approaches are interpreted by considering a very sensitive ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> -norm based error function between the actual and the reconstructed surface profiles. Regarding different scattering scenarios taken into account, the error values obtained for satisfactory reconstructions are generally in the range of 10% - 30% for both methods. It is also shown that the presented algorithms are capable of reconstructing the rough surfaces which oscillate for every λ horizontally and have a peak to peak variation 0.5λ at most.

The recursive variational Gaussian approximation (R-VGA)

We consider the problem of computing a Gaussian approximation to the posterior distribution of a parameter given N observations and a Gaussian prior. Owing to the need of processing large sample sizes N, a variety of approximate tractable methods revolving around online learning have flourished over the past decades. In the present work, we propose to use variational inference to compute a Gaussian approximation to the posterior through a single pass over the data. Our algorithm is a recursive version of variational Gaussian approximation we have called recursive variational Gaussian approximation. We start from the prior, and for each observation, we compute the nearest Gaussian approximation in the sense of Kullback–Leibler divergence to the posterior given this observation. In turn, this approximation is considered as the new prior when incorporating the next observation. This recursive version based on a sequence of optimal Gaussian approximations leads to a novel implicit update scheme which resembles the online Newton algorithm and which is shown to boil down to the Kalman filter for Bayesian linear regression. In the context of Bayesian logistic regression, the implicit scheme may be solved, and the algorithm is shown to perform better than the extended Kalman filter, while being less computationally demanding than its sampling counterparts.

PSNA: A pathwise semismooth Newton algorithm for sparse recovery with optimal local convergence and oracle properties

We propose a pathwise semismooth Newton algorithm (PSNA) for sparse recovery in high-dimensional linear models. PSNA is derived from a formulation of the KKT conditions for Lasso and Enet based on Newton derivatives. It solves the semismooth KKT equations efficiently by actively and continuously seeking the support of the regression coefficients along the solution path with warm start. At each knot in the path, PSNA converges locally superlinearly for the Enet criterion and achieves the best possible convergence rate for the Lasso criterion, i.e., PSNA converges in just one step at the cost of two matrix-vector multiplication per iteration. Under certain regularity conditions on the design matrix and the minimum magnitude of the nonzero elements of the target regression coefficients, we show that PSNA hits a solution with the same signs as the regression coefficients and achieves a sharp estimation error bound in finite steps with high probability. Extensive simulation studies support our theoretical results and indicate that PSNA is competitive with or outperforms state-of-the-art Lasso solvers in terms of efficiency and accuracy.

ALADIN‐—An open‐source MATLAB toolbox for distributed non‐convex optimization

AbstractThis article introduces an open‐source software for distributed and decentralized non‐convex optimization named ALADIN‐. ALADIN‐ is a MATLAB implementation of tailored variants of the Augmented Lagrangian Alternating Direction Inexact Newton (ALADIN) algorithm. It is user interface is convenient for rapid prototyping of non‐convex distributed optimization algorithms. An improved version of the recently proposed bi‐level variant of ALADIN is included enabling decentralized non‐convex optimization with reduced information exchange. A collection of examples from different applications fields including chemical engineering, robotics, and power systems underpins the potential of ALADIN‐.

A Krylov subspace type method for Electrical Impedance Tomography

Electrical Impedance Tomography (EIT) is a well-known imaging technique for detecting the electrical properties of an object in order to detect anomalies, such as conductive or resistive targets. More specifically, EIT has many applications in medical imaging for the detection and location of bodily tumors since it is an affordable and non-invasive method, which aims to recover the internal conductivity of a body using voltage measurements resulting from applying low frequency current at electrodes placed at its surface. Mathematically, the reconstruction of the internal conductivity is a severely ill-posed inverse problem and yields a poor quality image reconstruction. To remedy this difficulty, at least in part, we regularize and solve the nonlinear minimization problem by the aid of a Krylov subspace-type method for the linear sub problem during each iteration. In EIT, a tumor or general anomaly can be modeled as a piecewise constant perturbation of a smooth background, hence, we solve the regularized problem on a subspace of relatively small dimension by the Flexible Golub-Kahan process that provides solutions that have sparse representation. For comparison, we use a well-known modified Gauss–Newton algorithm as a benchmark. Using simulations, we demonstrate the effectiveness of the proposed method. The obtained reconstructions indicate that the Krylov subspace method is better adapted to solve the ill-posed EIT problem and results in higher resolution images and faster convergence compared to reconstructions using the modified Gauss–Newton algorithm.

Applications of Hilbert transform and Newton's root finding algorithm in delineating buried fracture from magnetic anomaly data

ABSTRACTI propose a simple and sufficiently robust data‐based semi‐automated interpretation of a total magnetic intensity profile to delineate an isolated fracture or a fractured zone embedded within a homogeneous host medium. The interpretation technique, which is based on the premise of induced magnetization, uses a model of a thin, two‐dimensional dipping sheet embedded in a homogeneous half‐space. The proposed quantitative interpretation uses a set of closed form formulae in delineating model parameters; in those the Hilbert transform, the first‐ and second‐order derivatives of the local amplitude of the zero‐order analytical signal are the essential components. The locations of zero crossings of the first‐ and second‐order derivatives of analytical signal and the Hilbert transform remain the essential keys in estimating structural parameters of the model. The magnetic susceptibility contrast is delineated from the peak amplitude of the analytical signal. An appropriately designed Savitzky–Golay derivative filter and Hilbert–Noda transformation matrix are used as robust and efficient data‐processing tools. An additional tool, Newton's root finding algorithm, is proposed for locating the zero crossings. The reconstructed data fit is validated using a statistical appraisal, and the estimated model parameters are defined with an uncertainty measure using the confidence limit. The proposed technique has been applied to synthetic data generated from a model depicting a realistic example, and also on field data from two examples, one of which closely resembles the example of synthetic data and the other as a benchmark example. Numerical experiments validate the applicability of the method.

A Lagrange–Newton algorithm for sparse nonlinear programming

The sparse nonlinear programming (SNP) problem has wide applications in signal and image processing, machine learning and finance, etc. However, the computational challenge posed by SNP has not yet been well resolved due to the nonconvex and discontinuous \(\ell _0\)-norm involved. In this paper, we resolve this numerical challenge by developing a fast Newton-type algorithm. As a theoretical cornerstone, we establish a first-order optimality condition for SNP based on the concept of strong \(\beta \)-Lagrangian stationarity via the Lagrangian function, and reformulate it as a system of nonlinear equations called the Lagrangian equations. The nonsingularity of the corresponding Jacobian is discussed, based on which the Lagrange–Newton algorithm (LNA) is then proposed. Under mild conditions, we establish the locally quadratic convergence and its iterative complexity estimation. To further demonstrate the efficiency and superiority of our proposed algorithm, we apply LNA to two specific problems arising from compressed sensing and sparse high-order portfolio selection, in which significant benefits accrue from the restricted Newton step.

Multinomial logistic regression classifier via [formula omitted]-proximal Newton algorithm

Multinomial logistic regression (MLR) is a useful tool for solving multi-classification problems. The lq,0(q⩾1) norm is an ideal regularization term for characterizing group sparsity in multinomial logistic regression and selecting important features in the high dimensional data. However, lq,0 regularized multinomial logistic regression (lq,0-MLR) is nonconvex, discontinuous, and NP-hard. Thus, most prior studies adopted a continuous approximation of the lq,0 norm. In this paper, we present a novel lq,0-proximal Newton algorithm (lq,0-PNA) to solve the lq,0-MLR. We first define a strong α-stationary point and prove that this point is a local minimizer of lq,0-MLR. We then convert such a point into a stationary equation and solve it by lq,0-PNA, which is a Newton-type method running on a group sparse subspace with a low computational cost. Furthermore, we establish a locally quadratic convergence of lq,0-PNA. Finally, numerical experiments on simulated and real data show the superiority of lq,0-PNA in terms of computational time and accuracy, when compared with six state-of-the-art solvers, especially for high dimensional data.

Semiparametric estimation for nonparametric frailty models using nonparametric maximum likelihood approach.

A consequence of using a parametric frailty model with nonparametric baseline hazard for analyzing clustered time-to-event data is that its regression coefficient estimates could be sensitive to the underlying frailty distribution. Recently, there has been a proposal for specifying both the baseline hazard and the frailty distribution nonparametrically, and estimating the unknown parameters by the maximum penalized likelihood method. Instead, in this paper, we propose the nonparametric maximum likelihood method for a general class of nonparametric frailty models, i.e. models where the frailty distribution is completely unspecified but the baseline hazard can be either parametric or nonparametric. The implementation of the estimation procedure can be based on a combination of either the Broyden-Fletcher-Goldfarb-Shanno or expectation-maximization algorithm and the constrained Newton algorithm with multiple support point inclusion. Simulation studies to investigate the performance of estimation of a regression coefficient by several different model-fitting methods were conducted. The simulation results show that our proposed regression coefficient estimator generally gives a reasonable bias reduction when the number of clusters is increased under various frailty distributions. Our proposed method is also illustrated with two data examples.