Newton Gradient Research Articles

Speeding up the Training of Neural Networks with the One-Step Procedure

In the last decade, research and corporate have shown a dramatically growing interest in the field of machine learning, mostly due to the performances of deep neural networks. These increasingly complex architectures solved a wide range of problems. However, training these sophisticated architectures require many computation on advanced hardware. With this paper, we introduce a new approach based on the One-Step procedure that may fasten their training. In this procedure, an initial guess estimator is computed on a subsample that is then improved with only one step of the Newton gradient descent on the whole dataset. To show the efficiency of this framework, we consider regression and classification tasks using simulated and real datasets. We consider classic architectures, namely multi-layer perceptrons and show, on our examples, that the One-Step procedure is often halving the computation time to train the neural networks while preserving the performances.

Small-disturbance input-to-state stability of perturbed gradient flows: Applications to LQR problem

This paper studies the effect of perturbations on the gradient flow of a general nonlinear programming problem, where the perturbation may arise from inaccurate gradient estimation in the setting of data-driven optimization. Under suitable conditions on the objective function, the perturbed gradient flow is shown to be small-disturbance input-to-state stable (ISS), which implies that, in the presence of a small-enough perturbation, the trajectories of the perturbed gradient flow must eventually enter a small neighborhood of the optimum. This work was motivated by the question of robustness of direct methods for the linear quadratic regulator problem, and specifically the analysis of the effect of perturbations caused by gradient estimation or round-off errors in policy optimization. We show small-disturbance ISS for three of the most common optimization algorithms: standard gradient flow, natural gradient flow, and Newton gradient flow.

Determined Reverberant Blind Source Separation of Audio Mixing Signals

Audio signal separation is an open and challenging issue in the classical “Cocktail Party Problem”. Especially in a reverberation environment, the separation of mixed signals is more difficult separated due to the influence of reverberation and echo. To solve the problem, we propose a determined reverberant blind source separation algorithm. The main innovation of the algorithm focuses on the estimation of the mixing matrix. A new cost function is built to obtain the accurate demixing matrix, which shows the gap between the prediction and the actual data. Then, the update rule of the demixing matrix is derived using Newton gradient descent method. The identity matrix is employed as the initial demixing matrix for avoiding local optima problem. Through the real-time iterative update of the demixing matrix, frequency-domain sources are obtained. Then, time-domain sources can be obtained using an inverse short-time Fourier transform. Experimental results based on a series of source separation of speech and music mixing signals demonstrate that the proposed algorithm achieves better separation performance than the state-of-the-art methods. In particular, it has much better superiority in the highly reverberant environment.

Static and incremental robust kernel factorization embedding graph regularization supporting ill-conditioned industrial data recovery

Low-rank approximation algorithms aim to utilize convex nuclear norm constraint of linear matrices to recover ill-conditioned entries caused by multi-sampling rates, sensor drop-out. However, these existing algorithms are often limited in solving high-dimensionality and rank minimization relaxation. In this paper, a robust kernel factorization embedding graph regularization method is developed to statically impute missing measurements. Specifically, the implicit high-dimensional feature space of ill-conditioned data is factorized by kernel sparse dictionary. Then, a robust sparse-norm and graph regularization constraints are performed in the objective function to ensure the consistency of the spatial information. For the optimization of the parameters involved in the model, a distributed adaptive proximal Newton gradient descent learning strategy is proposed to accelerate the convergence. Furthermore, considering the dynamic time-series and potentially non-stationary structure of industrial data, we propose extended incremental versions to alleviate the complexity of the overall model computation. Extensive data recovery experiments are conducted on two real industrial processes to evaluate the proposed method in comparison with existing state-of-the-art restorers. The results show that the proposed methods can impute better with different missing rates and have strong competitiveness in practical application.

Development of an Engine Calibration Model Using Gaussian Process Regression

To enhance the calibration efficiency, reduce the fuel consumption and improve the emission performance of the engine, a calibration method using Gaussian Process Regression (GPR) is proposed in this work. First, the design of experiment (DoE) is constructed by using the Space-filling method, and the engine bench sampling test is implemented according to results of DoE. Then, the square exponential covariance function is selected through the comparison of four covariance functions, and the corresponding hyper-parameters are optimized by using Newton gradient algorithm. Finally, the GPR model of the engine is established and its calibration performance is validated by the experimental data. The comparison shows that the performance of the developed GPR model is superior to the Polynomial model and Neural Network model, whose coefficient of determination (R2) of Fuel Consumption (FC), NOx emission (NOx) and Soot emission (Soot) are up to 0.9980, 0.9326 and 0.9247. The case study demonstrates that the virtual calibration optimization based on GPR model improves the fuel consumption performance greatly, while taking NOx and Soot emission indicators into account.

A Cauchy score DOA estimator for monostatic MIMO radar in impulsive noise environment

ABSTRACT It is interesting to determine the direction of arrival (DOA) of signals in the presence of impulsive noise for monostatic multiple-input multiple-output (MIMO) radar. Inspired by the robust statistics property of M-estimation theory and maximum likelihood estimation theory, a Cauchy score cost function is proposed to suppress the outliers, leading to robust DOA estimation. Firstly, the Cauchy score cost function for residual fitting error matrix of the radar data is derived. Then, an alternating convex optimisation algorithm based on the complex-valued Newton gradient descent is tailored to acquire an efficient solution for the Cauchy score cost function. Finally, using the signal subspace matrix obtained by the alternating convex optimisation algorithm, ESPRIT algorithm is adopted to provide a closed-form DOA estimations. Simulation results are presented to demonstrate the effectiveness of the proposed method.

Timescale Separation in Autonomous Optimization

Autonomous optimization refers to the design of feedback controllers that steer a physical system to a steady state that solves a predefined, possibly constrained, optimization problem. As such, no exogenous control inputs such as set points or trajectories are required. Instead, these controllers are modeled after optimization algorithms that take the form of dynamical systems. The interconnection of this type of optimization dynamics with a physical system is however not guaranteed to be stable unless both dynamics act on sufficiently different timescales. In this paper, we quantify the required timescale separation and give prescriptions that can be directly used in the design of this type of feedback controllers. Using ideas from singular perturbation analysis, we derive stability bounds for different feedback laws that are based on common continuous-time optimization schemes. In particular, we consider gradient descent and its variations, including projected gradient, and Newton gradient. We further give stability bounds for momentum methods and saddle-point flows. Finally, we discuss how optimization algorithms such as subgradient and accelerated gradient descent, while well-behaved in offline settings, are unsuitable for autonomous optimization due to their general lack of robustness.

Reply to comment by S. Vatankhah and R.A. Renaut: “Three‐dimensional potential field data inversion with L0 quasinorm sparse constraints”

The comment on “Three-dimensional potential field data inversion with L0 quasi-norm sparse constraints” by Vatankhah mainly gives some advice on my paper. Vatankhah pays attention on the L0 norm, which is applied on the potential field data inversion. Last and Kubik (1983) firstly introduced the compact inversion of gravity data. Later, other inversion studies were based on this principle (e.g.: Portniaguine and Zhdanov 1999). The strategy to obtain compact models is similar to the L0 norm constraint. However, they are not same. In my paper, I used the L0 norm which was firstly introduced in the compressive sensing theory. Compressive sensing is the reconstruction of sparse images or signals from very few samples, by means of solving a tractable optimization problem (Baraniuk 2008; Chartrand 2009). It is a great breakthrough. Later, this principle has been applied in many domains (e.g., Mohimani, Babaie-Zadeh and Jutten 2009; Zhang and Tian 2017). In these researches, an approximate zero norm replaces the L0 norm. In Vatankhah’s comment, there may be some inaccurate understanding. Function f 2 σ (m) (equation (3) in Vatankhah’s comment, corresponding to the function used in Meng (2018)) is better suited for potential field data inversion comparing with f 1 σ (m) (equation (1) in Vatankhah’s comment, introduced by Last and Kubik, 1983). In Vatankhah’s Figure 1(a), for large values of σ and for big absolute values of m, f 2 σ (m) has smaller values than f 1 σ (m), and therefore f 2 σ (m) is smoother than f 1 σ (m). For large σ more weight is imposed on the large elements of the parameter vector m in f 1 σ (m), as compared with f 2 σ (m), yielding a larger penalty on these large elements of m. Thus, the f 2 σ (m) can provide smoother solutions. This phenomenon demonstrates that noise has less effect on f 2 σ (m) than on f 1 σ (m), both used to approximate the L0 norm. Therefore, the f 2 σ (m) function has better performance with big values of σ. Even when σ = 0.1 (Vatankhah’s Fig. 1(b)), f 2 σ (m) have smaller values than f 1 σ (m) for absolute values of m lesser than 0.2, and bigger values than f 1 σ (m) for absolute values ofmbetween 0.2 and 0.6. Therefore, equation (3) can give sparser and better inversion results. Vatankhah’s Figure 1 shows this phenomenon clearly. In the potential field data inversion, the depth-weighting function and density constraint function are very important and commonly used. I think that the researchers can focus on the solution on the L0 norm. For example, the Newton gradient method is applied in my paper, which can effectively avoid the calculations of step size. I think that this reply can help the readers to have a better understanding of this topic.

Complex Factor Analysis and Extensions

Many subspace-based array signal processing algorithms assume that the noise is spatially white. In this case, the noise covariance matrix is a multiple of the identity and the eigenvectors of the data covariance matrix are not affected by it. If the noise covariance is an unknown arbitrary diagonal (e.g., for an uncalibrated array), the eigenvalue decomposition leads to incorrect subspace estimates and it has to be replaced by a more general “factor analysis” decomposition (FA), which then reveals all relevant information. We consider this data model and several extensions where the noise covariance matrix has a more general structure, such as banded, sparse, block diagonal, and cases, where we have multiple data covariance matrices that share the same noise covariance matrix. Starting from a nonlinear weighted least squares formulation, we propose new estimation algorithms for both classical FA and its extensions. The optimization is based on Gauss–Newton gradient descent. Generally, this leads to an iteration involving the inversion of a very large matrix. Using the structure of the problem, we show how this can be reduced to the inversion of a matrix with dimension equal to the number of unknown noise covariance parameters. This results in new algorithms that have faster numerical convergence and lower complexity compared to several maximum-likelihood based algorithms that could be considered state of the art. The new algorithms scale well to large dimensions and can replace eigenvalue decompositions in many applications even if the noise can be assumed to be white.

Low Complexity Moving Target Parameter Estimation for MIMO Radar Using 2D-FFT

In multiple-input multiple-output radar, to localize a target and estimate its reflection coefficient, a given cost function is usually optimized over a grid of points. The performance of such algorithms is directly affected by the grid resolution. Increasing the number of grid points enhances the resolution of the estimator but also increases its computational complexity exponentially. In this paper, two reduced complexity algorithms are derived based on Capon and amplitude and phase estimation (APES) to estimate the reflection coefficient, angular location, and Doppler shift of multiple moving targets. By exploiting the structure of the terms, the cost function is brought into a form that allows us to apply the two-dimensional fast Fourier transform (2D-FFT) and reduce the computational complexity of estimation. Using low resolution 2D-FFT, the proposed algorithm identifies suboptimal estimates and feeds them as initial points to the derived Newton gradient algorithm. In contrast to the grid-based search algorithms, the proposed algorithm can optimally estimate on- and off-the-grid targets in very low computational complexity. A new APES cost function with better estimation performance is also discussed. Generalized expressions of the Cramer–Rao lower bound are derived to assess the performance of the proposed algorithm.

SU-E-T-623: Utilizing a Hybrid Optimizer to Improve Dose Conformity during IMRT Planning.

For IMRT treatment planning, an index-dose based algorithm features a fast approach in optimizing beam shapes and weights, and the quasi-Newton method is adopted in segment weight optimization by many commercial products. By combining these two optimizers, we aim to improve IMRT plan quality by achieving better normal tissue sparing. An IMRT plan was generated using an in-house treatment planning system in three steps: 1) optimize fluence using beamlet intensity modulation, 2) generate Multi-collimator leaf sequence and segment weights, 3) tune the segment shapes and weights as each segment treated as a single beam. A quick converge was achieved by the optimizer implementing the index-dose concept in step (1 and 3). To further improve the plan's quality, we optimized the segment weights via a quasi-Newton gradient search method where a convex objective function was constructed using both existing segment shapes and dose constraints defined by the planner. Thus, the segment shapes were optimized with index-dose, the shapes optimized with quasi-Newton, alternately. The new approach was evaluated with patient cases including prostate and head & neck. Both plans had equivalent tumor dose coverage. For the prostate case, the rectal dose was reduced by 6% for V60% and 2% for V10cc, respectively. For the head and neck, better sparing was observed for the spinal cord, the left parotid, and the larynx. Combining index-dose and quasi- newton gradient search can effectively improve sparing of normal tissues without sacrificing target dose coverage. This work indicates the potential of improving treatment plan quality by integrating different optimization methods.

SU‐E‐T‐881: Simulation of a Multileaf Collimator vs. Iris Variable Aperture Collimation for Prostate Robotic Radiosurgery and IMRT

Purpose: To study potential advantages and disadvantages of a multileaf collimator (MLC) relative to iris variable aperture collimation for prostate cancer treatment with a CyberKnife robotic radiosurgery system. Methods: Six treatment plans for three prostate cases are generated using target conformal segments with a hypothetical MLC vs. circular beams with an iris variable aperture collimator. Conformality, homogeneity, number of MUs and segments are used as plan quality indicators. Quasi Newton Gradient descent optimization is applied as optimization strategy. MU reduction during the optimization process is achieved with two‐dimensional median fluence smoothing. The final sequencing is performed at sequencing levels 2 and 3 to reduce the number of segments. Iris baseline plans were generated using the vendorˈs TPS. Treatment plans A and B are conventionally fractionated IMRT with homogeneous PTV coverage. Plans C and D are IMRT with homogeneous PTV coverage (−/+ integrated boosts). Plans E and F are hyperfractionated SBRT with homogeneous PTV coverage (−/+ integrated boosts, and stricter OAR dose constraints). Results: With identical number of CyberKnife nodes for an iris and MLC treatment plan, an improvement in dose conformality is achieved with a MLC for all treatment plans and dose homogeneity is only slightly degraded. In all cases, the total number of MUs with multileaf collimated fields is reduced in comparison to circular collimated iris variable aperture fields (avg. reduction 28%). The number of segments for a comparable MLC treatment plan with an iris plan is also reduced for 4 cases and slightly increased for 2 of them (avg. reduction 17%). Conclusions: This investigation shows that a MLC could provide several benefits for intensity modulated prostate radiosurgery with a CyberKnife robotic radiosurgy system. Multileaf collimated beams for radiosurgery allow a highly conformal radiation. Their target conformal segments in addition reduce the required total number of MUs and segments.

Blind Adaptive Equalization of Complex Signals based on the Constant Modulus Algorithm

The paper discuss, applicability of the second-order Newton gradient descent method for blind equalization of complex signals based on the Constant Modulus Algorithm (CMA). The Constant Modulus (CM) loss function is real with complex valued arguments, and, hence, non-analytic. The Hessian for noiseless FIR channels and rederive the known fact that the full Hessian of the CM loss function is always singular in a simpler manner. The channel model shows that the perfectly equalizing solutions are stationary points of the CM loss function and also evaluate the bit error rate. The paper also discuss of the full Newton method. Finally, to validate the proposed algorithm, simulation studies have been carried out and results are presented and compared. The simulation results show the effectiveness of the proposed algorithm. General Terms Mobile communications, FIR channels.

On Refining Polynomial Phase Signal Parameter Estimates

Newton algorithms are commonly used in the final “refinement” stage of parameter estimation for sinusoids and higher order phase polynomial signals. Such parameter estimation scenarios arise commonly in radar applications where the radial target velocity (which is possibly time-varying) must be estimated. The author and coworkers have previously proposed an elegant and efficient alternative to a Newton gradient search type algorithm based on filtering and phase unwrapping. A statistical and computational analysis of this filtering/phase unwrapping method is presented here. The analysis shows the algorithm to be computationally efficient and much less sensitive to the accuracy of the initial guesses for the parameters than a Newton algorithm. A first-order statistical analysis of the filtering/phase unwrapping algorithm is performed, and guidelines are derived for the required accuracy of the initial estimates. Simulations are presented to confirm the analysis.

Use of the Newton Method for Blind Adaptive Equalization Based on the Constant Modulus Algorithm

We study the applicability of the second-order Newton gradient descent method for blind equalization of complex signals based on the constant modulus algorithm (CMA). The constant modulus (CM) loss function is real with complex valued arguments, and, hence, nonanalytic. We, therefore, use the framework of the Wirtinger calculus to derive a useful and insightful form of the Hessian for noiseless FIR channels and rederive the known fact that the full Hessian of the CM loss function is always singular in a simpler manner. For the implementation of a suboptimum version of Newton algorithm, we give the conditions under which the leading partial Hessian is nonsingular for a noiseless FIR channel model. For this channel model, we show that the perfectly equalizing solutions are stationary points of the CM loss function and also evaluate the leading partial Hessian and the full Hessian at a perfectly equalizing solution. We also discuss regularization of the full Newton method. Finally, some simulation results are given.

Application of second‐order adjoint technique for conduit flow problem

AbstractThis paper presents the way to obtain the Newton gradient by using a traction given by the perturbation for the Lagrange multiplier. Conventionally, the second‐order adjoint model using the Hessian/vector products expressed by the product of the Hessian matrix and the perturbation of the design variables has been researched (Comput. Optim. Appl. 1995; 4:241–262). However, in case that the boundary value would like to be obtained, this model cannot be applied directly. Therefore, the conventional second‐order adjoint technique is extended to the boundary value determination problem and the second‐order adjoint technique is applied to the conduit flow problem in this paper. As the minimization technique, the Newton‐based method is employed. The Broyden–Fletcher–Goldfarb–Shanno (BFGS) method is applied to calculate the Hessian matrix which is used in the Newton‐based method and a traction given by the perturbation for the Lagrange multiplier is used in the BFGS method. Copyright © 2007 John Wiley & Sons, Ltd.

NUMERICAL SOLUTIONS OF A VECTOR GINZBURG–LANDAU EQUATION WITH A TRIPLE-WELL POTENTIAL

We numerically compute solutions to the vector Ginzburg–Landau equation with a triple-well potential. We use the Galerkin Newton Gradient Algorithm of Neuberger and Swift and bifurcation techniques to find solutions. With a small parameter, we find a Morse index 2 triple junction solution. This is the solution for which Flores, Padilla and Tonegawa gave an existence proof. We classify all of the solutions guaranteed to exist by the Equivariant Branching Lemma at the first bifurcation points of the trivial solutions. Guided by the symmetry analysis, we numerically compute the solution branches.

Absence of multiple local minima effects in intensity modulated optimization with dose–volume constraints

This paper reports on the analysis of intensity modulated radiation treatment optimization problems in the presence of non-convex feasible parameter spaces caused by the specification of dose–volume constraints for the organs-at-risk (OARs). The main aim was to determine whether the presence of those non-convex spaces affects the optimization of clinical cases in any significant way. This was done in two phases: (1) Using a carefully designed two-dimensional mathematical phantom that exhibits two controllable minima and with randomly initialized beamlet weights, we developed a methodology for exploring the nature of the convergence characteristics of quadratic cost function optimizations (deterministic or stochastic). The methodology is based on observing the statistical behaviour of the residual cost at the end of optimizations in which the stopping criterion is progressively more demanding and carrying out those optimizations to very small error changes per iteration. (2) Seven clinical cases were then analysed with dose–volume constraints that are stronger than originally used in the clinic. The clinical cases are two prostate cases differently posed, a meningioma case, two head-and-neck cases, a spleen case and a spine case. Of the 14 different sets of optimizations (with and without the specification of maximum doses allowed for the OARs), 12 fail to show any effect due to the existence of non-convex feasible spaces. The remaining two sets of optimizations show evidence of multiple minima in the solutions, but those minima are very close to each other in cost and the resulting treatment plans are practically identical, as measured by the quality of the dose–volume histograms (DVHs). We discuss the differences between fluence maps resulting from those similar treatment plans. We provide a possible reason for the observed results and conclude that, although the study is necessarily limited, the annealing characteristics of a simulated annealing method may not be justified in clinical optimization in the presence of dose–volume constraints. The results of optimizations by the Newton gradient (NG) method with a quadratic cost function are reported in detail. An adaptive simulated annealing method, optimizing the same function, and the dynamically penalized likelihood method, optimizing a log likelihood function, have also been used in the study. The results of the latter two methods have only been discussed briefly, as they yielded the same conclusions as the NG method.

Comparative behaviour of the Dynamically Penalized Likelihood algorithm in inverse radiation therapy planning

This paper presents a description of tests carried out to compare the behaviour of five algorithms in inverse radiation therapy planning: (1) The Dynamically Penalized Likelihood (DPL), an algorithm based on statistical estimation theory; (2) an accelerated version of the same algorithm; (3) a new fast adaptive simulated annealing (ASA) algorithm; (4) a conjugate gradient method; and (5) a Newton gradient method. A three-dimensional mathematical phantom and two clinical cases have been studied in detail. The phantom consisted of a U-shaped tumour with a partially enclosed ‘spinal cord’. The clinical examples were a cavernous sinus meningioma and a prostate case. The algorithms have been tested in carefully selected and controlled conditions so as to ensure fairness in the assessment of results. It has been found that all five methods can yield relatively similar optimizations, except when a very demanding optimization is carried out. For the easier cases, the differences are principally in robustness, ease of use and optimization speed. In the more demanding case, there are significant differences in the resulting dose distributions. The accelerated DPL emerges as possibly the algorithm of choice for clinical practice. An appendix describes the differences in behaviour between the new ASA method and the one based on a patent by the Nomos Corporation.

A high performance neural network for solving nonlinear programming problems with hybrid constraints

A continuous neural network is proposed in this Letter for solving optimization problems. It not only can solve nonlinear programming problems with the constraints of equality and inequality, but also has a higher performance. The main advantage of the network is that it is an extension of Newton's gradient method for constrained problems, the dynamic behavior of the network under special constraints and the convergence rate can be investigated. Furthermore, the proposed network is simpler than the existing networks even for solving positive definite quadratic programming problems. The network considered is constrained by a projection operator on a convex set. The advanced performance of the proposed network is demonstrated by means of simulation of several numerical examples.