Nonlinear Conjugate Gradient Research Articles

A Hybrid Sobolev Gradient Method for Learning NODEs

The inverse problem of supervised reconstruction of depth-variable (time-dependent) parameters in ordinary differential equations is considered, with the typical application of finding weights of a neural ordinary differential equation (NODE) for a residual network with time continuous layers. The differential equation is treated as an abstract and isolated entity, termed a standalone NODE (sNODE), to facilitate for a wide range of applications. The proposed parameter reconstruction is performed by minimizing a cost functional covering a variety of loss functions and penalty terms. Regularization via penalty terms is incorporated to enhance ethical and trustworthy AI formulations. A nonlinear conjugate gradient mini-batch optimization scheme (NCG) is derived for the training having the benefit of including a sensitivity problem. The model (differential equation)-based approach is thus combined with a data-driven learning procedure. Mathematical properties are stated for the differential equation and the cost functional. The adjoint problem needed is derived together with the sensitivity problem. The sensitivity problem itself can estimate changes in the output under perturbation of the trained parameters. To preserve smoothness during the iterations, the Sobolev gradient is calculated and incorporated. Numerical results are included to validate the procedure for a NODE and synthetic datasets and compared with standard gradient approaches. For stability, using the sensitivity problem, a strategy for adversarial attacks is constructed, and it is shown that the given method with Sobolev gradients is more robust than standard approaches for parameter identification.

Regularized step directions in nonlinear conjugate gradient methods

Conjugate gradient minimization methods (CGM) and their accelerated variants are widely used. We focus on the use of cubic regularization to improve the CGM direction independent of the step length computation. In this paper, we propose the Hybrid Cubic Regularization of CGM, where regularized steps are used selectively. Using Shanno’s reformulation of CGM as a memoryless BFGS method, we derive new formulas for the regularized step direction. We show that the regularized step direction uses the same order of computational burden per iteration as its non-regularized version. Moreover, the Hybrid Cubic Regularization of CGM exhibits global convergence with fewer assumptions. In numerical experiments, the new step directions are shown to require fewer iteration counts, improve runtime, and reduce the need to reset the step direction. Overall, the Hybrid Cubic Regularization of CGM exhibits the same memoryless and matrix-free properties, while outperforming CGM as a memoryless BFGS method in iterations and runtime.

Improving Deep Neural Networks' Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum.

Momentum is crucial in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well-calibrated constant. However, tuning the hyperparameter for momentum can be a significant computational burden. In this article, we propose a novel adaptive momentum for improving DNNs training; this adaptive momentum, with no momentum-related hyperparameter required, is motivated by the nonlinear conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this new adaptive momentum eliminates the need for the momentum hyperparameter calibration, allows using a significantly larger learning rate, accelerates DNN training, and improves the final accuracy and robustness of the trained DNNs. For instance, SGD with this adaptive momentum reduces classification errors for training ResNet110 for CIFAR10 and CIFAR100 from 5.25% to 4.64% and 23.75% to 20.03%, respectively. Furthermore, SGD, with the new adaptive momentum, also benefits adversarial training and, hence, improves the adversarial robustness of the trained DNNs.

Nonlinear conjugate gradient methods for unconstrained set optimization problems whose objective functions have finite cardinality

In this paper, we propose nonlinear conjugate gradient methods for unconstrained set optimization problems in which the objective function is given by a finite number of continuously differentiable vector-valued functions. First, we provide a general algorithm for the conjugate gradient method using Wolfe line search but without imposing an explicit restriction on the conjugate parameter. Later, we study two variants of the algorithm, namely Fletcher-Reeves and conjugate descent, using two different choices of the conjugate parameter but with the same line search rule. In the general algorithm, the direction of movement at each iterate is identified by finding out a descent direction of a vector optimization problem. This vector optimization problem is identified with the help of the concept of partition set at the current iterate. As this vector optimization problem is different at different iterations, the conventional conjugate gradient method for vector optimization cannot be straightly extended to solve the set optimization problem under consideration. The well-definedness of the methods is provided. Further, we prove the Zoutendijk-type condition, which assists in proving the global convergence of the methods even without a regular condition of the stationary points. No convexity assumption is assumed on the objective function to prove the convergence. Lastly, some numerical examples are illustrated to exhibit the performance of the proposed method. We compare the performance of the proposed conjugate gradient methods with the existing steepest descent method. It is found that the proposed method commonly outperforms the existing steepest descent method for set optimization.

An improved preconditioned conjugate gradient method for unconstrained optimization problem with application in Robot arm control

AbstractThis work suggests improved conjugate gradient methods for enhancing the efficiency and robustness of the classical conjugate gradient methods. The study modifies the diagonal of the inverse Hessian approximation of the Broyden–Fletcher–Goldfarb–Shanno (BFGS) quasi‐Newton update in order to build a preconditioner for nonlinear conjugate gradient (NCG) methods applied to large‐scale unconstrained optimization problems. Damping techniques were embedded into the algorithm to impose the positive definiteness of the diagonal approximation. This made the methods easy to use and presented a viable way to increase the effectiveness of unconstrained optimization techniques. Experimental findings from a collection of benchmark problems demonstrate the efficiency and robustness of the proposed method when compared to five other existing NCG algorithms. Moreover, the successful application of the algorithm to manipulate robotic planar motion control systems with 3 degrees of freedom has been demonstrated, highlighting the practicality of the proposed approach.

The Dai–Liao-type conjugate gradient methods for solving vector optimization problems

This paper attempts to propose Dai–Liao (DL)-type nonlinear conjugate gradient (CG) methods for solving vector optimization problems. Four variants of the DL method are extended and analysed from the scalar case to the vector setting. We first give a direct vector extended version of the modified Dai–Liao (DL+) method. Then the second algorithm is presented by combining a new extension form of the well-known Hager–Zhang (HZ) parameter. This new scheme equips a good descent property and its parameter also reduces to the classical one in the scalar case. The last two algorithms extend two sufficient descent DL-type methods. Particularly, two different vector forms of their parameters are considered and compared. As a result, we reveal some differences between scalar and vector cases to a certain extent. Without any convex assumption, the sufficient descent property and global convergence of all algorithms are established under inexact line searches. Finally, preliminary numerical experiments illustrate the practical behaviour of our algorithms by using Dolan and Moré performance profile.

A New Modified Secant Condition for Non-linear Conjugate Gradient Methods with Global Convergence

A New Modified Secant Condition for Non-linear Conjugate Gradient Methods with Global Convergence

A novel hybrid conjugate gradient method for multiobjective optimization problems

In this paper, we propose a hybrid nonlinear conjugate gradient method with the strong Wolfe line search step-size rule to find the Pareto critical point of a multiobjective optimization problem. The novelty of our multiobjective conjugate gradient method (in short, MCGM) lies in that it uses a hybrid parameter β k CD − DY which is combined the CD parameter β k CD with the DY parameter β k DY introduced by Lucambio Pérez and Prudente. The global convergence of the proposed hybrid nonlinear MCGM is established under mild assumptions. Numerical experiments demonstrate the effectiveness of the proposed method.

Geothermal resources and deep tectonic in Leh Ladakh (NW Himalaya), India: Inference from magnetotelluric studies

The geothermal resources and their connectivity with the deep tectonic are meaningful to understand through their structural geometry in the NW Himalayas. Therefore, over the NW-SE profiles, 23 audio Magnetotelluric (AMT) sites and over the SW-NE profile, 15 broadband Magnetotelluric (BBMT) sites were carefully chosen to provide the detailed structure. The electrical models were produced from a 2-D inversion algorithm based on the non-linear conjugate gradient method. The Chumathang-Mahe and Puga-Sumdo regions offer excellent geothermal potential as heat/fluid passes through the Mahe Fault (MF), Zildat Fault (ZF) and Kiagor Tso Fault (KTF). These faults are well revealed in the electrical models beneath the subsurface. The Puga Valley has an excellent geothermal resource at shallow depth. A ∼1000 m thick sulfide mineral body is defined with high conductivity (∼1 Ω-m). The crustal structure shows a highly conductive mid-crust beneath the Ladakh batholiths with no manifestation roots. The signs of the Indus-Tsangpo Suture Zone (ITSZ) and Shyok Suture Zone (SSZ) are well represented in the crustal model and are associated with the steep-dipping faults. The Chumathang-Mahe and Puga-Sumdo geothermal regions are connected with low resistivity body (C1) at shallow depths in the crustal model along the SW-NE profile and offer a path for deep fluid flow. The frictional heat generated in the process of the Indian plate subducting beneath the Tibetan plateau and the collision zone between India and Asia carried to melt. The high heat surface flow indicates the thermal origin of low resistivity. Thus, a ∼10 sq. km low resistivity reservoir is identified beneath the Tso Morari dome in the crustal model that overlies a resistivity feature. The high conductivity at mid-crust determined at active tectonic fabric may be potential geothermal resources yielding high well-productivity.

A Liouville optimal control framework in prostate cancer

In this work we present a new stochastic framework for obtaining optimal treatment regimes in prostate cancer. We model the realistic scenario of randomized clinical trials for incorporating randomness related to interaction between a prostate cancer cell and androgen cell quota, due to cancer heterogeneities, across different patients in a given group, using a Liouville partial differential equation. We then solve two optimization problems: one for determining the model parameters to fit the measured data and the second to determine the optimal androgen deprivation therapy. The optimization problems are implemented using a positive, stable, and conservative finite volume solver for the Liouville equations and the projected non-linear conjugate gradient method. Several numerical results, including comparison with ordinary differential equations modeling framework, demonstrate the robustness and accuracy of our proposed framework to obtain optimal treatment regimes in real time.

On the convergence of orthogonalization-free conjugate gradient method for extreme eigenvalues of Hermitian matrices: A Riemannian optimization interpretation

In many applications, it is desired to obtain extreme eigenvalues and eigenvectors of large Hermitian matrices by efficient and compact algorithms. In particular, orthogonalization-free methods are preferred for large-scale problems for finding eigenspaces of extreme eigenvalues without explicitly computing orthogonal vectors in each iteration. For the top p eigenvalues, the simplest orthogonalization-free method is to find the best rank-p approximation to a positive semi-definite Hermitian matrix by algorithms solving the unconstrained Burer–Monteiro formulation. We show that the nonlinear conjugate gradient method for the unconstrained Burer–Monteiro formulation is equivalent to a Riemannian conjugate gradient method on a quotient manifold with the Bures–Wasserstein metric, thus its global convergence to a stationary point can be proven. Numerical tests suggest that it is efficient for computing the largest k eigenvalues for large-scale matrices if the largest k eigenvalues are nearly distributed uniformly.

A New Preconditioned Nonlinear Conjugate Gradient Method in Real Arithmetic for Computing the Ground States of Rotational Bose–Einstein Condensate

A New Preconditioned Nonlinear Conjugate Gradient Method in Real Arithmetic for Computing the Ground States of Rotational Bose–Einstein Condensate

Lithospheric electrical structure of the Alxa Block, NW China, southern central Asian Orogenic Belt, revealed by 2D inversion of magnetotelluric data

The Alxa Block, situated between the Tarim and North China cratons, is essential to understand the tectonics of northwest China. It has undergone several tectonic cycles, including rifting from northern Gondwana in Neoproterozoic, subduction-accretion of the Paleo-Asian Ocean during Paleozoic, and intracontinental deformation in Mesozoic. Broadband magnetotelluric data covering a frequency range of 320 Hz to ∼ 10000 s was collected along two NW-SE trending lines across major tectonic units of the Alxa region to investigate the deep structures of the Alxa Block. By applying the nonlinear conjugate gradient (NLCG) inversion approach, two-dimensional (2D) electrical resistivity models were obtained at a 100 km depth. These findings indicate that the Zhusileng-Hangwula and Zongnaishan-Shalazhashan tectonic zones have high resistivity in their lower crust and upper mantle, with an overall layered high-low–high resistivity pattern. The upper crust of the Nuru-Langshan tectonic zone is primarily characterized by high resistivity values, whereas the lower crust and upper mantle contain a large-scale low-resistivity (or conductive) zone that is likely the result of partial melting. The large-scale Early–Middle Jurassic nappe structures may be connected to sub-horizontal low-resistivity anomalies in the shallow strata, which are located up to 20 km below the Yagan and Zhusileng–Hangwula tectonic zones. Below the Engger Us fault, two low-resistivity bands are interpreted as the product of bidirectional subduction and final closure of the Paleo-Asian Ocean. The Alxa Block may have subducted southwards beneath the Ordos Block, as suggested by a clear low-resistivity anomaly between the two blocks. Furthermore, partial melting may have happened for high-conductivity bodies in the upper mantle based on the modified Archie formula’s evaluation of their melting degree.

Tuned Windings: the window of opportunity for the implementation of solid conductors in high frequency cryogenic electric machines

Optimization techniques can significantly improve the performance of electrical machines. The analysis and the optimization algorithms focus on computational efficiency. In this paper, the optimization of a high frequency electrical machine in cryogenic conditions is proposed. The losses introduced by AC current in a conductor are directly proportional with the applied frequency in terms of skin effect and proximity effect. The stator coil, made from two aluminium (Al) turns, in the presence of cryogenic coolants has been analysed at frequencies up to 1000 Hz. In order to reduce the AC losses introduced by the magnetic fields, Nonlinear Conjugate Gradient (NCG) method has been applied for the stator coil thickness optimization. Frequency dependence is observed in most simulations. In the presence of Liquid Hydrogen (LH2) coolant, Al 1100 conductor presents the minimum power loss.

Nonlinear Conjugate Gradient Methods for Unconstrained Test Functions through an Approximate Wolfe Line Search

Nonlinear Conjugate Gradient Methods for Unconstrained Test Functions through an Approximate Wolfe Line Search

A family of limited memory three term conjugate gradient methods

Based on several modified Hestenes–Stiefel and Polak–Ribière–Polyak nonlinear conjugate gradient methods, a family of three term limited memory CG methods are developed. When the current search direction falls into the subspace spanned by the previous m directions, the algorithm branches to optimize the objective function on the subspace by using L-BFGS method. We use this strategy to avoid potential local loops and accelerate the convergence. The proposed methods are sufficient descent. When the steplength is determined by Wolfe or Armijo line search, we establish the global convergence of the methods in a concise way. Numerical results verify the efficiency of the proposed method for the unconstrained optimization problems in the CUTEst library.

Optimal control of a semiclassical Boltzmann equation for charge transport in graphene

An ensemble optimal control problem governed by a semiclassical space-homogeneous Boltzmann equation for charge transport in graphene is formulated and analyzed.The control mechanism is an external time-dependent electric field having the purpose of driving the material density of electrons in graphene to follow a desired trajectory. For this purpose, an ensemble cost functional with quadratic H1 costs is considered.Well-posedness of the control problem is proved, and the characterization of optimal controls by an optimality system is discussed. This system is approximated by a discontinuous Galerkin scheme and solved using a nonlinear conjugate gradient method. Results of numerical experiments are presented that demonstrate the ability of the proposed control framework to design control fields.

Two Sets of High-Conductivity Systems with Different Scales Reveal the Seismogenic Mechanism of Earthquakes in the Songyuan Area, Northeastern China

To reveal the deep seismogenic environment and mechanism of earthquakes in Songyuan City, Northeastern China, 59 broadband magnetotelluric sites in the Songyuan area were arranged in this study at a spacing of 5 km. In addition, two intersecting magnetotelluric profiles, with a total of 23 measuring sites and a spacing of 2 km, were established near the Ningjiang earthquake swarm. Using a nonlinear conjugate gradient (NLCG) algorithm, resistivity structures in the lithosphere were obtained at different scales using three-dimensional (3D) inversion. The research results show that: a deep high-conductivity system (<10 Ω·m) was identified at 25–85 km depth in the lithosphere under Songyuan, corresponding closely to a region of high heat flow. It is inferred to be the molten material of mantle upwelling. In addition, a shallow high-conductivity system (<10 Ω·m) was identified beneath the Ningjiang earthquake swarm, which is interpreted to correspond to the Fuyu North fault. It is the main seismo-controlling structure of the Ningjiang earthquake swarm. The deep seismogenic environment and seismogenic mechanism of the Ningjiang earthquake swarm can be described as a deep upwelling of molten mantle material, which provides the power source. The deep magma intruded into the lower crust and accumulated, then intruded along faults and fissures, resulting in the activation of the North Fuyu fault and triggering the Ningjiang earthquake. It is attributed to the activation of shallow faults caused by the upwelling of molten mantle material.

A modified type of Fletcher-Reeves conjugate gradient method with its global convergence

<p><span>The conjugate gradient methods are one of the most important techniques used to address problems involving minimization or maximization, especially nonlinear optimization problems with no constraints at all. That is because of their simplicity and low memory needed. They can be applied in many areas, such as economics, engineering, neural networks, image restoration, machine learning, and deep learning. The convergence of Fletcher-Reeves (FR) conjugate gradient method has been established under both exact and strong Wolfe line searches. However, it is performance in practice is poor. In this paper, to get good numerical performance from the FR method, a little modification is done. The global convergence of the modified version has been established for general nonlinear functions. Preliminary numerical results show that the modified method is very efficient in terms of number of iterations and CPU time.</span></p>

A new family of hybrid conjugate gradient method for unconstrained optimization and its application to regression analysis

We know many conjugate gradient algorithms (CG) for solving unconstrained optimization problems. In this paper, based on the three famous Liu–Storey (LS), Fletcher–Reeves (FR) and Polak–Ribiére–Polyak (PRP) conjugate gradient methods, a new hybrid CG method is proposed. Furthermore, the search direction satisfies the sufficient descent condition independent of the line search. Likewise, we prove, under the strong Wolfe line search, the global convergence of the new method. In this respect, numerical experiments are performed and reported, which show that the proposed method is efficient and promising. In virtue of this, the application of the proposed method for solving regression models of COVID-19 is provided.