Descent Direction Research Articles

The deep minimizing movement scheme

Solutions of certain partial differential equations (PDEs) are often represented by the steepest descent curves of corresponding functionals. Minimizing movement scheme was developed in order to study such curves in metric spaces. Especially, Jordan-Kinderlehrer-Otto studied the Fokker-Planck equation in this way with respect to the Wasserstein metric space. In this paper, we propose a deep learning-based minimizing movement scheme for approximating the solutions of PDEs. The proposed method is highly scalable for high-dimensional problems as it is free of mesh generation. We demonstrate through various kinds of numerical examples that the proposed method accurately approximates the solutions of PDEs by finding the steepest descent direction of a functional even in high dimensions.

A bundle-type method for nonsmooth DC programs

A bundle method for minimizing the difference of convex (DC) and possibly nonsmooth functions is developed. The method may be viewed as an inexact version of the DC algorithm, where each subproblem is solved only approximately by a bundle method. We always terminate the bundle method after the first serious step. This yields a descent direction for the original objective function, and it is shown that a stepsize of at least one is accepted in this way. Using a line search, even larger stepsizes are possible. The overall method is shown to be globally convergent to critical points of DC programs. The new algorithm is tested and compared to some other solution methods on several examples and realistic applications.

Interior-point algorithm for linear programming based on a new descent direction

We present a full-Newton step feasible interior-point algorithm for linear optimization based on a new search direction. We apply a vector-valued function generated by a univariate function on a new type of transformation on the centering equations of the system which characterizes the central path. For this, we consider a new function ψ(t) = t7\4. Furthermore, we show that the algorithm finds the ϵ-optimal solution of the underlying problem in polynomial time, namely O(√nlog(n+3\7√2)) iterations. Finally, a comparative numerical study is reported in order to analyze the efficiency of the proposed algorithm.

Green non-fluorinated gradient anisotropic microcolumn array: Enables droplet transition to a low viscosity state on highly viscous substrates

Superhydrophobic surfaces with low adhesion and self-cleaning properties have broad application prospected in various fields. However, the current methods for preparing low-adhesion superhydrophobic surfaces typically require toxic chemicals or fluorination modifications, which are prone to environmental pollution and health hazards, and still face challenges for applications in industries such as food and medical care. This paper aims to prepare anisotropic gradient array surfaces (AGAS) with low adhesion on stainless steel substrates by WEDM, without the need for modification with low-surface-energy materials. AGAS can achieve a stable Cassie state with a low adhesion performance by controlling the dimensional parameters and structural inclination of columnar arrays. Experiments and numerical simulations show that owing to the anisotropic gradient structure, pinning and de-pinning occur asymmetrically in time at the contact lines of the leading and trailing edges of droplets impacting the AGAS. On AGAS, droplets can jump and roll in the direction of gradient descent during the impact process to complete self-cleaning. Due to the discontinuity of the solid–liquid–gas three-phase contact line and the stepped arrangement of the column array, the prepared surface can achieve low adhesion and self-cleaning properties for many common liquid foods in daily life, including milk and juice.

Least squares reverse time migration imaging with illumination preconditioned based on improved PRP conjugate gradients

Least squares reverse time migration (LSRTM) imaging is the one of the most accurate methods for migration imaging at present, and Polak–Ribiere–Polyak conjugate gradient (PRPCG) for LSRTM has the good numerical performance but weak convergence, so we construct an optimization factor to improve the iteration direction of the gradient, which can automatically generate a sufficient descent direction. The improved PRPCG (IPRPCG) can reduce the data residual values and speed up the iteration. And the illumination preconditioned (IP) operator is employed to IPRPCG-LSRTM which solves the problem of low resolution due to the insufficient iterative gradient information. In this paper, the experiments show that the imaging results of the proposed method (IPRPCG-IP-LSRTM) is improved greatly in detail characterization and events continuity, the iterative curve converged faster significantly, and the normalized data residual was reduced by 6.55% on average, which improved the accuracy of migration imaging effectively.

A class of projected-search methods for bound-constrained optimization

Projected-search methods for bound-constrained optimization are based on performing a search along a piecewise-linear continuous path obtained by projecting a search direction onto the feasible region. A potential benefit of a projected-search method is that many changes to the active set can be made at the cost of computing a single search direction. As the objective function is not differentiable along the search path, it is not possible to use a projected-search method with a step that satisfies the Wolfe conditions, which require the directional derivative of the objective function at a point on the path. For this reason, methods based in full or in part on a simple backtracking procedure must be used to give a step that satisfies an ‘Armijo-like’ sufficient decrease condition. As a consequence, conventional projected-search methods are unable to exploit sophisticated safeguarded polynomial interpolation techniques that have been shown to be effective for the unconstrained case. This paper describes a new framework for the development of a general class of projected-search methods for bound-constrained optimization. At each iteration, a descent direction is computed with respect to a certain extended active set. This direction is used to specify a search direction that is used in conjunction with a step length computed by a quasi-Wolfe search. The quasi-Wolfe search is designed specifically for use with a piecewise-linear search path and is similar to a conventional Wolfe line search, except that a step is accepted under a wider range of conditions. These conditions take into consideration steps at which the restriction of the objective function on the search path is not differentiable. Standard existence and convergence results associated with a conventional Wolfe line search are extended to the quasi-Wolfe case. In addition, it is shown that under a standard nondegeneracy assumption, any method within the framework will identify the optimal active set in a finite number of iterations. Computational results are given for a specific projected-search method that uses a limited-memory quasi-Newton approximation of the Hessian. The results show that, in this context, a quasi-Wolfe search is substantially more efficient and reliable than an Armijo-like search based on simple backtracking. Comparisons with a state-of-the-art bound-constrained optimization package are also presented.

A clustering heuristic to improve a derivative-free algorithm for nonsmooth optimization

In this paper we propose an heuristic to improve the performances of the recently proposed derivative-free method for nonsmooth optimization CS-DFN. The heuristic is based on a clustering-type technique to compute an estimate of Clarke’s generalized gradient of the objective function, obtained via calculation of the (approximate) directional derivative along a certain set of directions. A search direction is then calculated by applying a nonsmooth Newton-type approach. As such, this direction (as it is shown by the numerical experiments) is a good descent direction for the objective function. We report some numerical results and comparison with the original CS-DFN method to show the utility of the proposed improvement on a set of well-known test problems.

Surrogate‐Model Assisted Plausibility‐Check, Calibration, and Posterior‐Distribution Evaluation of Subsurface‐Flow Models

AbstractModern physics‐based subsurface‐flow models often require many parameters and computationally costly simulations. This prohibits traditional ensemble‐based conditioning. We expedite the calibration of such models by using surrogate/proxy models based on Gaussian Process Regression (GPR). In an iterative procedure, we use the proxy models to (a) estimate the direction of steepest descent, (b) propose only parameter combinations for full‐model runs that are likely to lead to plausible results, and (c) preselect proposed parameter combinations by their predicted performance. This method yields an ensemble of full‐model runs covering the full plausible parameter space, but at higher resolution close to the optimum. This is the basis for Markov‐Chain Monte Carlo (MCMC) simulations using GPR to estimate the posterior parameter distribution. We tested several variants of the scheme on a 3‐D variably‐saturated steady‐state subsurface‐flow model and compared it to a Neural Posterior Estimation (NPE) scheme, which requires samples of the prior distribution only. While the estimated posterior distributions of the two approaches were similar, the GPR‐based MCMC approach reproduced the data better than samples from the NPE‐based posterior distributions.

Vertical water entry of a hydrophobic sphere into waves: Numerical computations and experiments

The water entry cavity evolution and its flow structures for a sphere interacting with periodic waves are investigated numerically and experimentally. The large eddy simulation is applied in the simulation to accurately capture the turbulent flow near the surface and within the cavity of the sphere. An overset mesh-based numerical wave tank is developed, integrating an overset mesh with a method for generating regular waves, to ensure high resolution simulation of velocity fields around the water entry cavity in waves. To validate the numerical model, a physical experiment system is developed, featuring a free-falling setup and an asynchronous pulse trigger system. This experimental setup allows for precise control of the vertical water entry of a sphere at a predetermined phase of a periodic wave. The computed cavity shape and the sphere motion are in good agreement with the experimental results. Notably, the hydrodynamic forces exerted on the sphere exhibit two distinct peaks at the moment of impact and the pinch-off of the cavity, respectively. The gas-phase force acting on the dry surface of the sphere, as the cavity forms and evolves, experiences significant fluctuations along the direction of the sphere's descent. These fluctuations are caused by the accelerating gas flow prior to the pinch-off of the cavity. The changes of the hydrodynamic forces on the sphere for the cases of different water entry phase locations of waves and Froude numbers are discussed.

A Fast Temporal Decomposition Procedure for Long-Horizon Nonlinear Dynamic Programming

We propose a fast temporal decomposition procedure for solving long-horizon nonlinear dynamic programs. The core of the procedure is sequential quadratic programming (SQP) that utilizes a differentiable exact augmented Lagrangian as the merit function. Within each SQP iteration, we approximately solve the Newton system using an overlapping temporal decomposition strategy. We show that the approximate search direction is still a descent direction of the augmented Lagrangian provided the overlap size and penalty parameters are suitably chosen, which allows us to establish the global convergence. Moreover, we show that a unit step size is accepted locally for the approximate search direction and further establish a uniform, local linear convergence over stages. This local convergence rate matches the rate of the recent Schwarz scheme (Na et al. 2022). However, the Schwarz scheme has to solve nonlinear subproblems to optimality in each iteration, whereas we only perform a single Newton step instead. Numerical experiments validate our theories and demonstrate the superiority of our method. Funding: This work was supported by the National Science Foundation [Grant CNS-1545046] and the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research [Grant DE-AC02-06CH11347].

Lagrangian approach and shape gradient for inverse problem of breaking line identification in solid: contact with adhesion

A class of inverse identification problems constrained by variational inequalities is studied with respect to its shape differentiability. The specific problem appearing in failure analysis describes elastic bodies with a breaking line subject to simplified adhesive contact conditions between its faces. Based on the Lagrange multiplier approach and smooth Lavrentiev penalization, a semi-analytic formula for the shape gradient of the Lagrangian linearized on the solution is proved, which contains both primal and adjoint states. It is used for the descent direction in a gradient algorithm for identification of an optimal shape of the breaking line from boundary measurements. The theoretical result is supported by numerical simulation tests of destructive testing in 2D configuration with and without contact.

FedMDFG: Federated Learning with Multi-Gradient Descent and Fair Guidance

Fairness has been considered as a critical problem in federated learning (FL). In this work, we analyze two direct causes of unfairness in FL - an unfair direction and an improper step size when updating the model. To solve these issues, we introduce an effective way to measure fairness of the model through the cosine similarity, and then propose a federated multiple gradient descent algorithm with fair guidance (FedMDFG) to drive the model fairer. We first convert FL into a multi-objective optimization problem (MOP) and design an advanced multiple gradient descent algorithm to calculate a fair descent direction by adding a fair-driven objective to MOP. A low-communication-cost line search strategy is then designed to find a better step size for the model update. We further show the theoretical analysis on how it can enhance fairness and guarantee the convergence. Finally, extensive experiments in several FL scenarios verify that FedMDFG is robust and outperforms the SOTA FL algorithms in convergence and fairness. The source code is available at https://github.com/zibinpan/FedMDFG.

Structured BFGS Method for Optimal Doubly Stochastic Matrix Approximation

Doubly stochastic matrix plays an essential role in several areas such as statistics and machine learning. In this paper we consider the optimal approximation of a square matrix in the set of doubly stochastic matrices. A structured BFGS method is proposed to solve the dual of the primal problem. The resulting algorithm builds curvature information into the diagonal components of the true Hessian, so that it takes only additional linear cost to obtain the descent direction based on the gradient information without having to explicitly store the inverse Hessian approximation. The cost is substantially fewer than quadratic complexity of the classical BFGS algorithm. Meanwhile, a Newton-based line search method is presented for finding a suitable step size, which in practice uses the existing knowledge and takes only one iteration. The global convergence of our algorithm is established. We verify the advantages of our approach on both synthetic data and real data sets. The experimental results demonstrate that our algorithm outperforms the state-of-the-art solvers and enjoys outstanding scalability.

Gradient-based descent linesearch to solve interval-valued optimization problems under gH-differentiability with application to finance

In this article, gradient based descent line search scheme is proposed to solve interval optimization problems under generalized Hukuhara differentiability. The innovation and importance of these concepts are presented from practical and computational point of view. The necessary condition for existence of critical point is presented in inclusion form of interval-valued gradient, without transforming it into real valued function. Suitable efficient descent direction is chosen based on the monotonic property of the interval-valued function and specific interval ordering. Mathematical convergence of the scheme is proved under the assumption of Inexact line search for interval optimization problem. The theoretical developments are implemented with a set of interval test problems in different dimensions. A possible application in finance is provided and solved by our proposed scheme.

Some New Descent Nonlinear Conjugate Gradient Methods for Unconstrained Optimization Problems with Global Convergence

In this paper, we develop some three term nonlinear conjugate gradient methods based on the Hestenes–Stiefel (HS), the Polak–Ribière–Polyak (PRP) and the Liu–Storey (LS) methods. The proposed algorithms always generate sufficient descent directions which satisfy [Formula: see text]. When the Wolfe or the Armijo line search is used, we establish the global convergence of the proposed methods in a concise way. Moreover, the linear convergence rate of the methods is discussed as well. The extensive numerical results show the efficiency of the proposed methods.

Differentiable Hierarchical Optimal Transport for Robust Multi-View Learning.

Traditional multi-view learning methods often rely on two assumptions: ( i) the samples in different views are well-aligned, and ( ii) their representations obey the same distribution in a latent space. Unfortunately, these two assumptions may be questionable in practice, which limits the application of multi-view learning. In this work, we propose a differentiable hierarchical optimal transport (DHOT) method to mitigate the dependency of multi-view learning on these two assumptions. Given arbitrary two views of unaligned multi-view data, the DHOT method calculates the sliced Wasserstein distance between their latent distributions. Based on these sliced Wasserstein distances, the DHOT method further calculates the entropic optimal transport across different views and explicitly indicates the clustering structure of the views. Accordingly, the entropic optimal transport, together with the underlying sliced Wasserstein distances, leads to a hierarchical optimal transport distance defined for unaligned multi-view data, which works as the objective function of multi-view learning and leads to a bi-level optimization task. Moreover, our DHOT method treats the entropic optimal transport as a differentiable operator of model parameters. It considers the gradient of the entropic optimal transport in the backpropagation step and thus helps improve the descent direction for the model in the training phase. We demonstrate the superiority of our bi-level optimization strategy by comparing it to the traditional alternating optimization strategy. The DHOT method is applicable for both unsupervised and semi-supervised learning. Experimental results show that our DHOT method is at least comparable to state-of-the-art multi-view learning methods on both synthetic and real-world tasks, especially for challenging scenarios with unaligned multi-view data.

Improvement accuracy in deep learning: An increasing neurons distance approach with the penalty term of loss function

The increasing use of neural networks for solving complex tasks has emphasized the need to optimize their performance. In recent years, the development of neural networks has enabled them to tackle increasingly challenging problems for applications in various fields. Existing methods, such as L1 and L2 penalty terms, improve the neural network performance by reducing the neurons' weight and removing some neurons' output. However, these algorithms reduce the number of active neurons and the theoretical maximum capacity of the neural network. A loss function penalty term can improve the network performance by guiding the gradient descent direction of neurons during the training process. Moreover, it does not need to alter the network structure, increase the data, or change the training process. In this study, a novel algorithm is proposed to improve the performance of the network by adding a new loss function penalty term. The algorithm proposed in this study enhances the accuracy and anti-interference capabilities of neural networks by increasing the number of active neurons based on the finding that identical neurons extract identical features from the data, while distinct neurons extract unique features. The neurons are treated as hyperspace points and the distance between them is converted into repulsive forces to increase the distance between these points. To verify the validity of the proposed algorithm, we tested it on image classification datasets (CIFAR-100 and CALTECH-256), text datasets (SST-2 and THUCNews), and an image detection dataset (PASCAL VOC). The results show that the proposed algorithm can improve the network performance for a variety of neural network structures, data types, and task types. This algorithm has demonstrated its effectiveness in robustness against data noise and compatibility with pre-trained models such as Faster R-CNN, YOLO, and RetinaNet.

Regression analysis of data based on the method of least absolute deviations in dynamic estimation problems

The use of regression analysis in dynamic problems of system estimation requires a high-speed algorithm of model parameter determination. Moreover, the original data may have stochastic heterogeneity which entails the necessity of the estimates of model parameters be resistant to various data anomalies. However, stable estimation methods, including the least absolute deviations method, are significantly inferior to the parametric ones. The goal of the study is to describe a computationally efficient algorithm for implementing the method of least absolute deviations for dynamic estimation of regression models and to study its capabilities for solving practical problems. This algorithm is based on descending along nodal lines. In this case, instead of the values of the objective function, its derivative in the direction of descent is considered. The computational complexity of the algorithm is also reduced due to the use of the solution of the problem at the previous step as a starting point and efficient updating of observations in the current data sample. The external performance of the proposed dynamic version of the algorithm of gradient descent along nodal lines has been compared with the static version and with the least squares method. It is shown that the dynamic version of the algorithm of gradient descent along the nodal lines make it possible to bring the speed close to that of the least squares method for common practical situations and to use the proposed version in dynamic estimation problems for a wide class of systems.

Dual-scale similarity-guided cycle generative adversarial network for unsupervised low-dose CT denoising

Removing the noise in low-dose CT (LDCT) is crucial to improving the diagnostic quality. Previously, many supervised or unsupervised deep learning-based LDCT denoising algorithms have been proposed. Unsupervised LDCT denoising algorithms are more practical than supervised ones since they do not need paired samples. However, unsupervised LDCT denoising algorithms are rarely used clinically due to their unsatisfactory denoising ability. In unsupervised LDCT denoising, the lack of paired samples makes the direction of gradient descent full of uncertainty. On the contrary, paired samples used in supervised denoising allow the parameters of networks to have a clear direction of gradient descent. To bridge the gap in performance between unsupervised and supervised LDCT denoising, we propose dual-scale similarity-guided cycle generative adversarial network (DSC-GAN). DSC-GAN uses similarity-based pseudo-pairing to better accomplish unsupervised LDCT denoising. We design a Vision Transformer-based global similarity descriptor and a residual neural network-based local similarity descriptor for DSC-GAN to effectively describe the similarity between two samples. During training, pseudo-pairs, i.e., similar LDCT samples and normal-dose CT (NDCT) samples, dominate parameter updates. Thus, the training can achieve equivalent effect as training with paired samples. Experiments on two datasets demonstrate that DSC-GAN beats the state-of-the-art unsupervised algorithms and reaches a level close to supervised LDCT denoising algorithms.

A Family of Multi-Step Subgradient Minimization Methods

For solving non-smooth multidimensional optimization problems, we present a family of relaxation subgradient methods (RSMs) with a built-in algorithm for finding the descent direction that forms an acute angle with all subgradients in the neighborhood of the current minimum. Minimizing the function along the opposite direction (with a minus sign) enables the algorithm to go beyond the neighborhood of the current minimum. The family of algorithms for finding the descent direction is based on solving systems of inequalities. The finite convergence of the algorithms on separable bounded sets is proved. Algorithms for solving systems of inequalities are used to organize the RSM family. On quadratic functions, the methods of the RSM family are equivalent to the conjugate gradient method (CGM). The methods are intended for solving high-dimensional problems and are studied theoretically and numerically. Examples of solving convex and non-convex smooth and non-smooth problems of large dimensions are given.