Descent Direction Research Articles

An Explicit Three-Term Polak–Ribière–Polyak Conjugate Gradient Method for Bicriteria Optimization

A three-term Polak–Ribière–Polyak conjugate gradient-like method for bicriteria optimization without scalarization is proposed in this paper. Three advantages are to be noted. First, the descent directions are given explicitly and can then be directly computed. Second, the descent property turns out to be sufficient and independent of the line search. Third, without Lipschitzian hypotheses, global convergence towards Pareto stationary points is proved under an Armijo type condition. Numerical experiments including comparisons with other methods are also reported.

Compressed sensing with smooth L0 constraints for moving force identification from bridge response measurements

Abstract Compressed sensing (CS), as an emerging information sampling technique, has been successfully applied in the field of moving force identification (MFI). However, existing MFI CS models often fail to obtain the optimal sparse solutions and frequently underestimate the amplitude of local impact forces. To effectively address this issue, a new CS method is proposed for MFI based on smooth L0 norm constraints and bridge response measurements. Firstly, a smooth function is used to approximate the L0 norm, establishing a noise CS reconstruction model for MFI. The introduction of the smoothing function can locally convexify the original MFI problem and enhance the smoothness and differentiability of the objective function, making the optimization problem easier to solve. Subsequently, the Polak–Ribiere–Polyak formula is adopted to point the descent direction of the new objective function, and the sparse solution is iteratively advanced through the conjugate gradient algorithm. Finally, the applicability and feasibility of the proposed method is confirmed by numerical simulations and vehicle–bridge interaction tests, respectively. The results show that the proposed method can accurately identify moving forces from limited measurements of bridge responses. Compared with existing methods, it can provide more precise sparse solutions with higher robustness to measurement noises, and address the issue of underestimating on the amplitude of local impact forces, which is expected to enhance the performance and in-situ applicability of MFI.

A Lagrange barrier approach for the minimum concave cost supply problem via a logarithmic descent direction algorithm

The minimisation of concave costs in the supply chain presents a challenging non-deterministic polynomial (NP) optimisation problem, widely applicable in industrial and management engineering. To approximate solutions to this problem, we propose a logarithmic descent direction algorithm (LDDA) that utilises the Lagrange logarithmic barrier function. As the barrier variable decreases from a high positive value to zero, the algorithm is capable of tracking the minimal track of the logarithmic barrier function, thereby obtaining top-quality solutions. The Lagrange function is utilised to handle linear equality constraints, whilst the logarithmic barrier function compels the solution towards the global or near-global optimum. Within this concave cost supply model, a logarithmic descent direction is constructed, and an iterative optimisation process for the algorithm is proposed. A corresponding Lyapunov function naturally emerges from this descent direction, thus ensuring convergence of the proposed algorithm. Numerical results demonstrate the effectiveness of the algorithm.

Computing Euclidean distance and maximum likelihood retraction maps for constrained optimization

Riemannian optimization uses local methods to solve optimization problems whose constraint set is a smooth manifold. A linear step along some descent direction usually leaves the constraints, and hence retraction maps are used to approximate the exponential map and return to the manifold. For many common matrix manifolds, retraction maps are available, with more or less explicit formulas. For implicitly-defined manifolds, suitable retraction maps are difficult to compute. We therefore develop an algorithm which uses homotopy continuation to compute the Euclidean distance retraction for any implicitly-defined submanifold of Rn, and prove convergence results.We also consider statistical models as Riemannian submanifolds of the probability simplex with the Fisher metric. Replacing Euclidean distance with maximum likelihood results in a map which we prove is a retraction. In fact, we prove the retraction is second-order; with the Levi-Civita connection associated to the Fisher metric, it approximates geodesics to second-order accuracy.

Level-set based shape optimization for plane elastic structures using radial basis functions and Hilbertian descent direction

Structural optimization problems are often associated with the so-called shape functionals depending on a shape through its geometry and the state being a solution of given partial differential equation. In such a framework it is convenient to work with the gradient-like method based on a concept of a shape derivative and level set method. The key idea of level set method is to represent the structural boundary with zero level set of given function (level set function—LSF). Now, changing the shape of a structure under optimization is equivalent to transport the LSF in such a direction that ensures decreasing the value of the objective functional. To this end, we make use of coercive bilinear form taken from the weak formulation of elasticity problem to obtain descent direction at each iteration. This descent direction is a solution of an additional variational problem, involving the bilinear form mentioned above and the volumetric expression of the shape derivative plays the role of a linear form. In this paper, we combine level set method with radial basis functions (RBFs) used to approximate LSF. We focus on the so-called multiquadric RBFs, but other classes of RBFs are also briefly considered. This eventually leads to transformation of partial differential equation (linear transport equation governing the evolution of shapes) to a system of linear ordinary differential equations which admits analytical formula for the solution. We apply our method to compliance minimization of a cantilever problem as well as to total potential energy minimization of a structure with kinematic loading. To run all the numerical experiments, we wrote our own code in Wolfram Mathematica environment.

Solving Multi-Objective Optimal Control Problems Using a Multiresolution Approach

This paper presents an adaptive multiresolution strategy for multi-objective optimal control problems. The optimal control problem is solved using a direct approach, with individualistic grid adaptation facilitated by a local error analysis at element boundaries. Multiple objectives are considered using a dominance-based approach applying both local and global search methods to a collaborative population of unique solutions. These aspects are simultaneously incorporated via a novel application of evolutionary algorithms for adaptive optimal control problems. Together, this avoids the need for a priori specification of the quantity and temporal location of element boundaries and the set of scalarization weights defining the multi-objective descent directions. Solution fidelity can thus increase concurrently with the exploration of the design space, which leads to increased numerical efficiency while propagating and maintaining population diversity. The benefits of the proposed approach over traditional uniform-grid implementations are demonstrated. Results show that the multiresolution approach is capable of striking an effective balance between solution fidelity, population diversity, and computational cost unachievable using uniform grids.

A Descent Method for Nonsmooth Multiobjective Optimization in Hilbert Spaces

The efficient optimization method for locally Lipschitz continuous multiobjective optimization problems from Gebken and Peitz (J Optim Theory Appl 188:696–723, 2021) is extended from finite-dimensional problems to general Hilbert spaces. The method iteratively computes Pareto critical points, where in each iteration, an approximation of the Clarke subdifferential is computed in an efficient manner and then used to compute a common descent direction for all objective functions. To prove convergence, we present some new optimality results for nonsmooth multiobjective optimization problems in Hilbert spaces. Using these, we can show that every accumulation point of the sequence generated by our algorithm is Pareto critical under common assumptions. Computational efficiency for finding Pareto critical points is numerically demonstrated for multiobjective optimal control of an obstacle problem.

Solving Mathematical Programs with Complementarity Constraints Arising in Nonsmooth Optimal Control

AbstractThis paper examines solution methods for mathematical programs with complementarity constraints (MPCC) obtained from the time-discretization of optimal control problems (OCPs) subject to nonsmooth dynamical systems. The MPCC theory and stationarity concepts are reviewed and summarized. The focus is on relaxation-based methods for MPCCs, which solve a (finite) sequence of more regular nonlinear programs (NLP), where a regularization/homotopy parameter is driven to zero. Such methods perform reasonably well on currently available benchmarks. However, these results do not always generalize to MPCCs obtained from nonsmooth OCPs. To provide a more complete picture, this paper introduces a novel benchmark collection of such problems, which we call . The problem set includes 603 different MPCCs and we split it into a few representative subsets to accelerate the testing. We compare different relaxation-based methods, NLP solvers, homotopy parameter update and relaxation parameter steering strategies. Moreover, we check whether the obtained stationary points allow first-order descent directions, which may be the case for some of the weaker MPCC stationarity concepts. In the best case, the Scholtes’ relaxation (SIAM J. Optim. 11, 918–936, 2001) with (Math. Program. 106, 25–57, 2006) as NLP solver manages to solve 73.8% of the problems. This highlights the need for further improvements in algorithms and software for MPCCs.

Nonlinear conjugate gradient methods for unconstrained set optimization problems whose objective functions have finite cardinality

In this paper, we propose nonlinear conjugate gradient methods for unconstrained set optimization problems in which the objective function is given by a finite number of continuously differentiable vector-valued functions. First, we provide a general algorithm for the conjugate gradient method using Wolfe line search but without imposing an explicit restriction on the conjugate parameter. Later, we study two variants of the algorithm, namely Fletcher-Reeves and conjugate descent, using two different choices of the conjugate parameter but with the same line search rule. In the general algorithm, the direction of movement at each iterate is identified by finding out a descent direction of a vector optimization problem. This vector optimization problem is identified with the help of the concept of partition set at the current iterate. As this vector optimization problem is different at different iterations, the conventional conjugate gradient method for vector optimization cannot be straightly extended to solve the set optimization problem under consideration. The well-definedness of the methods is provided. Further, we prove the Zoutendijk-type condition, which assists in proving the global convergence of the methods even without a regular condition of the stationary points. No convexity assumption is assumed on the objective function to prove the convergence. Lastly, some numerical examples are illustrated to exhibit the performance of the proposed method. We compare the performance of the proposed conjugate gradient methods with the existing steepest descent method. It is found that the proposed method commonly outperforms the existing steepest descent method for set optimization.

A Line-Search-Based Algorithm for Multiphase Flash Calculations with CO2-Hydrocarbon System.

Multiphase flash calculations are pivotal in compositional simulation, necessitating a robust and efficient computational algorithm. In this work, we have developed a line-search-based algorithm framework for stability analysis and multiphase flash calculations. This algorithm is rooted in the modified Newton step and line-search method. The modified Newton step, derived from modified Cholesky factorization, ensures a descent direction, while the line search determines the degree of decrease. This combination facilitates convergence even in challenging regions for phase stability analysis and multiphase flash calculations, exhibiting superlinear convergence speed. Unlike traditional approaches that rely on successive substitution iteration and may resort to Newton iteration only if the Hessian matrix is positive definite, our algorithm incorporates modification via modified Cholesky factorization upon encountering a nonpositive definite Hessian matrix. We tested our algorithm with several classical fluids, demonstrating its efficiency and robustness. Furthermore, we assessed the algorithm's performance by computing the pressure-composition diagram for the CO2-hydrocarbon system, where all calculations achieved rapid convergence without failure. This newly developed algorithm for phase stability analysis and multiphase flash calculations represents a significant advancement for compositional or chemical process simulations.

An effective subgradient algorithm via Mifflin’s line search for nonsmooth nonconvex multiobjective optimization

We propose a descent subgradient algorithm for unconstrained nonsmooth nonconvex multiobjective optimization problems. To find a descent direction, we present an iterative process that efficiently approximates the ɛ-subdifferential of each objective function. To this end, we develop a new variant of Mifflin’s line search in which the subgradients are arbitrary and its finite convergence is proved under a semismooth assumption. To reduce the number of subgradient evaluations, we employ a backtracking line search that identifies the objectives requiring an improvement in the current approximation of the ɛ-subdifferential. Meanwhile, for the remaining objectives, new subgradients are not computed. Unlike bundle-type methods, the proposed approach can handle nonconvexity without the need for algorithmic adjustments. Moreover, the quadratic subproblems have a simple structure, and hence the method is easy to implement. We analyze the global convergence of the proposed method and prove that any accumulation point of the generated sequence satisfies a necessary Pareto optimality condition. Furthermore, our convergence analysis addresses a theoretical challenge in a recently developed subgradient method. Through numerical experiments, we observe the practical capability of the proposed method and evaluate its efficiency when applied to a diverse range of nonsmooth test problems.

A modified quasi-Newton method for uncertain multiobjective optimization problems under a finite uncertainty set

ABSTRACT In this article, a modified quasi-Newton method is developed for the robust counterpart (RC) of an uncertain multiobjective optimization problem (UMOP). Specifically, the RC of a UMOP is the minimum of an objective-wise worst-case (OWWC) type RC. To determine the descent direction for the RC, a subproblem using a common Hessian approximation is constructed. An inexact line search technique based on the Armijo technique is employed to find an appropriate step length. Furthermore, a modified version of the Davidon–Fletcher–Powell (DFP) update formula is developed for the RC. A modified quasi-Newton algorithm is built using descent direction step length, and DFP Hessian update formula. Furthermore, the convergence of the modified quasi-Newton algorithm is also discussed. Finally, the modified quasi-Newton algorithm is illustrated with a set of test problems and compared with the weighted sum method. Moreover, it is demonstrated that the algorithm is effective for both convex and non-convex problems.

Massive MIMO secure beamforming design via manifold optimization combined with momentum

Secure beamforming with constant modulus has been regarded as a promising solution in enhancing the physical-layer security by intelligently designing the antenna phase of massive multiple-input multiple-output (MIMO) devices. The resulting problem is non-convex and NP-hard due to the constant modulus constraint (CMC). Existing methods mainly solve the problem by relaxing the constraints or the objective function, which inevitably introduces the relaxation error. We notice that the complex circle manifold (CCM), where complex numbers maintain a constant modulus, naturally satisfies the CMC. Further, considering the iteration points as high-dimensional particle points, the optimization process on the CCM can be regarded as the motion of the particle on the high-dimensional plane formed by the manifold. Based on this feature, we propose the manifold optimization method combining momentum (MOCM). Concretely, the original problem is first transformed into an unconstrained problem on the complex circle manifold. Then, the descent direction is derived to solve the problem by incorporating momentum into gradient information, which speeds up the convergence process. Simulation results show that the proposed method is 1 bps/Hz higher than existing works.

Gradient projection-based trajectory tracking control for automatic guided vehicle

In order to solve the problems of long operation time for solving the trajectory tracking controller and low real-time tracking. This paper proposes an improved model predictive control algorithm. By introducing the gradient projection method (GPM), the projection direction is used as the fastest descent direction of the objective function, which is used to reduce the number of iterations of the operation. Meanwhile, this paper adds constraints and relaxation factors to improve the trajectory tracking accuracy and prevent the controller operations from falling into local optimal solutions. Simulation by Simulink, the improved model predictive control algorithm reduces the average operation time by 24.74% and the single maximum operation time by 36.04%, effectively improving the real-time trajectory tracking performance. Finally, the verification experiment of trajectory tracking control is carried out by actual vehicle, proving the method’s effectiveness.

Meta-learning based infrared ship object detection model for generalization to unknown domains

Infrared images exhibit considerable variations in probability distributions, stemming from the utilization of distinct infrared sensors and the influence of diverse environmental conditions. The variations pose great challenges for deep learning models to detect ship objects and adapt to unseen maritime environments. To address the domain shift problem, we propose an end-to-end infrared ship object detection model based on meta-learning neural network to improve domain adaptation for target domain where data is not available at training phase. Different from existing domain generalization methods, the novelty of our model lies in the effective exploitation of meta-learning and domain adaptation, ensuring that the extracted domain-independent features are meaningful and domain-invariant at the semantic level. Firstly, a double gradient-based meta-learning algorithm is designed to solve the common optimal descent direction between different domains through two gradient updates in the inner and outer loops. The algorithm enables extraction of domain-invariant features from the pseudo-source and pseudo-target domain data. Secondly, a domain discriminator with dynamic-weighted gradient reversal layer (DWGRL) is designed to accurately classify domain-invariant features and provide additional global supervision information. Finally, a multi-scale feature aggregation method is proposed to improve the extraction of multi-scale domain-invariant features. It can effectively fuse local features at different scales and global features of targets. Extensive experimental results conducted in real nighttime water surface scenes demonstrate that the proposed model achieves very high detection accuracy on target domain data, even no target domain data was used during the training phase. Compared to the existing methods, our method not only improves the detection accuracy of infrared ships by 18%, but also exhibits the smallest standard deviation with a value of 0.93, indicating its superior generalization performance.

An explicit spectral Fletcher–Reeves conjugate gradient method for bi-criteria optimization

Abstract In this paper, we propose a spectral Fletcher–Reeves conjugate gradient-like method for solving unconstrained bi-criteria minimization problems without using any technique of scalarization. We suggest an explicit formulae for computing a descent direction common to both criteria. The latter further verifies a sufficient descent property that does not depend on the line search nor on any convexity assumption. After proving the existence of a bi-criteria Armijo-type stepsize, global convergence of the proposed algorithm is established. Finally, some numerical results and comparisons with other methods are reported.

Contrastive Unfolding Deraining Network.

Due to the fact that the degradation of image quality caused by rain usually affects outdoor vision tasks, image deraining becomes more and more important. Focusing on the single image deraining (SID) task, in this article, we propose a novel Contrastive Unfolding DEraining Network (CUDEN), which combines the traditional iterative algorithm and deep network, exhibiting excellent performance and nice interpretability. CUDEN transforms the challenge of locating rain streaks into discovering rain features and defines the relationship between the image and feature domains in terms of mapping pairs. To obtain the mapping pairs efficiently, we propose a dynamic multidomain translation (DMT) module for decomposing the original mapping into sub-mappings. To enhance the feature extraction capability of networks, we also propose a new serial multireceptive field fusion (SMF) block, which extracts complex and variable rain features with convolution kernels of different receptive fields. Moreover, we are the first to introduce contrastive learning to the SID task and combine it with perceptual loss to propose a new contrastive perceptual loss (CPL), which is quite generalized and greatly helpful in identifying the appropriate gradient descent direction during training. Extensive experiments on synthetic and real-world datasets demonstrate that our proposed CUDEN outperforms the state-of-the-art (SOTA) deraining networks.

Interval-based KKT framework for support vector machines and beyond

Our article proves inequalities for interval optimization and shows that feasible and descent directions do not intersect in constrained cases. Mainly, we establish some new interval inequalities for interval-valued functions by defining LC-partial order. We use LC-partial order to study Karush–Kuhn–Tucker (KKT) conditions and expands Gordan's theorems for interval linear inequality systems. By applying Gordan's theorem, we can determine the best outcomes for interval optimization problems (IOPs) that have constraints, such as Fritz John and KKT conditions. The optimality conditions are observed with inclusion relations rather than equality. We can use the KKT condition for binary classification with interval data and support vector machines(SVMs). We present some examples to illustrate our results.

Generalized Derivatives and Optimality Conditions in Nonconvex Optimization

In this paper, we study the radial epiderivative notion for nonconvex functions, which extends the (classical) directional derivative concept. The paper presents new definition and new properties for this notion and establishes relationships between the radial epiderivative, the Clarke’s directional derivative, the Rockafellar’s subderivative and the directional derivative. The radial epiderivative notion is used to establish new regularity conditions without convexity conditions. The paper presents explicit formulations for computing the radial epiderivatives in terms of weak subgradients and vice versa. We also present an iterative algorithm for approximate computing of radial epiderivatives and show that the algorithm terminates in a finite number of iterations. The paper analyzes necessary and sufficient conditions for global optimums in nonconvex optimization via the radial epiderivatives. We formulate a necessary and sufficient condition for a global descent direction for radially epidifferentiable nonconvex functions. All the properties and theorems presented in this paper are illustrated and interpreted on examples.

QLABGrad: A Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learning

The learning rate is a critical hyperparameter for deep learning tasks since it determines the extent to which the model parameters are adjusted during the learning course. However, the choice of learning rates typically depends on empirical judgment, which may not result in satisfactory outcomes without intensive try-and-error experiments. In this study, we propose a novel learning rate adaptation scheme called QLABGrad. Without any user-specified hyperparameter, QLABGrad automatically determines the learning rate by optimizing the quadratic loss approximation-based (QLAB) function for a given gradient descent direction, where only one extra forward propagation is required. We theoretically prove the convergence of QLABGrad under the smooth Lipschitz condition on the loss function. Experiment results on multiple architectures, including MLP, CNN, and ResNet, on MNIST, CIFAR10, and ImageNet datasets, demonstrate that QLABGrad outperforms widely adopted schemes for deep learning.