Unconstrained Optimization Research Articles

On Spectral Dai-Yuan Structured Technique for Solving Nonlinear Least-Squares Optimisation

Handling nonlinear least squares (NLS) problems as unconstrained optimisation problems has led to the development and adaptation of numerous numerical approaches. Calculating the Hessian matrix expensively for each iteration is one of the problems with the current numerical approaches for solving NLS problems. Some methods designed to rely solely on first-derivative information while ignoring second-order details tend to underperform when dealing with problems involving non-zero residuals. This paper suggests a modification to the Dai-Yuan type (DY) formula by deriving a spectral parameter for the proposed search direction. The sufficient descent results of the new formula are established using the standard assumptions under the strong Wolfe line search. Furthermore, experiments are conducted on some benchmark functions to demonstrate the computational efficiency of the proposed technique. The results highlight the effectiveness of the algorithm compared to existing methods.Keywords: Conjugate gradient, convergence theory, Hessian matrix, optimisation, Quasi-Newton, spectral, structured secant method

An efficient hybrid conjugate gradient method with an adaptive strategy and applications in image restoration problems

In this study, we introduce a novel hybrid conjugate gradient method with an adaptive strategy called asHCG method. The asHCG method exhibits the following characteristics. (i) Its search direction guarantees sufficient descent property without dependence on any line search. (ii) It possesses strong convergence for the uniformly convex function using a weak Wolfe line search, and under the same line search, it achieves global convergence for the general function. (iii) Employing the Armijo line search, it provides an approximate guarantee for worst-case complexity for the uniformly convex function. The numerical results demonstrate promising and encouraging performances in both unconstrained optimization problems and image restoration problems.

An Investigation Of A New Hybrid Conjugates Gradient Approach For Unconstrained Optimization Problems

This work introduces a novel hybrid conjugate gradient (CG) technique for tackling unconstrained optimisation problems with improved efficiency and effectiveness. The parameter is computed as a convex combination of the standard conjugate gradient techniques using and . Our proposed method has shown that when using the strong Wolfe-line-search(SWC) under specific conditions, it achieves global theoretical convergence. In addition, the new hybrid CG approach has the ability to generate a search direction that moves downward with each iteration. The quantitative findings obtained by applying the recommended technique about 30 functions with varying dimensions clearly illustrate its effectiveness and potential.

Two Algorithms for Solving Unconstrained Global Optimization by Auxiliary Function Method

Two Algorithms for Solving Unconstrained Global Optimization by Auxiliary Function Method

Efficient molecular conformation generation with quantum-inspired algorithm.

Conformation generation, also known as molecular unfolding (MU), is a crucial step in structure-based drug design, remaining a challenging combinatorial optimization problem. Quantum annealing (QA) has shown great potential for solving certain combinatorial optimization problems over traditional classical methods such as simulated annealing (SA). However, a recent study showed that a 2000-qubit QA hardware was still unable to outperform SA for the MU problem. Here, we propose the use of quantum-inspired algorithm to solve the MU problem, in order to go beyond traditional SA. We introduce a highly compact phase encoding method which can exponentially reduce the representation space, compared with the previous one-hot encoding method. For benchmarking, we tested this new approach on the public QM9 dataset generated by density functional theory (DFT). The root-mean-square deviation between the conformation determined by our approach and DFT is negligible (less than about 0.5Å), which underpins the validity of our approach. Furthermore, the median time-to-target metric can be reduced by a factor of five compared to SA. Additionally, we demonstrate a simulation experiment by MindQuantum using quantum approximate optimization algorithm (QAOA) to reach optimal results. These results indicate that quantum-inspired algorithms can be applied to solve practical problems even before quantum hardware becomes mature. The objective function of MU is defined as the sum of all internal distances between atoms in the molecule, which is a high-order unconstrained binary optimization (HUBO) problem. The degree of freedom of variables is discretized and encoded with binary variables by the phase encoding method. We employ the quantum-inspired simulated bifurcation algorithm for optimization. The public QM9 dataset is generated by DFT. The simulation experiment of quantum computation is implemented by MindQuantum using QAOA.

Performance analyses of weighted superposition attraction-repulsion algorithms in solving difficult optimization problems

ABSTRACT The purpose of this paper is to test the performance of the recently proposed weighted superposition attraction-repulsion algorithms (WSA and WSAR) on unconstrained continuous optimization test problems and constrained optimization problems. WSAR is a successor of weighted superposition attraction algorithm (WSA). WSAR is established upon the superposition principle from physics and mimics attractive and repulsive movements of solution agents (vectors). Differently from the WSA, WSAR also considers repulsive movements with updated solution move equations. WSAR requires very few algorithm-specific parameters to be set and has good convergence and searching capability. Through extensive computational tests on many benchmark problems including CEC’2015 and CEC’2020 performance of the WSAR is compared against WSA and other metaheuristic algorithms. It is statistically shown that the WSAR algorithm is able to produce good and competitive results in comparison to its predecessor WSA and other metaheuristic algorithms.

On a model-free meta-heuristic approach for unconstrained optimization

On a model-free meta-heuristic approach for unconstrained optimization

A family of spectral conjugate gradient methods with strong convergence and its applications in image restoration and machine learning

In this paper, we propose a family of spectral conjugate gradient methods for solving unconstrained optimization problems. Specifically, we provide two classes of bounded spectral parameters to be chosen, design a new truncation scheme of the non-negative conjugate parameter and set a restart procedure in our search direction. Independently of the specific spectral parameter, conjugate parameter and line search criterion, we prove that our proposed family satisfies the sufficient descent condition. We also prove its strong convergence under mild assumptions and the weak Wolfe line search. Numerical comparisons with other methods demonstrate the outstanding performances of our algorithm for solving medium–large-scale unconstrained optimization, image restoration and machine learning problems.

Another hybrid conjugate gradient method as a convex combination of WYL and CD methods

Abstract Conjugate gradient (CG) methods are a popular class of iterative methods for solving linear systems of equations and nonlinear optimization problems. In this paper, a new hybrid conjugate gradient (CG) method is presented and analyzed for solving unconstrained optimization problems, where the parameter β k \beta_{k} is a convex combination of β k WYL \beta_{k}^{\mathrm{WYL}} and β k CD \beta_{k}^{\mathrm{CD}} . Under the strong Wolfe line search, the new method possesses the sufficient descent condition and the global convergence properties. The preliminary numerical results show the efficiency of our method in comparison with other CG methods. Furthermore, the proposed algorithm HWYLCD was extended to solve the problem of a mode function.

Two efficient spectral hybrid CG methods based on memoryless BFGS direction and Dai–Liao conjugacy condition

The spectral conjugate gradient (CG) method is one of the effective methods for solving unconstrained optimization problems. In this work, we introduce a composite hybrid CG parameter which is a convex combination of two new adaptive hybrid CG parameters. To derive the optimal choice of the combination coefficient in our proposed composite CG parameter, two approaches to calculating it are introduced. One is to minimize the distance between the hybrid CG direction and the self-scaling memoryless BFGS direction and the other is to apply the Dai–Liao conjugacy condition. Further, to make the search direction have better theoretical performance, two effective spectral hybrid CG methods are generated. Our proposed methods ensure the sufficient descent property regardless of the line search. And the global convergence results for general non-convex functions are established under some fundamental assumptions and Wolfe line search. Numerical experiments on solving unconstrained optimization problems illustrate the effectiveness of our proposed methods.

Rydberg‐Atom Graphs for Quadratic Unconstrained Binary Optimization Problems

AbstractThere is a growing interest in harnessing the potential of the Rydberg‐atom system to address complex combinatorial optimization challenges. Here an experimental demonstration of how the quadratic unconstrained binary optimization (QUBO) problem can be effectively addressed using Rydberg‐atom graphs is presented. The Rydberg‐atom graphs are configurations of neutral atoms organized into mathematical graphs, facilitated by programmable optical tweezers, and designed to exhibit many‐body ground states that correspond to the maximum independent set (MIS) of their respective graphs. Four elementary Rydberg‐atom subgraph components are developed, not only to eliminate the need of local control but also to be robust against interatomic distance errors, while serving as the building blocks sufficient for formulating generic QUBO graphs. To validate the feasibility of the approach, a series of Rydberg‐atom experiments selected to demonstrate proof‐of‐concept operations of these building blocks are conducted. These experiments illustrate how these components can be used to programmatically encode the QUBO problems to Rydberg‐atom graphs and, by measuring their many‐body ground states, how their QUBO solutions are determined subsequently.

Integrating quantum and classical computing for multi-energy system optimization using Benders decomposition

During recent years, quantum computers have received increasing attention, primarily due to their ability to significantly increase computational performance for specific problems. Computational performance could be improved for mathematical optimization by quantum annealers. This special type of quantum computer can solve quadratic unconstrained binary optimization problems. However, multi-energy systems optimization commonly involves integer and continuous decision variables. Due to their mixed-integer problem structure, quantum annealers cannot be directly used for multi-energy system optimization.To solve multi-energy system optimization problems, we present a hybrid Benders decomposition approach combining optimization on quantum and classical computers. In our approach, the quantum computer solves the master problem, which involves only the integer variables from the original energy system optimization problem. The subproblem includes the continuous variables and is solved by a classical computer. For better performance, we apply improvement techniques to the Benders decomposition. We test the approach on a case study to design a cost-optimal multi-energy system. While we provide a proof of concept that our Benders decomposition approach is applicable for the design of multi-energy systems, the computational time is still higher than for approaches using classical computers only. We therefore estimate the potential improvement of our approach to be expected for larger and fault-tolerant quantum computers.

New Hybrid Conjugate Gradient Method as a Convex Combination of PRP and RMIL+ Methods

The Conjugate Gradient (CG) method is a powerful iterative approach for solving large-scale minimization problems, characterized by its simplicity, low computation cost and good convergence. In this paper, a new hybrid conjugate gradient HLB method (HLB: Hadji-Laskri-Bechouat) is proposed and analysed for unconstrained optimization. By comparing numerically CGHLB with PRP and RMIL+ and by using the Dolan and More CPU performance, we deduce that CGHLB is more efficient. Keywords: Unconstrained optimization, hybrid conjugate gradient method, line search, descent property, global convergence.

Two sufficient descent spectral conjugate gradient algorithms for unconstrained optimization with application

Two sufficient descent spectral conjugate gradient algorithms for unconstrained optimization with application

Coherent point drift with Skewed Distribution for accurate point cloud registration

Point cloud registration methods based on Gaussian Mixture Models (GMMs) exhibit high robustness. However, GMM cannot precisely depict point clouds, because the Gaussian distribution is spatially symmetric and local surfaces of point clouds are typically non-symmetric. In this paper, we propose a novel method for rigid point cloud registration, termed coherent point drift with Skewed Distribution (Skewed CPD). Our method employs an asymmetric distribution constructed from the local surface normals and curvature radii. Compared to the Gaussian distribution, this skewed distribution provides a more accurate spatial description of points on local surfaces. Additionally, we integrate an adaptive multiplier to the covariance, which reallocates the weight of the covariance for different components in the probabilistic mixture model. We employ the EM algorithm to address this maximum likelihood estimation (MLE) issue and leverage GPU acceleration. In the M-step, we adopt an unconstrained optimization technique rooted in a Lie group and Lie algebra to attain the optimal transformation. Experimental results indicate that our method outperforms state-of-the-art methods in both accuracy and robustness. Remarkably, even without loop closure detection, the cumulative error of our approach remains minimal.

Neural network approach to portfolio optimization with leverage constraints: a case study on high inflation investment

Motivated by the current global high inflation scenario, we aim to discover a dynamic multi-period allocation strategy to optimally outperform a passive benchmark while adhering to a bounded leverage limit. We formulate an optimal control problem to outperform a benchmark portfolio throughout the investment horizon. To obtain strategies under the bounded leverage constraint among other realistic constraints, we propose a novel leverage-feasible neural network (LFNN) to represent the control, which converts the original constrained optimization problem into an unconstrained optimization problem that is computationally feasible with gradient descent, without dynamic programming. We establish mathematically that the LFNN approximation can yield a solution that is arbitrarily close to the solution of the original optimal control problem with bounded leverage. We further validate the performance of the LFNN empirically by deriving a closed-form solution under jump-diffusion asset price models and show that a shallow LFNN model achieves comparable results on synthetic data. In the case study, we apply the LFNN approach to a four-asset investment scenario with bootstrap-resampled asset returns from the filtered high inflation regimes. The LFNN strategy is shown to consistently outperform the passive benchmark strategy by about 200 bps (median annualized return), with a greater than 90% probability of outperforming the benchmark at the end of the investment horizon.

Perturbed Utility Stochastic Traffic Assignment

This paper develops a fast algorithm for computing the equilibrium assignment with the perturbed utility route choice (PURC) model. Without compromise, this allows the significant advantages of the PURC model to be used in large-scale applications. We formulate the PURC equilibrium assignment problem as a convex minimization problem and find a closed-form stochastic network loading expression that allows us to formulate the Lagrangian dual of the assignment problem as an unconstrained optimization problem. To solve this dual problem, we formulate a quasi-Newton accelerated gradient descent algorithm (qN-AGD*). Our numerical evidence shows that qN-AGD* clearly outperforms a conventional primal algorithm and a plain accelerated gradient descent algorithm. qN-AGD* is fast with a runtime that scales about linearly with the problem size, indicating that solving the perturbed utility assignment problem is feasible also with very large networks. Funding: This work has been financed by the European Union—NextGenerationEU.

ILP-based resource optimization realized by quantum annealing for optical wide-area communication networks—A framework for solving combinatorial problems of a real-world application by quantum annealing

Resource allocation of wide-area internet networks is inherently a combinatorial optimization problem that if solved quickly, could provide near real-time adaptive control of internet-protocol traffic ensuring increased network efficacy and robustness, while minimizing energy requirements coming from power-hungry transceivers. In recent works we demonstrated how such a problem could be cast as a quadratic unconstrained binary optimization (QUBO) problem that can be embedded onto the D-Wave Advantage™ quantum annealer system, demonstrating proof of principle. Our initial studies left open the possibility for improvement of D-Wave solutions via judicious choices of system run parameters. Here we report on our investigations for optimizing these system parameters, and how we incorporate machine learning (ML) techniques to further improve on the quality of solutions. In particular, we use the Hamming distance to investigate correlations between various system-run parameters and solution vectors. We then apply a decision tree neural network (NN) to learn these correlations, with the goal of using the neural network to provide further guesses to solution vectors. We successfully implement this NN in a simple integer linear programming (ILP) example, demonstrating how the NN can fully map out the solution space that was not captured by D-Wave. We find, however, for the 3-node network problem the NN is not able to enhance the quality of space of solutions.

A modified Fletcher-Reeves conjugate gradient method for unconstrained optimization with applications in image restoration

A modified Fletcher-Reeves conjugate gradient method for unconstrained optimization with applications in image restoration

Combined methods for solving degenerate unconstrained optimization problems

UDC 519.853.6 : 519.613.2 We present constructive second- and fourth-order methods for solving degenerate unconstrained optimization problems. The fourth-order method applied in the present work is a combination of the Newton method and the method that uses fourth-order derivatives. Our approach is based on the decomposition of ℝ n into the direct sum of the kernel of a Hessian matrix and its orthogonal complement. The fourth-order method is applied to the kernel of the Hessian matrix, whereas the Newton method is applied to its orthogonal complement. This method proves to be efficient in the case of a one-dimensional kernel of the Hessian matrix. In order to get the second-order method, Newton's method is combined with the steepest-descent method. We study the efficiency of these methods and analyze their convergence rates. We also propose a new adaptive combined quasi-Newton-type method (ACQNM) based on the use of the second- and fourth-order methods in the degenerate case. The efficiency of ACQNM is demonstrated by analyzing an example of some most common test functions.