Trust Region Algorithm Research Articles

Deep Neural Networks Training by Stochastic Quasi-Newton Trust-Region Methods

While first-order methods are popular for solving optimization problems arising in deep learning, they come with some acute deficiencies. To overcome these shortcomings, there has been recent interest in introducing second-order information through quasi-Newton methods that are able to construct Hessian approximations using only gradient information. In this work, we study the performance of stochastic quasi-Newton algorithms for training deep neural networks. We consider two well-known quasi-Newton updates, the limited-memory Broyden–Fletcher–Goldfarb–Shanno (BFGS) and the symmetric rank one (SR1). This study fills a gap concerning the real performance of both updates in the minibatch setting and analyzes whether more efficient training can be obtained when using the more robust BFGS update or the cheaper SR1 formula, which—allowing for indefinite Hessian approximations—can potentially help to better navigate the pathological saddle points present in the non-convex loss functions found in deep learning. We present and discuss the results of an extensive experimental study that includes many aspects affecting performance, like batch normalization, the network architecture, the limited memory parameter or the batch size. Our results show that stochastic quasi-Newton algorithms are efficient and, in some instances, able to outperform the well-known first-order Adam optimizer, run with the optimal combination of its numerous hyperparameters, and the stochastic second-order trust-region STORM algorithm.

Interpretable model learning in variational imaging: a bilevel optimization approach

Abstract In this paper, we investigate the use of bilevel optimization for model learning in variational imaging problems. Bilevel learning is an alternative approach to deep learning methods, which leads to fully interpretable models. However, it requires a detailed analytical insight into the underlying mathematical model. We focus on the bilevel learning problem for total variation models with spatially- and patch-dependent parameters. Our study encompasses the directional differentiability of the solution mapping, the derivation of optimality conditions, and the characterization of the Bouligand subdifferential of the solution operator. We also propose a two-phase trust-region algorithm for solving the problem and present numerical tests using the CelebA dataset.

Gradient-based optimization for multi-scale geographically weighted regression

Multi-scale geographically weighted regression (MGWR) is among the most popular methods to analyze non-stationary spatial relationships. However, the current model calibration algorithm is computationally intensive: its runtime has a cubic growth with the sample size, while its memory use grows quadratically. We propose calibrating MGWR with gradient-based optimization. This is obtained by analytically deriving the gradient vector and the Hessian matrix of the corrected Akaike information criterion (AICc) and wrapping them with a trust-region optimization algorithm. We evaluate the model quality empirically. Our method converges to the same coefficients and produces the same inference as the current method but it has a substantial computational gain when the sample size is large. It reduces the runtime to quadratic convergence and makes the memory use linear with respect to sample size. Our new algorithm outperforms the existing alternatives and makes MGWR feasible for large spatial datasets.

Measuring the Optical Properties of Highly Diffuse Materials.

Measuring the optical properties of highly diffuse materials is a challenge as it could be related to the white colour or an oversaturation of pixels in the acquisition system. We used a spatially resolved method and adapted a nonlinear trust-region algorithm to the fit Farrell diffusion theory model. We established an inversion method to estimate two optical properties of a material through a single reflectance measurement: the absorption and the reduced scattering coefficient. We demonstrate the validity of our method by comparing results obtained on milk samples, with a good fitting and a retrieval of linear correlations with the fat content, given by R2 scores over 0.94 with low p-values. The values of absorption coefficients retrieved vary between 1 × 10-3 and 8 × 10-3 mm-1, whilst the values of the scattering coefficients obtained from our method are between 3 and 8 mm-1 depending on the percentage of fat in the milk sample, and under the assumption of the anisotropy factor g>0.8. We also measured and analyzed the results on white paint and paper, although the paper results were difficult to relate to indicators. Thus, the method designed works for highly diffuse isotropic materials.

Integrated fire/flight control coupler design for armed helicopters based on improved non-monotone adaptive trust region algorithm with distributionally robust optimization

Aiming at enhancing the extent of military automatization and the effectiveness of armed helicopters in complex battlefield environment, integrated fire/flight control (IFFC) coupler connects the fire control system and the flight control system which is used to generate the optimal flight command signals. However, it is difficult to obtain satisfactory flight command signals within a certain firing accuracy due to the unavoidable uncertain information in battlefield environment. In this paper, a new IFFC scheme is proposed for armed helicopters against complex combat by using the idea of distributionally robust optimization combined with the framework of trust region. Initially, a vector equation and coordinate transformation are employed to acquire the relative motion model of the IFFC coupler problem in complex battlefield environment. Moreover, an improved performance index is designed according to the constraints, and the original non-convex problem is handled by the Taylor formula combined with an improved BFGS (Broyden–Fletcher–Goldfard–Shanno) algorithm. Considering the uncertain parameters during optimization, the non-linear distributionally robust optimization is transformed into a linear matrix inequality optimization problem based on the worst-case conditional value-at-risk (WC-CVaR) approximation. Then, an algorithmic framework with a non-monotone adaptive trust region algorithm is given to obtain the optimal flight command signals. Finally, a simulation example is given to illustrate the effectiveness of the proposed approach.

Two adaptive nonmonotone trust-region algorithms for solving multiobjective optimization problems

ABSTRACT This paper attempts to propose two adaptive nonmonotone trust-region algorithms in the multiobjective optimization (MO) case. The first proposed trust-region algorithm uses a modified maximum nonmonotone technique, which takes a convex combination of the maximum value of some preceding successful iterates and the function value in the current iterate. The second one employs the average nonmonotone technique, which takes a weighted average of the successive function values. Under some suitable assumptions, the convergence of the sequences generated by the trust-region algorithms that use the aforementioned nonmonotone techniques to a critical point is shown. Using some well-known test problems, we compare our proposed adaptive nonmonotone MO algorithms with some other MO trust-region algorithms. To conduct a thorough comparison in this regard, some performance criteria are used. These numerical results confirm a significant advantage of applying the proposed adaptive nonmonotone trust-region algorithms in solving MO problems. Finally, the proposed algorithms are implemented to optimize one of the optimization problems of the abrasive water-jet machining process.

Parallel EM optimization using improved pole‐residue‐based neuro‐TF surrogate and isomorphic orthogonal DOE sampling for microwave components design

AbstractDirect electromagnetic (EM) optimization for microwave components design is usually a time‐consuming process. To improve the optimization efficiency, this paper proposes a novel parallel EM optimization technique exploiting improved pole‐residue‐based neuro‐transfer function (neuro‐TF) surrogate and isomorphic orthogonal design of experiment (DOE) sampling strategy. We propose a new sampling method combining isomorphic orthogonal DOE and parallel EM simulations to generate training data for developing the neuro‐TF surrogate. This proposed sampling method can ensure the scattered distribution of data samples in the overall optimization process, thus effectively improving the surrogate accuracy and increasing the optimization speed. We also propose a new pole‐residue tracking technique for order‐changing to solve the discontinuity problem of pole/residues during the neuro‐TF surrogate development. Different from the fixed split ratio in existing pole‐residue tracking technique, the split ratio of poles and residues in the proposed technique is adaptive and determined according to the information of neighboring samples. Therefore, the continuity and smoothness of pole/residues after the splitting are improved, so as the neuro‐TF surrogate accuracy. In addition, the trust region algorithm is exploited during EM optimization to improve the convergence speed. In this way, the proposed EM optimization technique obtains the optimal solution in a shorter time with fewer iterations than the existing techniques. Two examples of EM optimizations of microwave components are used to illustrate the proposed technique.

Riemannian Trust Region Method for Minimization of the Fourth Central Moment for Localized Molecular Orbitals.

The importance of localized molecular orbitals (MOs) in correlation treatments beyond mean-field calculation and in the illustration of chemical bonding (and antibonding) can hardly be overstated. However, the generation of orthonormal localized occupied MOs is significantly more straightforward than obtaining orthonormal localized virtual MOs. Orthonormal MOs allow facile use of highly efficient group theoretical methods (e.g., graphical unitary group approach) for calculation of Hamiltonian matrix elements in multireference configuration interaction calculations (such as MRCISD) and in quasi-degenerate perturbation treatments, such as the Generalized Van Vleck Perturbation Theory. Moreover, localized MOs can elucidate qualitative understanding of bonding in molecules, in addition to high-accuracy quantitative descriptions. We adopt the powers of the fourth moment cost function introduced by Jørgensen and coworkers. Because the fourth moment cost functions are prone to having multiple negative Hessian eigenvalues when starting from easily available canonical (or near-canonical) MOs, standard optimization algorithms can fail to obtain the orbitals of the virtual or partially occupied spaces. To overcome this drawback, we applied a trust region algorithm on an orthonormal Riemannian manifold with an approximate retraction from the tangent space built into the first and second derivatives of the cost function. Moreover, the Riemannian trust region outer iterations were coupled to truncated Conjugate Gradient inner loops, which avoided any costly solutions of simultaneous linear equations or eigenvector/eigenvalue solutions. Numerical examples are provided on model systems, including the high-connectivity H10 set in 1-, 2-, and 3-dimensional arrangements, and on a chemically realistic description of cyclobutadiene (c-C4H4) and the propargyl radical (C3H3). In addition to demonstrating the algorithm on occupied and virtual blocks of orbitals, the method is also shown to work on the active space at the MCSCF level of theory.

Optimization of the Hardware Layer for IoT Systems Using a Trust-Region Method With Adaptive Forward Finite Differences

Trust-region (TR) algorithms represent a popular class of local optimization methods. Owing to straightforward setup and low computational cost, TR routines based on linear models determined using forward finite differences (FD) are often utilized for performance tuning of microwave and antenna components incorporated within the Internet of Things systems. Despite usefulness for design of complex structures, performance of TR methods vastly depends on the quality of FD-based local models. The latter are normally identified from perturbations determined a priori using either rules-of-thumb, or as a result of manual tuning. In this work, a framework for automatic determination of FD steps and their adjustment between the TR algorithm iterations is proposed. The method involves numerical optimization of perturbations so as to equalize the objective function changes w.r.t. the center design to the desirable precision. To maintain acceptable cost, the FD-tuning procedure is executed using the same approximation model as the one exploited in the course of the structure optimization. The proposed framework has been tested on a total of twelve design problems. Furthermore, the presented method has been thoroughly validated against TR-algorithms with static, a priori selected perturbations. Numerical results indicate that the proposed framework provides up to 50% performance improvement (in terms of the optimized designs quality) compared to the state-of-the-art TR-based approaches. Usefulness of the proposed method for the real-world Internet of Things systems has been implicitly demonstrated through utilization of one of the optimized structures in a hardware layer of a real-time localization system.

Convergence analysis of a subsampled Levenberg-Marquardt algorithm

In this work, a subsampled Levenberg-Marquardt algorithm is proposed for solving nonconvex finite-sum optimization problem. At each iteration, based on subsampled function value, gradient and simplified Hessian, a linear system is inexactly solved and the regularized parameter is updated as trust-region algorithms. Provided the sample size increases asymptotically, we prove that the generated sequence converges to a stationary point almost surely.

Nonintrusive Aerodynamic Shape Optimisation with a POD-DEIM Based Trust Region Method

This work presents a strategy to build reduced-order models suitable for aerodynamic shape optimisation, resulting in a multifidelity optimisation framework. A reduced-order model (ROM) based on a discrete empirical interpolation (DEIM) method is employed in lieu of computational fluid dynamics (CFD) solvers for fast, nonlinear, aerodynamic modelling. The DEIM builds a set of interpolation points that allows it to reconstruct the flow fields from sets of basis obtained by proper orthogonal decomposition of a matrix of snapshots. The aerodynamic reduced-order model is completed by introducing a nonlinear mapping function between surface deformation and the DEIM interpolation points. The optimisation problem is managed by a trust region algorithm linking the multiple-fidelity solvers, with each subproblem solved using a gradient-based algorithm. The design space is initially restricted; as the optimisation trajectory evolves, new samples enrich the ROM. The proposed methodology is evaluated using a series of transonic viscous test cases based on wing configurations. Results show that for cases with a moderate number of design variables, the approach proposed is competitive with state-of-the-art gradient-based methods; in addition, the use of trust region methodology mitigates the likelihood of the optimiser converging to, shallower, local minima.

Faster Riemannian Newton-type optimization by subsampling and cubic regularization

This work is on constrained large-scale non-convex optimization where the constraint set implies a manifold structure. Solving such problems is important in a multitude of fundamental machine learning tasks. Recent advances on Riemannian optimization have enabled the convenient recovery of solutions by adapting unconstrained optimization algorithms over manifolds. However, it remains challenging to scale up and meanwhile maintain stable convergence rates and handle saddle points. We propose a new second-order Riemannian optimization algorithm, aiming at improving convergence rate and reducing computational cost. It enhances the Riemannian trust-region algorithm that explores curvature information to escape saddle points through a mixture of subsampling and cubic regularization techniques. We conduct rigorous analysis to study the convergence behavior of the proposed algorithm. We also perform extensive experiments to evaluate it based on two general machine learning tasks using multiple datasets. The proposed algorithm exhibits improved computational speed, e.g., a speed improvement from 12% :text {to} :227%, and improved convergence behavior, e.g., an iteration number reduction from mathcal{O}left(maxleft(epsilon_g^{-2}epsilon_H^{-1},epsilon_H^{-3}right)right) ,text {to}: mathcal{O}left(maxleft(epsilon_g^{-2},epsilon_H^{-3}right)right), compared to a large set of state-of-the-art Riemannian optimization algorithms.

A Human–Robot Collaboration Method Using a Pose Estimation Network for Robot Learning of Assembly Manipulation Trajectories From Demonstration Videos

The wide application of industrial robots has greatly improved assembly efficiency and reliability. However, determining how to efficiently teach a robot to perform assembly manipulation trajectories from demonstration videos is a challenging issue. This paper proposes a method integrating deep learning, image processing, and an iteration model to predict the real assembly manipulation trajectory of a human hand from a video without specific depth information. First, a pose estimation network, Keypoint-RCNN, is used to accurately estimate hand pose in the two-dimensional (2-D) image of each frame in a video. Second, image processing is applied to map the 2-D hand pose estimated by the neural network with the real 3-D assembly space. An iteration model based on the trust region algorithm is proposed to solve for the quaternions and translation vectors of two frames. All the quaternions and translation vectors form the predicted assembly manipulation trajectories. Finally, a UR3 robot is used to imitate the assembly operation based on the predicted manipulation trajectories. The results show that the robot could successfully imitate various operations based on the predicted manipulation trajectories.

Design and Optimization of Metamaterial-Based Dual-Band 28/38 GHz 5G MIMO Antenna With Modified Ground for Isolation and Bandwidth Improvement

This letter presents a high-isolation dual-band multiple-input multiple-output (MIMO) antenna based on the ground plane modification and optimized metamaterials (MMs) for 5G millimeter-wave applications. The antenna is a monopole providing a dual-band response at 5G 28/38 bands with a small physical size (4.8 × 2.9 × 0.762 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> , excluding the feeding line). The MIMO consists of two symmetric radiating elements arranged adjacently with the mutual coupling of −19 dB and −17 dB at 28 and 38 GHz, respectively. Reducing the ground length enhances the isolation to 34.6 dB at 28 GHz, whereas embedding the MMs diminishes the coupling to −47 dB at 38 GHz. The trust-region gradient-based search algorithm is employed to optimize the MM structure dimensions and to enhance the isolation at 38 GHz. The MMs also contribute to the enhancement of bandwidth from 1.6 GHz to 1.7 GHz at 28 GHz (27.1–28.8 GHz) and from 2.75 GHz to 3.7 GHz at 38 GHz (35.2–38.9 GHz). The proposed system is capable of covering both 28/38 bands with low profile structure, high isolation, low envelope correlation coefficient (ECC) ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\lt 0.2 \cdot 10^{-4}$</tex-math></inline-formula> ), and high diversity gain (DG) (>9.99 dB). These properties demonstrate the system applicability for 5G millimeter-wave cellular communications.

Rapid multi-criterial design of microwave components with robustness analysis by means of knowledge-based surrogates

Manufacturing tolerances and uncertainties concerning material parameters, e.g., operating conditions or substrate permittivity are detrimental to characteristics of microwave components. The knowledge of relations between acceptable parameter deviations (not leading to violation of design specifications) and the nominal performance (not considering uncertainties), and is therefore indispensable. This paper proposes a multi-objective optimization technique of microwave components with tolerance analysis. The goal is to identify a set of trade-off designs: nominal performance versus robustness (quantified by the maximum input tolerance values that allow for achieving 100-percent fabrication yield). Our approach exploits knowledge-driven regression predictors rendered using characteristic points (features) of the component’s response for a rapid evaluation of statistical performance figures, along with trust-region algorithm to enable low execution cost as well as convergence. The proposed methodology is verified with the use of three microstrip circuits, a broadband filter, and two branch-line couplers (a single- and a dual-band one). It is demonstrated that a Pareto set w.r.t. nominal performance and robustness objectives can be produced using only 40 to 60 EM simulations of the respective structure (per design). Reliability of the proposed algorithm is corroborated with the use of EM-based Monte Carlo simulation.

Efficient Solution of Discrete Subproblems Arising in Integer Optimal Control with Total Variation Regularization

We consider a class of integer linear programs (IPs) that arise as discretizations of trust-region subproblems of a trust-region algorithm for the solution of control problems, where the control input is an integer-valued function on a one-dimensional domain and is regularized with a total variation term in the objective, which may be interpreted as a penalization of switching costs between different control modes. We prove that solving an instance of the considered problem class is equivalent to solving a resource-constrained shortest-path problem (RCSPP) on a layered directed acyclic graph. This structural finding yields an algorithmic solution approach based on topological sorting and corresponding run-time complexities that are quadratic in the number of discretization intervals of the underlying control problem, the main quantifier for the size of a problem instance. We also consider the solution of the RCSPP with an [Formula: see text] algorithm. Specifically, the analysis of a Lagrangian relaxation yields a consistent heuristic function for the [Formula: see text] algorithm and a preprocessing procedure, which can be employed to accelerate the [Formula: see text] algorithm for the RCSPP without losing optimality of the computed solution. We generate IP instances by executing the trust-region algorithm on several integer optimal control problems. The numerical results show that the accelerated [Formula: see text] algorithm and topological sorting outperform a general-purpose IP solver significantly. Moreover, the accelerated [Formula: see text] algorithm is able to outperform topological sorting for larger problem instances. We also give computational evidence that the performance of the superordinate trust-region algorithm may be improved if it is initialized with a solution obtained with the combinatorial integral approximation. History: Accepted by Andrea Lodi, Area Editor for Design and Analysis of Algorithms–Discrete. Supplemental Material: The online supplement is available at https://doi.org/10.1287/ijoc.2023.1294 .

Impedance eduction experiments covering higher frequencies based on the multimodal straightforward method

The multimodal straightforward method is developed further, validated and applied experimentally. The trust-region algorithm is introduced to reduce the eduction time, a flow duct setup with a cross section of 51 × 51 mm2 is built for the impedance eduction, and several single- and multi-layer perforated liners are tested from 0.5 kHz up to 6.0 kHz at two average Mach numbers under several types of modal sound sources. It is validated that, this method, by utilizing a diagonal microphone array and improving Prony’s algorithm, enables the automatic match between axial and transverse wavenumbers, and thus realizes the full decomposition to three-dimension multimodal sound fields. By introducing the trust-region algorithm, it can educe the impedance with a good consistency and a higher time efficiency. Generally, this method is applicable to the practical impedance eduction in multimodal aeroacoustic environments on a real flow duct, with the upper frequency extended from 3.0 to 6.0 kHz covering the main frequency range of aeroengine fan noises. For multilayer liners, the educed impedance spectrum is considerably flatter than the prediction by an existing impedance model in some cases especially near the antiresonance, and the grazing flow magnifies such a flatting trend accompanied by more evident predicting deviations. The spectra of educed impedance and transmission loss show that the multilayer liner possesses more resonant frequencies within the concerned range and broadband absorption potentials if well designed.

Rapid antenna optimization with restricted sensitivity updates by automated dominant direction identification

Meticulous tuning of geometry parameters turns pivotal in improving performance of antenna systems. It is more and more often realized using formal optimization methods, which is demonstrably the most efficient way of handling multiple design variables, objectives, and constraints. Although in some cases a need for launching global search arises, a typical design scenario only requires local optimization, especially when a decent initial design can be rendered using engineering experience or parametric studies. At the same time, antenna optimization is typically conducted using full-wave electromagnetic (EM) simulations, which entails considerable computational expenses. In this paper, we introduce a novel procedure for expedited antenna tuning. Its fundamental mechanism is to restrict the antenna response sensitivity updates to the selected dominant directions within the parameter space, determined based on the problem-specific knowledge, in particular, the estimated changes of antenna characteristics when moving across one-dimensional affine subspaces spanned by these directions. Thus, the said dominant directions affect the most the responses of the antenna structure under design as assessed using the introduced metrics. The decision making process concerning the number of directions to be used relies on quantification of the aggregated system response variability metrics. The proposed approach is demonstrated by means of several antenna structures and benchmarked against conventional trust-region algorithm, but also its accelerated versions. The results indicate considerable (up to over 60%) speedup over the reference procedure without noticeable quality degradation.

Discriminative subspace learning via optimization on Riemannian manifold

Discriminative subspace learning is an important problem in machine learning, which aims to find the maximum separable decision subspace. Traditional Euclidean-based methods usually use Fisher discriminant criterion for finding an optimal linear mapping from a high-dimensional data space to a lower-dimensional subspace, which hardly guarantee a quadratic rate of global convergence and suffers from the singularity problem. Here, we propose the manifold optimization-based discriminant analysis (MODA) which is constructed by using the latent subspace alignment and the geometry of objective function with orthogonality constraint. MODA is solved by using Riemannian version of trust-region algorithm. Experimental results on various image datasets and electroencephalogram (EEG) datasets show that MODA achieves the best separability and is significantly superior to the competing algorithms. Especially for the time series of EEG signals, the accuracy of MODA is 20–30% higher than existing algorithms. The code for MODA is available at https://github.com/ncclabsustech/MODA-algorithm.

A Bundle Trust Region Algorithm for Minimizing Locally Lipschitz Functions

We propose a new algorithm for minimizing locally Lipschitz functions that combines both the bundle and trust region techniques. Based on the bundle methods the objective function is approximated by a piecewise linear working model which is updated by adding cutting planes at unsuccessful trial steps. The algorithm defines, at each iteration, a new trial point by solving a subproblem that employs the working model in the objective function subject to a region, which is called the trust region. The algorithm is studied from both theoretical and practical points of view. Under a assumption on the objective function, global convergence of it is verified to stationary points. In order to demonstrate the reliability and efficiency of the proposed algorithm, a MATLAB implementation of it is prepared and numerical experiments have been made using some academic nonsmooth test problems. Computational results show that the developed method is efficient for solving nonsmooth and nonconvex optimization problems.