Convergence Rate Research Articles

Fast convergence of the primal-dual dynamical system and corresponding algorithms for a nonsmooth bilinearly coupled saddle point problem

AbstractThis paper studies the convergence rate of a second-order dynamical system associated with a nonsmooth bilinearly coupled convex-concave saddle point problem, as well as the convergence rate of its corresponding discretizations. We derive the convergence rate of the primal-dual gap for the second-order dynamical system with asymptotically vanishing damping term. Based on an implicit discretization scheme, we propose a primal-dual algorithm and provide a non-ergodic convergence rate under a general setting for the inertial parameters when one objective function is continuously differentiable and convex and the other is a proper, convex and lower semicontinuous function. For this algorithm we derive a $$O\left( 1/k^2 \right) $$ O 1 / k 2 convergence rate under three classical rules proposed by Nesterov, Chambolle-Dossal and Attouch-Cabot without assuming strong convexity, which is compatible with the results of the continuous-time dynamic system. For the case when both objective functions are continuously differentiable and convex, we further present a primal-dual algorithm based on an explicit discretization. We provide a corresponding non-ergodic convergence rate for this algorithm and show that the sequence of iterates generated weakly converges to a primal-dual optimal solution. Finally, we present numerical experiments that indicate the superior numerical performance of both algorithms.

Reducing measurement costs by recycling the Hessian in adaptive variational quantum algorithms

Abstract Adaptive protocols enable the construction of more efficient state preparation circuits in variational quantum algorithms (VQAs) by utilizing data obtained from the quantum processor during the execution of the algorithm. This idea originated with Adaptive Derivative-Assembled Problem-Tailored variational quantum eigensolver (ADAPT-VQE), an algorithm that iteratively grows the state preparation circuit operator by operator, with each new operator accompanied by a new variational parameter, and where all parameters acquired thus far are optimized in each iteration. In ADAPT-VQE and other adaptive VQAs that followed it, it has been shown that initializing parameters to their optimal values from the previous iteration speeds up convergence and avoids shallow local traps in the parameter landscape. However, no other data from the optimization performed at one iteration is carried over to the next. In this work, we propose an improved quasi-Newton optimization protocol specifically tailored to adaptive VQAs. The distinctive feature in our proposal is that approximate second derivatives of the cost function are recycled across iterations in addition to optimal parameter values. We implement a quasi-Newton optimizer where an approximation to the inverse Hessian matrix is continuously built and grown across the iterations of an adaptive VQA. The resulting algorithm has the flavor of a continuous optimization where the dimension of the search space is augmented when the gradient norm falls below a given threshold. We show that this inter-optimization exchange of second-order information leads the approximate Hessian in the state of the optimizer to be consistently closer to the exact Hessian. As a result, our method achieves a superlinear convergence rate even in situations where the typical implementation of a quasi-Newton optimizer converges only linearly. Our protocol decreases the measurement costs in implementing adaptive VQAs on quantum hardware as well as the runtime of their classical simulation.

Groupers and moray eels (GME) optimization: a nature-inspired metaheuristic algorithm for solving complex engineering problems

AbstractAs engineering technology advances and the number of complex engineering problems increases, there is a growing need to expand the abundance of swarm intelligence algorithms and enhance their performance. It is crucial to develop, assess, and hybridize new powerful algorithms that can be used to deal with optimization issues in different fields. This paper proposes a novel nature-inspired algorithm, namely the Groupers and Moray Eels (GME) optimization algorithm, for solving various optimization problems. GME mimics the associative hunting between groupers and moray eels. Many species, including chimpanzees and lions, have shown cooperation during hunting. Cooperative hunting among animals of different species, which is called associative hunting, is extremely rare. Groupers and moray eels have complementary hunting approaches. Cooperation is thus mutually beneficial because it increases the likelihood of both species successfully capturing prey. The two predators have complementary hunting methods when they work together, and an associated hunt creates a multi-predator attack that is difficult to evade. This example of hunting differs from that of groups of animals of the same species due to the high level of coordination among the two species. GME consists of four phases: primary search, pair association, encircling or extended search, and attacking and catching. The behavior characteristics are mathematically represented to allow for an adequate balance between GME exploitation and exploration. Experimental results indicate that the GME outperforms competing algorithms in terms of accuracy, execution time, convergence rate, and the ability to locate all or the majority of local or global optima.

Triply Bifurcations and Metric Universalities in 1D Trimodal Maps

The famous Feigenbaum’s constants such as scaling factor and convergence rate can be measured and recomputed in unimodal maps, one of the methods is by the n-tupling bifurcations to chaos and parameter calculation algorithm which are called star transformation and word-lifting technique respectively. In the paper, we presented a kind of simple triply star transformation rule and the corresponding proof of admissible three super-stable kneading sequences (TSSKS), a few of TSSKS to chaos by triply bifurcations and the metric universalities are investigated.

On the Redundancy of Hessian Non-Singularity for Linear Convergence Rate of the Newton Method Applied to the Minimization of Convex Functions

A new property of convex functions that makes it possible to achieve the linear rate of convergence of the Newton method during the minimization process is established. Namely, it is proved that, even in the case of singularity of the Hessian at the solution, the Newtonian system is solvable in the vicinity of the minimizer; i.e., the gradient of the objective function belongs to the image of the matrix of second derivatives and, therefore, analogs of the Newton method may be used.

Sequential Kalman Tuning of the t-preconditioned Crank-Nicolson algorithm: efficient, adaptive and gradient-free inference for Bayesian inverse problems

Abstract Ensemble Kalman Inversion (EKI) has been proposed as an efficient method for the approximate solution of Bayesian inverse problems with expensive forward models. However, when applied to the Bayesian inverse problem EKI is only exact in the regime of Gaussian target measures and linear forward models. In this work we propose embedding EKI and Flow Annealed Kalman Inversion (FAKI), its normalizing flow (NF) preconditioned variant, within a Bayesian annealing scheme as part of an adaptive implementation of the $t$-preconditioned Crank-Nicolson (tpCN) sampler. The tpCN sampler differs from standard pCN in that its proposal is reversible with respect to the multivariate $t$-distribution. The more flexible tail behaviour allows for better adaptation to sampling from non-Gaussian targets. Within our Sequential Kalman Tuning (SKT) adaptation scheme, EKI is used to initialize and precondition the tpCN sampler for each annealed target. The subsequent tpCN iterations ensure particles are correctly distributed according to each annealed target, avoiding the accumulation of errors that would otherwise impact EKI. We demonstrate the performance of SKT for tpCN on three challenging numerical benchmarks, showing significant improvements in the rate of convergence compared to adaptation within standard SMC with importance weighted resampling at each temperature level, and compared to similar adaptive implementations of standard pCN. The SKT scheme applied to tpCN offers an efficient, practical solution for solving the Bayesian inverse problem when gradients of the forward model are not available. Code implementing the SKT schemes for tpCN is available at \url{https://github.com/RichardGrumitt/KalmanMC}.

Quadratic Programming Problems and Its Solution Techniques With Neural Network Modeling

ABSTRACTThis paper presents a study on quadratic programming problems (QPP) and introduces a new neural network model with feasibility analysis. The stability analysis of QPP using neural network modeling is also discussed. The proposed neural network model has a simple form, and its optimal feasibility for both primal and dual QP problems is established. The model demonstrates a good convergence rate with a minimal number of iterations, achieving a very fast convergence to the exact solutions of both the primal and dual QP problems. The optimal solutions for the original QP problem and its dual QPP are obtained. Finally, two simple numerical examples are simulated to illustrate the findings.

Error Analysis of the Vector Penalty-Projection Methods for the Time-Dependent Stokes Equations with Open Boundary Conditions

Abstract We present in this paper a rigorous error analysis of the vector penalty-projection method for solving the time-dependent incompressible Stokes equations with open boundary conditions on part of the boundary. First, we prove the stability of the scheme. Then we provide an error analysis for the second-order vector penalty-projection method which shows that the convergence rate of the error on the velocity and the pressure is of order 2 in l ∞ ⁢ ( L 2 ⁢ ( Ω ) ) l^{\infty}(\mathbf{L}^{2}(\Omega)) and l 2 ⁢ ( L 2 ⁢ ( Ω ) ) l^{2}(L^{2}(\Omega)) respectively. In addition, it is shown that the splitting errors of the method varies as O ⁢ ( ε ) \mathcal{O}(\varepsilon) , where 𝜀 is a penalty parameter chosen as small as desired. Several numerical tests in agreement with the theoretical results are presented. To the best of our knowledge, this paper provides the first rigorous proof of optimal error estimates for second-order splitting schemes with open boundary conditions.

A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search

Monte-Carlo Tree Search (MCTS) is a class of methods for solving complex decisionmaking problems through the synergy of Monte-Carlo planning and Reinforcement Learning (RL). The highly combinatorial nature of the problems commonly addressed by MCTS requires the use of efficient exploration strategies for navigating the planning tree and quickly convergent value backup methods. These crucial problems are particularly evident in recent advances that combine MCTS with deep neural networks for function approximation. In this work, we propose two methods for improving the convergence rate and exploration based on a newly introduced backup operator and entropy regularization. We provide strong theoretical guarantees to bound convergence rate, approximation error, and regret of our methods. Moreover, we introduce a mathematical framework based on the use of the α-divergence for backup and exploration in MCTS. We show that this theoretical formulation unifies different approaches, including our newly introduced ones, under the same mathematical framework, allowing to obtain different methods by simply changing the value of α. In practice, our unified perspective offers a flexible way to balance between exploration and exploitation by tuning the single α parameter according to the problem at hand. We validate our methods through a rigorous empirical study from basic toy problems to the complex Atari games, and including both MDP and POMDP problems.

A genetic algorithm with tabu search method for 3 dimensional detect profile reconstruction using MFL measurement.

In nondestructive testing, magnetic flux leakage (MFL) inspection is extensively employed for the inversion of pipeline defects. Exact reconstruction of the defect plane with measurements is a pressing issue in the field of MFL detection. This article proposes a method that incorporates a layer-by-layer genetic algorithm with tabu search (GA-TS). During the iterative process, our objective is to utilize a tabu search based on evolutionary algorithms to avoid local optimal solutions, obtain global optimal solutions, and reconstruct defects with enhanced accuracy. In addition to enhancing the accuracy of pipeline defect categorization, our approach has two significant limitations: its limited global search capability and slow convergence rate. Finally, the method is evaluated via experiments in which MFL signals obtained from an experimental platform and simulation signals are used. Practical results and a thorough comparative study of alternative approaches confirm the model's superiority.

Binocular camera calibration using a BP algorithm optimized by an improved subtraction-average-based optimizer

The BP algorithm exhibits drawbacks such as reduced precision and extended iteration duration during the process of calibrating a binocular camera system. A technique employing an improved subtraction-average-based optimizer to optimize the BP algorithm (ISABO-BP) is introduced to address these limitations and complete the calibration of the binocular camera. The pixel values of the corner in 2D space and the coordinate values in 3D space are used as the input and output for the BP algorithm, respectively. First, the logistic chaotic mapping is used to initialize the population to increase the diversity of the initial population. The piecewise mapping is employed to generate random values to replace the coefficient values in the individual position update formula, making the population distribution more uniform. The golden sine strategy is adopted to help particles escape from local optima. The ISABO-BP algorithm is utilized to perform the process of calibrating a binocular camera. Then, through numerical experiments, the mean calibration precision for the BP algorithm is 0.1501 and 0.0445 cm for the ISABO-BP algorithm. The calibration precision has been enhanced by 70.4%. The iterative steps of the BP algorithm are 353 epochs, while the ISABO-BP algorithm are 76 epochs, and the iterative speed is increased by nearly four times. It is evident that the ISABO-BP algorithm enhances the calibration precision and expedites the convergence rate. Finally, other intelligent algorithms to optimize the BP algorithm for calibrating a binocular camera are compared, and the superiority of this algorithm is also verified.

Online Learning and Decision Making Under Generalized Linear Model with High-Dimensional Data

We propose a minimax concave penalized multiarmed bandit algorithm under the generalized linear model (G-MCP-Bandit) for decision-makers facing high-dimensional data in an online learning and decision-making environment. We demonstrate that in the data-rich regime, the G-MCP-Bandit algorithm attains the optimal cumulative regret in the sample size dimension and a tight bound in the covariate dimension and the significant covariate dimension. In the data-poor regime, the G-MCP-Bandit algorithm maintains a tight regret upper bound. In addition, we develop a local linear approximation method, the two-step weighted Lasso procedure, to identify the minimax concave penalty (MCP) estimator for the G-MCP-Bandit algorithm when samples are not independent and identically distributed. Under this procedure, the MCP estimator can match the oracle estimator with high probability and converge to the true parameters at the optimal convergence rate. Finally, through experiments based on both synthetic and real data sets, we show that the G-MCP-Bandit algorithm outperforms other benchmarking algorithms in terms of cumulative regret and that the benefits of the G-MCP-Bandit algorithm increase in the data’s sparsity level and the size of the decision set. This paper was accepted by J. George Shanthikumar, big data analytics. Funding: This work was supported by the National Natural Science Foundation of China (NSFC) [Grants W2441021, 72371172, 72342023, and 71929101] and the National Science Foundation (NSF) [Grant DMS 1820702]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.01557 .

A Special Corner Element for Solving Heat Conduction Problems

A hybrid Trefftz finite element method is proposed for solving heat conduction problems in a domain with concave or convex angles. In this method, two temperature fields are independently assumed, one of which is the intra-element field defined inside the element and the other is the frame field defined on the boundary. The corner effects embed in the truncated complete solutions which are used to construct the internal interpolation function for approximating the intra-element field. It is able to accurately analyze the problems with particular corners. The key characteristics lie in the finite element formulation which is evaluated along the element boundary. Compared with the conventional FEM, the corner element does not lose any accuracy by using a small number of elements. Numerical examples demonstrate that the proposed method exhibits low sensitivity to mesh distortion. Compared with ABAQUS, the proposed method also exhibits higher accuracy and a faster convergence rate.

Prediction intervals with controlled length in the heteroscedastic Gaussian regression

We tackle the problem of building a prediction interval in heteroscedastic Gaussian regression. We focus on prediction intervals with constrained expected length in order to guarantee interpretability of the output. In this framework, we derive a closed-form expression of the optimal prediction interval that allows for the development of a data-driven prediction interval based on plug-in. The construction of the proposed algorithm is based on two samples, one labelled and another unlabelled. Under mild conditions, we show that our procedure is asymptotically as good as the optimal prediction interval both in terms of expected length and error rate. In particular, the control of the expected length is distribution-free. We also derive rates of convergence under smoothness and the Tsybakov noise conditions. We conduct a numerical analysis that exhibits the good performance of our method. It also indicates that even with a few amount of unlabelled data, our method is very effective in enforcing the length constraint.

Mean square consistency and improved rate of convergence of generalized subsampling estimator for non‐stationary time series

In this article, we show the mean square consistency for a generalized subsampling estimator based on the aggregation of the mean, median, and trimmed mean of some subsampling estimators for general non‐stationary time series. Consistency requires standard assumptions, including the existence of moments and ‐mixing conditions. We apply our results to the Fourier coefficients of the autocovariance function of periodically correlated time series. Furthermore, as in the i.i.d. case, we show that the generalized subsampling estimator satisfies Bernstein inequality and concentrates at an improved rate (under the condition of no or small bias) compared with the original estimator. Finally, we illustrate our results with some simulation data examples.

A Squared Smoothing Newton Method for Semidefinite Programming

This paper proposes a squared smoothing Newton method via the Huber smoothing function for solving semidefinite programming problems (SDPs). We first study the fundamental properties of the matrix-valued mapping defined upon the Huber function. Using these results and existing ones in the literature, we then conduct rigorous convergence analysis and establish convergence properties for the proposed algorithm. In particular, we show that the proposed method is well-defined and admits global convergence. Moreover, under suitable regularity conditions, that is, the primal and dual constraint nondegenerate conditions, the proposed method is shown to have a super-linear convergence rate. To evaluate the practical performance of the algorithm, we conduct extensive numerical experiments for solving various classes of SDPs. Comparison with the state-of-the-art SDP solvers demonstrates that our method is also efficient for computing accurate solutions of SDPs. Funding: The research of D. Sun was supported in part by the Hong Kong Research Grants Council under Grant 15307523, and the research of K.-C. Toh was supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 3 grant call [MOE-2019-T3-1-010].

Performance Analysis of ML and DL Models: Impact of Linear and Non-Linear Optimizers on Model Efficiency

Introduction: Emphasizing the effect of linear and non-linear optimizers on the performance of machine learning (ML) and deep learning (DL) models, this work addresses the CIFAR-10 image classification task. The work explores over multiple data sets how optimizers influence model behavior and efficiency. Objectives: With a view on five primary assessment metrics—accuracy, precision, recall, F1-score, and AUC the main purpose is to evaluate on SVM and CNN models the performance of linear optimizers (gradient descent, SGD) and non-linear optimizers (Adam, RMSProp). Methods: The study offers fair and complete comparison by way of a consistent evaluation strategy. Analyzing how different optimizer influences model convergence rates, computational cost, and training stability takes front stage in the experimental setting. Designed on CIFAR-10, optimizers are used on ML and DL models whose performance on all metrics is noted. Results: Results show that non-linear optimizers especially Adam much increase CNN model performance by means of faster convergence, higher classification accuracy, and better model stability. While helpful for simpler models, linear optimizers such SGD show slower convergence and limited adaptation with more complicated data. Conclusions: This study provides direction on selecting appropriate optimizers depending on model complexity, job needs, and computing restrictions coupled with important information on optimizing model training in image classification and other areas.

Simultaneous solution of implicitly defined curved, linear Timoshenko beams in two‐dimensional bulk domains

AbstractA novel formulation of curved Timoshenko beams which are defined by all level sets of a scalar function in a two‐dimensional bulk domain is presented, including a corresponding finite element method for the simultaneous solution of these beams. The required geometrical setup and differential operators are introduced and the model is defined in strong form. The weak form for the simultaneous solution involves the co‐area formula. A crucial aspect is how the difference vector, describing the rotational deformation in this model, is discretized wherefore two different approaches are shown in this paper. Higher‐order convergence studies in the residual errors confirm that the proposed method is valid and achieves expected convergence rates.

Quantitative Convergence of a Discretization of Dynamic Optimal Transport Using the Dual Formulation

AbstractWe present a discretization of the dynamic optimal transport problem for which we can obtain the convergence rate for the value of the transport cost to its continuous value when the temporal and spatial stepsize vanish. This convergence result does not require any regularity assumption on the measures, though experiments suggest that the rate is not sharp. Via an analysis of the duality gap we also obtain the convergence rates for the gradient of the optimal potentials and the velocity field under mild regularity assumptions. To obtain such rates, we discretize the dual formulation of the dynamic optimal transport problem and use the mature literature related to the error due to discretizing the Hamilton–Jacobi equation.

Model-Free Filter-Based Trajectory Tracking Controller for Two-Wheeled Vehicles Through Pole-Zero Cancellation Technique

Considering the nonlinear dynamics, this paper devises an advanced position trajectory tracking controller with a model-free filter for two-wheeled vehicle (TWV) applications. The proposed technique preserves a simple structure in the form of the proportional–integral (PI) controller involving the model-free filter and nonlinearly structured feedback gains, which make the following contributions: (a) the proposed filter smooths the position and yaw angle measurements according to the first-order convergence rate without any model information; and (b) the PI control with the nonlinearly structured feedback gains robustly stabilizes the position and yaw angle errors along the desired first-order system to accomplish the trajectory tracking mission, which is obtained by the pole-zero cancellation (PZC) in the presence of modeling errors. MATLAB/Simulink was used to emulate the resulting feedback system and validate the effectiveness of the proposed technique.