Barzilai Borwein Research Articles

Riemannian SVRG Using Barzilai–Borwein Method as Second-Order Approximation for Federated Learning

Riemannian SVRG Using Barzilai–Borwein Method as Second-Order Approximation for Federated Learning

A stochastic ridesharing user equilibrium model with origin-destination-based ride-matching strategy

Ridesharing, as an emerging mode of modern urban transportation, has gained widespread popularity due to its ability to improve convenience and reduce the expenses in daily travel. To capture its impact on the network-wide traffic flow pattern, this paper develops a new variational inequality (VI) formulation for modeling the stochastic ridesharing user equilibrium (SRUE) problem considering an origin–destination (O-D) -based ride-matching strategy. The proposed SRUE problem is formulated based on a hierarchical logit choice model to depict the multi-stage decision-making (i.e., role choice, ride-matching, and route choice) of the ridesharing travelers. The SRUE model also reflects a two-sided market equilibrium considering the interaction effect between the ridesharing market and the road traffic equilibrium market. The logit choice model captures the ride-matching problem between the drivers and the passengers traveling on the same/different O-D pairs. Moreover, the lower- and upper-bound constraints for a ridesharing driver picking up passengers are established at the O-D level. For solving the SRUE model with O-D-based ride-matching constraints, the Lagrangian dual method (LDM) is adopted by incorporating the Barzilai-Borwein (BB) step size to update the dual variables, which accelerates the convergence of the LDM. Three numerical examples are conducted to illustrate the features of the proposed SRUE model and demonstrate the performance of the solution algorithm.

Parameter identification method of information propagation models based on different network structures

In this paper, we create the rumor propagation model with diffusion behavior by considering the state of the rumor in both the time dimension and the space dimension comprehensively. Meanwhile, we demonstrate the reaction–diffusion model using Turing patterns after determining the prerequisites for their occurrence. In order to achieve the purpose of predicting and controlling rumors in time, we choose to utilize the parameter identification technique based on the Barzilai–Borwein (BB) algorithm and the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm. In the numerical simulation section, we first investigate how the rumor avoidance rate and cross-diffusion coefficients affect the propagation of rumors. Then, based on a continuous spatio-temporal system and complex network system, respectively, we perform parameter identification for the propagation model. We thoroughly examine how the type of algorithm, the quantity of unknown parameters, and the network structure affect the identification outcomes in terms of the cost function, error curve, and program function time. When the model constructed in this paper is used for parameter identification on different network structures, the error gap between the final value and the target value is not significant. However, the cost function and time consumption for parameter identification on complex networks are much smaller than on the continuous medium.

Barzilai Borwein Incremental Grey Polynomial Regression for train delay prediction

AbstractThe swift societal evolution and ceaseless advancement of human value of life have been set forth for reliability as well as rapidity of railway transportation. Latest advances in machine learning approaches as well as surging accessibility of numerous information sources is produced state‐of‐the‐art probabilities for significant, precise train delay identification. In this method called, Barzilai Borwein Incremental Grey Polynomial Regression (BBI‐GPR) is introduced for predicting train arrival/departure delays, which utilized for later delay management in an accurate manner with this method comprised into three sections such as, pre‐processing, feature selection and classification. First, with the raw ETA train delay dataset as input, Barzilai–Borwein Feature Rescaling‐based Pre‐processing is applied to model computationally efficient feature rescaled and normalized values. Second with processed features as input, Incremental Maximum Relevance Minimum Redundant‐based Feature Selection is applied to select error minimized optimal features. Finally, with optimal features selected as input, Grey Polynomial Regression‐based Prediction algorithm is employed to analyse train delay. For confirming proposed BBI‐GPR, as well as analyse its performance, compare standard train delay prediction method with existing machine learning‐based regression method. Results show that new variants outperform existing train delay prediction method by minimizing train delay prediction time, error rate by 25% and 27% respectively, with improved accuracy rate of 7%, therefore paving ways for efficient train delay prediction.

Stabilized BB projection algorithm for large-scale convex constrained nonlinear monotone equations to signal and image processing problems

The Barzilai–Borwein (BB) method is a popular and efficient gradient method for solving large-scale unconstrained optimization problems. In general, it converges much faster than the steepest descent (Cauchy) method. However, it may not converge, even when the objective function is strongly convex. To overcome this, by virtue of bounding the distance between sequential iterates, [Burdakov et al. (2019)] proposed a stabilized BB method. In this paper, by combining the stabilized BB method with a projection approach, we develop a stabilized BB projection (SBBP) method to solve nonlinear monotone equations with convex constraints. SBBP does not involve computing matrices proving the usefulness for solving large-scale problems. Its global convergence and Q-linear convergence rate are also established. We report numerical experiments on recovering sparse signals and restoring blurred images arising from compressive sensing, demonstrating the usefulness of SBBP.

Network traffic recovery from link-load measurements using tensor triple decomposition strategy for third-order traffic tensors

Network traffic data is the pivot of input in many network tasks but its direct measurement can be insufferably costly. In this paper, we propose a network traffic recovery method which only requires the conveniently measurable link-load traffics. We arrange the traffic data as a third-order tensor and utilize the triple decomposition technique proposed very recently by Qi et al. (2021). The studied model is a differentialble unconstrained minimization problem, which can be efficiently solved by a Barzilai–Borwein (BB) gradient algorithm. We prove that the generated iteration sequence can globally converge to a certain stationary point of the objective function. The numerical simulations on three open-source traffic datasets demonstrate the superiority of our method in comparison with other state-of-the-art algorithms.

A hybrid complex spectral conjugate gradient learning algorithm for complex-valued data processing

Complex-valued neural networks (CVNNs) have become a powerful modelling tool for complex-valued data processing. Because most of the critical points of CVNNs are saddle points, the gradient-based learning algorithms for CVNNs enjoy more chances to reach the global minima while suffering from slow convergence. To this end, we propose a hybrid complex spectral conjugate gradient learning algorithm for fast training CVNNs in this paper. The proposed algorithm combines the scaled negative gradient with a Barzilai–Borwein stepsize and an optimized conjugate term to define a new training direction, thus providing an accurate approximation of the second-order curvature of the objective function. The complex Wolfe conditions are employed to adaptively determine the optimal training stepsize. Under mild conditions, the descent property of the training direction and the convergence of the proposed algorithm are theoretically established. Simulation results on a number of benchmark complex-valued data processing problems demonstrate the efficiency of the proposed algorithm.

Barzilai–Borwein-like rules in proximal gradient schemes for ℓ1-regularized problems

We propose a novel steplength selection rule in proximal gradient methods for minimizing the sum of a differentiable function plus an ℓ 1 -norm penalty term. The proposed rule modifies one of the classical Barzilai–Borwein steplength, extending analogous results obtained in the context of gradient projection methods for constrained optimization. We analyse the spectral properties of the Barzilai–Borwein-like steplength when the differentiable part is quadratic, showing that its reciprocal lies in the spectrum of the submatrix of the Hessian that depends on both the nonzero and the nonoptimal zero components of the current iterate, allowing for acceleration effects when the optimal zero components start to be identified. Furthermore, we insert the modified rule into a proximal gradient method with a nonmonotone line search, for which we prove global convergence towards a stationary point. Numerical experiments show the ability of the proposed rule to sweep the spectrum of the reduced Hessian on a series of quadratic ℓ 1 -regularized problems, as well as its effectiveness in recovering the ground truth in a least squares regularized problem arising in image restoration.

An FFT-based homogenization scheme for cohesive zones with an application to adhesives and the core material of thin metal sandwich plates

Cohesive Zone Models with finite thickness are widely used for the fracture mechanical modeling of material layers, e.g., adhesive layers. Within this approach, the whole layer is modeled as a cohesive zone. Moreover, computational homogenization techniques are crucial for the development of advanced engineering materials, which are often heterogeneous. Compared to the commonly used Finite Element Method (FEM), solvers based on the Fast Fourier Transform (FFT) are expected to reduce the computational effort needed for the homogenization. Originated from an existing method for the computational homogenization of cohesive zones using FEM, a novel FFT-based homogenization scheme for cohesive zone models is presented. Our implementation of the FFT solver uses a displacement-based Barzilai–Borwein scheme and a non-local ductile damage model for the fracture behavior. Finally, the practical application of the method is discussed using an adhesive layer and the core material of HybrixTM metal sandwich plates as examples.

SARAH-M: A fast stochastic recursive gradient descent algorithm via momentum

As a simple but effective way, the momentum method has been widely adopted in stochastic optimization algorithms for large-scale machine learning problems and the success of stochastic optimization with the momentum term for many applications in machine learning and other related areas has been reported everywhere. However, the understanding of how the momentum improves the performance of modern variance reduced stochastic gradient algorithms, e.g., the stochastic dual coordinate ascent average gradient (SDCA) method, the stochastically controlled stochastic gradient (SCSG) method, the stochastic recursive gradient algorithm (SARAH), etc., is still limited. To tackle this issue, this work studies the performance of SARAH with the momentum term theoretically and empirically, and develops a novel variance reduced stochastic gradient algorithm, termed as SARAH-M. We rigorously prove that SARAH-M attains a linear rate of convergence for minimizing the strongly convex function. We further propose an adaptive SARAH-M method (abbreviated as AdaSARAH-M) by incorporating the random Barzilai–Borwein (RBB) technique into SARAH-M, which provides an easy way to determine the step size for the original SARAH-M algorithm. The theoretical analysis that shows AdaSARAH-M with a linear convergence speed is also provided. Moreover, we show that the complexity of the proposed algorithms can outperform modern stochastic optimization algorithms. Finally, the numerical results, compared with state-of-the-art algorithms on benchmarking machine learning problems, verify the efficacy of the momentum in variance reduced stochastic gradient algorithms.

A mini-batch algorithm for large-scale learning problems with adaptive step size

The step size selection in stochastic optimization methods is a crucial aspect that holds significant importance in theoretical analysis and practical applications. We propose two stochastic optimization methods based on the competitive Barzilai-Borwein (BB) step size in both the inner and outer loops of the mini-batch semi-stochastic gradient descent (mS2GD) algorithm. The competitive BB step size is automatically updated using the latest and most accurate secant equation. We introduce two algorithms: mS2GD-CBB, which updates the step size in the outer loop, and mS2GD-RCBB, which updates the step size in the inner loop. We evaluate the performance of the proposed algorithms on the classical optimization problem and compare them against existing methods. Experimental results demonstrate that the methods exhibit favorable convergence properties and can effectively handle the challenges of big data arising from the fields of signal processing, statistics, and machine learning. The methods offer enhanced adaptability, flexibility, and exploration capabilities, making them promising approaches for a wide range of optimization problems.

On Kahan's automatic step-size control and an anadromic gradient descent iteration

Accelerating the basic gradient iteration is a critical issue in numerical optimization and deep-learning. Recently, William Kahan [Kahan, Automatic Step-Size Control for Minimization Iterations, Technical report, University of California, Berkeley CA, USA, 2019] proposed automatic step-size strategies for the gradient descent iteration without explicitly using any prior information of the Hessian; moreover, a new momentum-based gradient acceleration method, namely, an Anadromic Gradient Descent (AGD) iteration, was proposed. Besides the capability of accelerating the basic gradient descent iteration, AGD enables the iteration to return to the past iterates by just reversing steps in sign and order in the same updating formulation. Numerical performances of the automatic step-size strategies and AGD are claimed to be favourable. In this paper, for the quadratic model, through a revisit of some classical momentum-based gradient methods, we perform new analysis on the convergence and their optimal hyper-parameters. We also investigate the convergence behaviour of AGD with the optimal hyper-parameters and connect one Kahan's automatic step-size scheme with the long Barzilai–Borwein step-size. Numerical results are presented to reflect the theoretical convergence behaviours and demonstrate the practical performances of various momentum-based gradient methods.

Void fraction measurement of gas–liquid two-phase flow based on compressed sensing theory and capacitively coupled electrical resistance tomography system

Capacitively coupled electrical resistance tomography (CCERT) is an innovative technique for electrical resistance tomography based on capacitively coupled contactless conductivity detection (C4D). Despite its potential, there are only a few studies on image reconstruction algorithms for CCERT. To address this, a CCERT measurement system is developed, and the compressed sensing (CS) theory is applied to the inverse problem imaging of CCERT to improve the image reconstruction quality and speed. Firstly, a mathematical CCERT image reconstruction model is constructed under CS theory and three algorithms under CS theory are employed to solve the convex optimization problem of the reconstruction model. Then a thresholding operation is applied to obtain the post-processed image and compared it with the classical linear back projection and Landweber algorithms. The simulation results demonstrate that the Barzilai–Borwein gradient projection for sparse reconstruction (GPSR-BB) algorithm yields more satisfactory imaging results than the other four algorithms. Additionally, three sensitivity matrix optimization methods for GPSR-BB algorithm are compared and find that the new method of screening the rows of the sensitivity matrix to zero is more effective than the other two common optimization methods in terms of reconstructed image quality. Finally, the static experiments of void fraction measurement are conducted using the developed CCERT system. The results indicated that the absolute error of void fraction measurement by GPSR-BB algorithm was less than 6.59% in the range of void fraction from 0.90% to 66.29%.

A Systematic Approach for the Calibration of Route Choice Models Based on Stochastic User Equilibrium

Different from the traditional deterministic user equilibrium (DUE) model based on fixed travel time, the stochastic user equilibrium (SUE) model is established by assuming the perceived route travel time as a random variable. It plays an important role in traffic assignment, wherein the setting of route choice parameters decides system performance. This article studies parameter calibration of widely used SUE models including the multinomial logit (MNL), length-based C-logit (LCL), and congestion-based C-logit (CCL) models. We establish a bilevel model in which the upper level is designed to minimize the error between the real and model-based traffic flow patterns, whereas the lower level copes with the SUE traffic assignment. A series of innovative methods are built to solve this bilevel model. Considering the nonlinear calibration problem in the upper level, a Bayesian optimization method is used. For the SUE traffic assignment in the lower level, dedicated algorithms are proposed, which combine a partial linearization algorithm and a feasible direction method. Furthermore, two revised solution methods are provided: a Barzilai–Borwein (BB) step-size method and a parallel framework. Finally, the proposed model and corresponding solution approach are validated based on a case study of Yuyao, China. Solutions show that the CCL SUE model has the highest precision while requiring the longest computation time. LCL takes only 11.7% of the computation time of CCL when reaching an acceptable precision.

Total variation method based on modified Barzilai–Borwein algorithm to noise reduction in MRI images

Total variation method based on modified Barzilai–Borwein algorithm to noise reduction in MRI images

Action Recognition via Adaptive Semi-Supervised Feature Analysis

This study presents a new semi-supervised action recognition method via adaptive feature analysis. We assume that action videos can be regarded as data points in embedding manifold subspace, and their matching problem can be quantified through a specific Grassmannian kernel function while integrating feature correlation exploration and data similarity measurement into a joint framework. By maximizing the intra-class compactness based on labeled data, our algorithm can learn multiple features and leverage unlabeled data to enhance recognition. We introduce the Grassmannian kernels and the Projected Barzilai–Borwein (PBB) method to train a subspace projection matrix as a classifier. Experiment results show our method has outperformed the compared approaches when a few labeled training samples are available.

Adaptive learning rate optimization algorithms with dynamic bound based on Barzilai-Borwein method

The training effect of the neural network model is directly influenced by optimization algorithms. The Barzilai-Borwein(BB) method is used in the stochastic gradient descent (SGD) and other deep learning optimization algorithms because of its outstanding performance in terms of convergence speed. In order to improve the stability of BB step size and the output quality of network model in deep learning, this paper presents two improved optimization algorithms based on the BB method: BBbound and AdaBBbound. BBbound reduces the floating BB step size by generating the upper bound of the step in the current iteration. It avoids the occasional occurrence of long steps; AdaBBbound is an adaptive gradient method based on BB method with dynamic bound. The algorithm has a fast convergence effect in the early stage, then calculates learning rate and smoothly transitions to SGD by setting the better conditions of the BB method. We analyzed the performance of the two algorithms based on the initial conditions and the learning rate change curve. Meanwhile, we tested our algorithms on popular network models such as ResNet and DenseNet. The results showed that the new optimization algorithms have achieved high stability and significantly improved the performance of the network model.

A harmonic framework for stepsize selection in gradient methods

We study the use of inverse harmonic Rayleigh quotients with target for the stepsize selection in gradient methods for nonlinear unconstrained optimization problems. This not only provides an elegant and flexible framework to parametrize and reinterpret existing stepsize schemes, but it also gives inspiration for new flexible and tunable families of steplengths. In particular, we analyze and extend the adaptive Barzilai–Borwein method to a new family of stepsizes. While this family exploits negative values for the target, we also consider positive targets. We present a convergence analysis for quadratic problems extending results by Dai and Liao (IMA J Numer Anal 22(1):1–10, 2002), and carry out experiments outlining the potential of the approaches.

Trace minimization method via penalty for linear response eigenvalue problems

<p style='text-indent:20px;'>In various applications, such as the computation of energy excitation states of electrons and molecules, and the analysis of interstellar clouds, the linear response eigenvalue problem, which is a special type of the Hamiltonian eigenvalue problem, is frequently encountered. However, traditional eigensolvers may not be applicable to this problem owing to its inherently large scale. In fact, we are usually more interested in computing some of the smallest positive eigenvalues. To this end, a trace minimum principle optimization model with orthogonality constraint has been proposed. On this basis, we propose an unconstrained surrogate model called trace minimization via penalty, and we establish its equivalence with the original constrained model, provided that the penalty parameter is larger than a certain threshold. By avoiding the orthogonality constraint, we can use a gradient-type method to solve this model. Specifically, we use the gradient descent method with Barzilai–Borwein step size. Moreover, we develop a restarting strategy for the proposed algorithm whereby higher accuracy and faster convergence can be achieved. This is verified by preliminary experimental results.</p>

Modeling the capacity of multimodal and intermodal urban transportation networks that incorporate emerging travel modes

Given the increasing prevalence of new urban transport modes such as ridesharing, e-hailing, and combined transport, it is essential to evaluate their effects on the capacity of transportation networks. Hence, this paper develops a novel transportation network capacity model to capture the travel behaviors of inter-multimodal mobility in an urban transportation system that incorporates emerging travel modes. The novel model is formulated as a bi-level programming problem, in which the lower-level model is a combined modal split and traffic assignment (CMSTA) problem based on mathematical programming. The CMSTA problem adopts the cross-nested logit model to account for intermodal travel behavior in the modal split phase and the path-sized logit model to account for route overlap in the traffic assignment phase. Moreover, the logit-based trip distribution model is used to capture the dispatch of the e-hailing traffic flow and the matching of ridesharing drivers with passengers. Besides, we consider flow interactions (e.g., cars and buses sharing the same link) in the road network. We customize a solution framework for solving this novel model that adopts the recently developed fast path-based algorithm with the Barzilai–Borwein stepsize strategy to efficiently solve the CMSTA problem, and derive a sensitivity analysis-based (SAB) algorithm to solve the entire bi-level programming problem. The effectiveness of the novel model is verified in numerical experiments that demonstrate the effects of intermodal transportation, e-hailing, and ridesharing on the capacity of a multimodal transportation network.