Borwein Stepsize Research Articles

A hybrid complex spectral conjugate gradient learning algorithm for complex-valued data processing

Complex-valued neural networks (CVNNs) have become a powerful modelling tool for complex-valued data processing. Because most of the critical points of CVNNs are saddle points, the gradient-based learning algorithms for CVNNs enjoy more chances to reach the global minima while suffering from slow convergence. To this end, we propose a hybrid complex spectral conjugate gradient learning algorithm for fast training CVNNs in this paper. The proposed algorithm combines the scaled negative gradient with a Barzilai–Borwein stepsize and an optimized conjugate term to define a new training direction, thus providing an accurate approximation of the second-order curvature of the objective function. The complex Wolfe conditions are employed to adaptively determine the optimal training stepsize. Under mild conditions, the descent property of the training direction and the convergence of the proposed algorithm are theoretically established. Simulation results on a number of benchmark complex-valued data processing problems demonstrate the efficiency of the proposed algorithm.

On Kahan's automatic step-size control and an anadromic gradient descent iteration

Accelerating the basic gradient iteration is a critical issue in numerical optimization and deep-learning. Recently, William Kahan [Kahan, Automatic Step-Size Control for Minimization Iterations, Technical report, University of California, Berkeley CA, USA, 2019] proposed automatic step-size strategies for the gradient descent iteration without explicitly using any prior information of the Hessian; moreover, a new momentum-based gradient acceleration method, namely, an Anadromic Gradient Descent (AGD) iteration, was proposed. Besides the capability of accelerating the basic gradient descent iteration, AGD enables the iteration to return to the past iterates by just reversing steps in sign and order in the same updating formulation. Numerical performances of the automatic step-size strategies and AGD are claimed to be favourable. In this paper, for the quadratic model, through a revisit of some classical momentum-based gradient methods, we perform new analysis on the convergence and their optimal hyper-parameters. We also investigate the convergence behaviour of AGD with the optimal hyper-parameters and connect one Kahan's automatic step-size scheme with the long Barzilai–Borwein step-size. Numerical results are presented to reflect the theoretical convergence behaviours and demonstrate the practical performances of various momentum-based gradient methods.

A Weibit-Based sequential transit assignment model based on hyperpath graph and generalized extreme value network representation

This study aims to develop a weibit-based stochastic transit assignment model for situations in which passengers’ travel choice strategies are stochastic. The weibit choice model, which was originally developed for the stochastic traffic assignment problem, is adapted using a hyperpath graph network representation. Unlike the traditional logit model, the weibit model can account for the homogeneous perception variance problem. Besides, a joint choice network representation originated from the network generalized extreme value (N-GEV) model, known as the GEV network, is adopted to depict the correlation among the arcs within the hypergraph consisting of all efficient hyperpaths. The GEV network uses a link-based parameter, the degree of membership, to capture the overlap effect, which is thus well incorporated with the link-based transit assignment approach. For implementation, a link-based stochastic network loading approach is proposed for solving the weibit-based transit assignment model in the sequential process. This sequential assignment approach does not need to enumerate and store the transit hyperpaths in the solution process. This model is then extended to a stochastic transit equilibrium model under congestion conditions, which is formulated as a fixed-point problem. Thus, a recent fast line-search scheme, i.e., the Barzilai and Borwein step-size, is adopted for solving the stochastic transit equilibrium model. Numerical examples are provided to illustrate the features of the proposed weibit-based stochastic transit assignment model. A real transit network is used to demonstrate the algorithm efficiency and the detailed results of the weibit-based transit equilibrium model.

Modeling the capacity of multimodal and intermodal urban transportation networks that incorporate emerging travel modes

Given the increasing prevalence of new urban transport modes such as ridesharing, e-hailing, and combined transport, it is essential to evaluate their effects on the capacity of transportation networks. Hence, this paper develops a novel transportation network capacity model to capture the travel behaviors of inter-multimodal mobility in an urban transportation system that incorporates emerging travel modes. The novel model is formulated as a bi-level programming problem, in which the lower-level model is a combined modal split and traffic assignment (CMSTA) problem based on mathematical programming. The CMSTA problem adopts the cross-nested logit model to account for intermodal travel behavior in the modal split phase and the path-sized logit model to account for route overlap in the traffic assignment phase. Moreover, the logit-based trip distribution model is used to capture the dispatch of the e-hailing traffic flow and the matching of ridesharing drivers with passengers. Besides, we consider flow interactions (e.g., cars and buses sharing the same link) in the road network. We customize a solution framework for solving this novel model that adopts the recently developed fast path-based algorithm with the Barzilai–Borwein stepsize strategy to efficiently solve the CMSTA problem, and derive a sensitivity analysis-based (SAB) algorithm to solve the entire bi-level programming problem. The effectiveness of the novel model is verified in numerical experiments that demonstrate the effects of intermodal transportation, e-hailing, and ridesharing on the capacity of a multimodal transportation network.

A gradient method exploiting the two dimensional quadratic termination property

The quadratic termination property is important to the efficiency of gradient methods. We consider equipping a family of gradient methods, where the stepsize is given by the ratio of two norms, with two dimensional quadratic termination. Such a desired property is achieved by cooperating with a new stepsize which is derived by maximizing the stepsize of the considered family in the next iteration. It is proved that each method in the family will asymptotically alternate in a two dimensional subspace spanned by the eigenvectors corresponding to the largest and smallest eigenvalues. Based on this asymptotic behavior, we show that the new stepsize converges to the reciprocal of the largest eigenvalue of the Hessian. Furthermore, by adaptively taking the long Barzilai–Borwein stepsize and reusing the new stepsize with retard, we propose an efficient gradient method for unconstrained quadratic optimization. We prove that the new method is R-linearly convergent with a rate of \(1-1/\kappa\), where \(\kappa\) is the condition number of Hessian. Numerical experiments show the efficiency of our proposed method.

A Spectral Gradient Projection Method for the Positive Semi-definite Procrustes Problem

This paper addresses the positive semi-deffnite procrustes problem (PSDP). The PSDP corresponds to a least squares problem over the set of symmetric and semi-deffnite positive matrices. These kinds of problems appear in many applications such as structure analysis, signal processing, among others. A non-monotone spectral projected gradient algorithm is proposed to obtain a numerical solution for the PSDP. The proposed algorithm employs the Zhang and Hager's non-monotone technique in combination with the Barzilai and Borwein's step size to accelerate convergence. Some theoretical results are presented. Finally, numerical experiments are performed to demonstrate the effectiveness and efficiency of the proposed method, and comparisons are made with other state-of-the-art algorithms.

Joint Visual and Semantic Optimization for zero-shot learning

Zero-shot learning (ZSL) aims to classify instances whose classes could be unseen during training. Most existing ZSL methods project visual or semantic features into the space of the other one, or into a common subspace. The main goal of projection is to find out the similar features in the latent subspace. However, existing methods barely consider common features that preserve knowledge, here we refer to these features as the shared concepts, which are essential to model the relationship between the visual and semantic spaces. In this paper, we exploit the underlying concepts shared by both visual and semantic features in a latent common subspace and propose to match their latent visual and semantic representations. To reduce domain shift and information loss, we introduce reconstruction losses for both visual and semantic features. As a result, the reconstruction regularizations are added to the similar features and thereby obtain knowledge preserving shared concepts via the proposed method. Mathematically, it is formulated as the minimization problem for mutual orthogonal projection to their latent common subspace. The problem involves two projection variables, thus we develop an algorithm based on the Gauss–Seidel iteration scheme and split the problem into two subproblems in the scheme. These two subproblems are further solved by searching algorithms based on the Barzilai–Borwein stepsize. Extensive experiments on six benchmark data sets are conducted to demonstrate that the accuracy of the proposed method is better than that of existing ZSL methods.

A Family of Modified Spectral Projection Methods for Nonlinear Monotone Equations with Convex Constraint

In this paper, we propose a family of modified spectral projection methods for nonlinear monotone equations with convex constraints, where the spectral parameter is mainly determined by a convex combination of the modified long Barzilai–Borwein stepsize and the modified short Barzilai–Borwein stepsize. We obtain a trial point by the spectral method and then get the iteration point by the projection technique. The algorithm can generate a bounded iterative sequence automatically, and we obtain the global convergence of the proposed method in the sense that every limit point is a solution of the nonlinear equation. The proposed method can be used to resolve the large-scale nonlinear monotone equations with convex constraints including smooth and nonsmooth equations. Numerical results for nonlinear equation problems and the ℓ 1 -norm regularization problem in compressive sensing demonstrate the efficiency and efficacy of our method.

Multiple-Priors Ensemble Constrained Nonnegative Matrix Factorization for Spectral Unmixing

Nonnegative matrix factorization (NMF) is widely used in unmixing issue in recent years, because it can simultaneously estimate the endmembers and abundances. However, most existing NMF-based methods only consider single matrix constraints and the other one is ignored. In fact, due to the influence of various noise, the regularization effectiveness based on the single matrix constraint method may be limited. In addition, hyperspectral images contain a variety of prior information, while many approaches usually only consider one of the priors , and the synergistic effect of multiple priors unions and two matrix joint constraints is neglected. In this article, to overcome this limitation, we propose a new blind unmixing scheme, called multiple- priors ensemble constrained NMF. The article first analyses the HSI intrinsic feature priors from both geometric and statistical aspects, and three important priors learners are defined. Then, three learners are jointly introduced into the NMF model and work together for the first time to impose constraints on both the endmember and the abundance matrix. In order to effectively solve the proposed model, Barzilai–Borwein stepsize strategy accelerates optimization algorithm is developed by using the variable splitting and augmented Lagrangian framework. The effectiveness and superiority of the proposed method are demonstrated by comparing with other advanced approaches on both synthetic and real world datasets.

New adaptive Barzilai--Borwein step size and its application in solving large-scale optimization problems

We propose a new adaptive and composite Barzilai–Borwein (BB) step size by integrating the advantages of such existing step sizes. Particularly, the proposed step size is an optimal weighted mean of two classical BB step sizes and the weights are updated at each iteration in accordance with the quality of the classical BB step sizes. Combined with the steepest descent direction, the adaptive and composite BB step size is incorporated into the development of an algorithm such that it is efficient to solve large-scale optimization problems. We prove that the developed algorithm is globally convergent and it R-linearly converges when applied to solve strictly convex quadratic minimization problems. Compared with the state-of-the-art algorithms available in the literature, the proposed step size is more efficient in solving ill-posed or large-scale benchmark test problems. doi:10.1017/S1446181118000263

New gradient methods for sensor selection problems

In this article, we consider the sensor selection problem of choosing [Formula: see text] sensors from a set of [Formula: see text] possible sensor measurements. The sensor selection problem is a combinational optimization problem. Evaluating the performance for each possible combination is impractical unless [Formula: see text] and [Formula: see text] are small. We relax the original selection problem to be a convex optimization problem and describe a projected gradient method with Barzilai–Borwein step size to solve the proposed relaxed problem. Numerical results demonstrate that the proposed algorithm converges faster than some classical algorithms. The solution obtained by the proposed algorithm is closer to the truth.

Variable step size methods for solving simultaneous algebraic reconstruction technique (SART)-type cbct reconstructions.

Compared to analytical reconstruction by Feldkamp-Davis-Kress (FDK), simultaneous algebraic reconstruction technique (SART) offers a higher degree of flexibility in input measurements and often produces superior quality images. Due to the iterative nature of the algorithm, however, SART requires intense computations which have prevented its use in clinical practice. In this paper, we developed a fast-converging SART-type algorithm and showed its clinical feasibility in CBCT reconstructions. Inspired by the quasi-orthogonal nature of the x-ray projections in CBCT, we implement a simple yet much faster algorithm by computing Barzilai and Borwein step size at each iteration. We applied this variable step-size (VS)-SART algorithm to numerical and physical phantoms as well as cancer patients for reconstruction. By connecting the SART algebraic problem to the statistical weighted least squares problem, we enhanced the reconstruction speed significantly (i.e., less number of iterations). We further accelerated the reconstruction speed of algorithms by using the parallel computing power of GPU.

A New Modified Barzilai–Borwein Gradient Method for the Quadratic Minimization Problem

A new modified Barzilai---Borwein gradient method for solving the strictly convex quadratic minimization problem is proposed by properly changing the Barzilai---Borwein stepsize such that some cert...

Optimised projections for generalised distributed compressed sensing

Different signals from the various sensors of the same scene form an ensemble. Distributed compressed sensing (DCS) rests on a new concept called the joint sparsity of the ensemble. JSM-1 is a model that describes the joint sparsity by one dictionary. Previously, the generalisation of JSM-1 was proposed where the signal ensemble depends on two dictionaries. Its compressed sensing (CS) version is considered: generalised DCS (GDCS). Instead of using random projections (random Gaussian (rGauss)), a gradient method with Barzilai–Borwein stepsize (GBB) is developed to optimise the projections in the GDCS. It enhances the reconstruction performance of the GDCS. It is verified by some experiments on the synthesised signals.