Orthogonal Greedy Algorithm Research Articles

Threshold Estimation via Group Orthogonal Greedy Algorithm

A threshold autoregressive (TAR) model is an important class of nonlinear time series models that possess many desirable features such as asymmetric limit cycles and amplitude-dependent frequencies. Statistical inference for the TAR model encounters a major difficulty in the estimation of thresholds, however. This article develops an efficient procedure to estimate the thresholds. The procedure first transforms multiple-threshold detection to a regression variable selection problem, and then employs a group orthogonal greedy algorithm to obtain the threshold estimates. Desirable theoretical results are derived to lend support to the proposed methodology. Simulation experiments are conducted to illustrate the empirical performances of the method. Applications to U.S. GNP data are investigated.

The rate of convergence of weak greedy approximations over orthogonal dictionaries

The convergence rate of a weak orthogonal greedy algorithm is studied for the subspace l1 ⊂ l2 and orthogonal dictionaries. It is shown that general results on convergence rate of weak orthogonal greedy algorithms can be essentially improved in the studied case. It is also shown that this improvement is asymptotically sharp.

Learning capability of the truncated greedy algorithm

Pure greedy algorithm (PGA), orthogonal greedy algorithm (OGA) and relaxed greedy algorithm (RGA) are three widely used greedy type algorithms in both nonlinear approximation and supervised learning. in this paper, we apply another variant of greedy-type algorithm, called the truncated greedy algorithm (TGA) in the realm of supervised learning and study its learning performance. We rigorously prove that TGA is better than PGA in the sense that TGA possesses the faster learning rate than PGA. Furthermore, in some special cases, we also prove that TGA outperforms OGA and RGA. All these theoretical assertions are verified by both toy simulations and real data experiments.

Two‐dimensional adaptive Fourier decomposition

One‐dimensional adaptive Fourier decomposition, abbreviated as 1‐D AFD, or AFD, is an adaptive representation of a physically realizable signal into a linear combination of parameterized Szegö and higher‐order Szegö kernels of the context. In the present paper, we study multi‐dimensional AFDs based on multivariate complex Hardy spaces theory. We proceed with two approaches of which one uses Product‐TM Systems; and the other uses Product‐Szegö Dictionaries. With the Product‐TM Systems approach, we prove that at each selection of a pair of parameters, the maximal energy may be attained, and, accordingly, we prove the convergence. With the Product‐Szegö dictionary approach, we show that pure greedy algorithm is applicable. We next introduce a new type of greedy algorithm, called Pre‐orthogonal Greedy Algorithm (P‐OGA). We prove its convergence and convergence rate estimation, allowing a weak‐type version of P‐OGA as well. The convergence rate estimation of the proposed P‐OGA evidences its advantage over orthogonal greedy algorithm (OGA). In the last part, we analyze P‐OGA in depth and introduce the concept P‐OGA‐Induced Complete Dictionary, abbreviated as Complete Dictionary. We show that with the Complete Dictionary P‐OGA is applicable to the Hardy H2 space on 2‐torus. Copyright © 2016 John Wiley & Sons, Ltd.

Prediction-based Termination Rule for Greedy Learning with Massive Data.

The appearance of massive data has become increasingly common in contemporary scientific research. When sample size n is huge, classical learning methods become computationally costly for the regression purpose. Recently, the orthogonal greedy algorithm (OGA) has been revitalized as an efficient alternative in the context of kernel-based statistical learning. In a learning problem, accurate and fast prediction is often of interest. This makes an appropriate termination crucial for OGA. In this paper, we propose a new termination rule for OGA via investigating its predictive performance. The proposed rule is conceptually simple and convenient for implementation, which suggests an [Formula: see text] number of essential updates in an OGA process. It therefore provides an appealing route to conduct efficient learning for massive data. With a sample dependent kernel dictionary, we show that the proposed method is strongly consistent with an [Formula: see text] convergence rate to the oracle prediction. The promising performance of the method is supported by both simulation and real data examples.

Learning and approximation capabilities of orthogonal super greedy algorithm

We consider the approximation capability of orthogonal super greedy algorithms (OSGA) and its applications in supervised learning. OSGA focuses on selecting more than one atoms in each iteration, which, of course, reduces the computational burden when compared with the conventional orthogonal greedy algorithm (OGA). We prove that even for function classes that are not the convex hull of the dictionary, OSGA does not degrade the approximation capability of OGA, provided the dictionary is incoherent. Based on this, we deduce tight generalization error bounds for OSGA learning. Our results show that in the realm of supervised learning, OSGA provides a possibility to further reduce the computational burden of OGA on the premise of maintaining its prominent generalization capability.

Lebesgue-Type Inequalities for Greedy Approximation in Banach Spaces

We study sparse representations and sparse approximations with respect to incoherent dictionaries. We address the problem of designing and analyzing greedy methods of approximation. A key question in this regard is: How to measure efficiency of a specific algorithm? Answering this question, we prove the Lebesgue-type inequalities for algorithms under consideration. A very important new ingredient of the paper is that we perform our analysis in a Banach space instead of a Hilbert space. It is known that in many numerical problems, users are satisfied with a Hilbert space setting and do not consider a more general setting in a Banach space. There are known arguments that justify interest in Banach spaces. In this paper, we give one more argument in favor of consideration of greedy approximation in Banach spaces. We introduce a concept of M-coherent dictionary in a Banach space which is a generalization of the corresponding concept in a Hilbert space. We analyze the quasi-orthogonal greedy algorithm (QOGA), which is a generalization of the orthogonal greedy algorithm (orthogonal matching pursuit) for Banach spaces. It is known that the QOGA recovers exactly S -sparse signals after S iterations provided S <; (1+1/M)/2. This result is well known for the orthogonal greedy algorithm in Hilbert spaces. The following question is of great importance: Are there dictionaries in \BBRn such that their coherence in lpn is less than their coherence in l2n for some p ∈ (1,∞)? We show that the answer to the above question is “yes.” Thus, for such dictionaries, replacing the Hilbert space l2n by a Banach space lpn , we improve an upper bound for sparsity that guarantees an exact recovery of a signal.

Convergence of orthogonal greedy algorithm with errors in projectors

A model of orthogonal greedy algorithm is proposed. This model allows one to consider computational errors and to study the stability of this algorithm with respect to errors in projections onto subspaces. A criterion for the convergence of orthogonal greedy expansion to the expanded element is given in terms of computational errors.

Lebesgue-type Inequality for Orthogonal Matching Pursuit for Micro-coherent Dictionaries

In this paper, we investigate the efficiency of some kind of Greedy Algorithms with respect to dictionaries from Hilbert spaces. We establish ideal Lebesgue-type inequality for Orthogonal Matching Pursuit also known in literature as the Orthogonal Greedy Algorithm for -coherent dictionaries. We show that the Orthogonal Matching Pursuit provides an almost optimal approximation on the first. DOI: http://dx.doi.org/10.11591/telkomnika.v11i1.1890

Metric entropy and sparse linear approximation of [formula omitted]-hulls for [formula omitted

Consider ℓq-hulls, 0<q≤1, from a dictionary of M functions in Lp space for 1≤p<∞. Their precise metric entropy orders are derived. Sparse linear approximation bounds are obtained to characterize the number of terms needed to achieve accurate approximation of the best function in a ℓq-hull that is closest to a target function. Furthermore, in the special case of p=2, it is shown that a weak orthogonal greedy algorithm achieves the optimal approximation under an additional condition.

Comparison of the convergence rate of pure greedy and orthogonal greedy algorithms

The following two types of greedy algorithms are considered: the pure greedy algorithm (PGA) and the orthogonal greedy algorithm (OGA). From the standpoint of estimating the rate of convergence on the entire class A1(D), the orthogonal greedy algorithm is optimal and significantly exceeds the pure greedy algorithm. The main result in the present paper is the assertion that the situation can also be opposite for separate elements of the class A1(D) (and even of the class A0(D)): the rate of convergence of the orthogonal greedy algorithm can be significantly lower than the rate of convergence of the pure greedy algorithm.

Learning rates of multi-kernel regression by orthogonal greedy algorithm

We investigate the problem of regression from multiple reproducing kernel Hilbert spaces by means of orthogonal greedy algorithm. The greedy algorithm is appealing as it uses a small portion of candidate kernels to represent the approximation of regression function, and can greatly reduce the computational burden of traditional multi-kernel learning. Satisfied learning rates are obtained based on the Rademacher chaos complexity and data dependent hypothesis spaces.

Greedy Approximation of High-Dimensional Ornstein–Uhlenbeck Operators

We investigate the convergence of a nonlinear approximation method introduced by Ammar et al. (J. Non-Newtonian Fluid Mech. 139:153-176, 2006) for the numerical solution of high-dimensional Fokker-Planck equations featuring in Navier-Stokes-Fokker-Planck systems that arise in kinetic models of dilute polymers. In the case of Poisson's equation on a rectangular domain in R^2, subject to a homogeneous Dirichlet boundary condition, the mathematical analysis of the algorithm was carried out recently by Le Bris, Leli\`evre and Maday (Const. Approx. 30:621-651, 2009), by exploiting its connection to greedy algorithms from nonlinear approximation theory, explored, for example, by DeVore and Temlyakov (Adv. Comput. Math. 5:173-187, 1996); hence, the variational version of the algorithm, based on the minimization of a sequence of Dirichlet energies, was shown to converge. Here, we extend the convergence analysis of the pure greedy and orthogonal greedy algorithms considered by Le Bris et al. to a technically more complicated situation, where the Laplace operator is replaced by an Ornstein-Uhlenbeck operator of the kind that appears in Fokker-Planck equations that arise in bead-spring chain type kinetic polymer models with finitely extensible nonlinear elastic potentials, posed on a high-dimensional Cartesian product configuration space D = D_1 x ... x D_N contained in R^(N d), where each set D_i, i = 1, ..., N, is a bounded open ball in R^d, d = 2, 3.

On the efficiency of the Orthogonal Matching Pursuit in compressed sensing

The paper shows that if a matrix has the restricted isometry property (RIP) of order with isometry constant and if its coherence is less than , then the Orthogonal Matching Pursuit (the Orthogonal Greedy Algorithm) is capable to exactly recover an arbitrary -sparse signal from the compressed sensing in at most iterations. As a result, an arbitrary -sparse signal can be recovered by the Orthogonal Matching Pursuit from measurements.Bibliography: 23 titles.

On the optimality of the Orthogonal Greedy Algorithm for [formula omitted]-coherent dictionaries

In this article, we continue to study the performance of Greedy Algorithms. We show that the Orthogonal Greedy Algorithm (Orthogonal Matching Pursuit) provides an almost optimal approximation on the first [μ−1/20] steps for μ-coherent dictionaries.

A stepwise regression method and consistent model selection for high-dimensional sparse linear models

We introduce a fast stepwise regression method, called the orthogonal greedy algorithm (OGA), that selects input variables to enter a p-dimensional linear regression model (with p ? n, the sample size) sequentially so that the selected variable at each step minimizes the residual sum squares. We derive the convergence rate of OGA and develop a consistent model selection procedure along the OGA path that can adjust for potential spuriousness of the greedily chosen regressors among a large number of candidate variables. The resultant regression estimate is shown to have the oracle property of being equivalent to least squares regression on an asymptotically minimal set of relevant regressors under a strong sparsity condition.

Realizability of greedy algorithms

A purely greedy algorithm and an orthogonal greedy algorithm are studied. It is established that the set of objective functions for which a greedy algorithm can be “realized correctly” has second category for discrete dictionaries.

Improved stability conditions of BOGA for noisy block-sparse signals

The block orthogonal greedy algorithm (BOGA) has been proven to successfully recover block-sparse signals in noiseless environments and the associated stability problem dealing with noisy signals has also been studied in the literature. This paper demonstrates that the recovery conditions of the BOGA previously reported can be relaxed by using a different definition of the block-coherence. The presented results in this paper provide a generalization of those reported by Tseng for block-sparse signal and serve as a complement of the BOGA reported by Eldar for noisy signals.

On performance of greedy algorithms

We show that the Orthogonal Greedy Algorithm (OGA) for dictionaries in a Hilbert space with small coherence M performs almost as well as the best m -term approximation for all signals with sparsity close to the best theoretically possible threshold m = 1 2 ( M − 1 + 1 ) by proving a Lebesgue-type inequality for arbitrary signals. Additionally, we present a dictionary with coherence M and a 1 2 ( M − 1 + 1 ) -sparse signal for which OGA fails to pick up any atoms from the support, showing that the above threshold is sharp. We also show that the Pure Greedy Algorithm (PGA) matches the rate of convergence of the best m -term approximation beyond the saturation limit of m − 1 2 .

The Group Lasso for Stable Recovery of Block-Sparse Signal Representations

Group Lasso is a mixed l1/l2-regularization method for a block-wise sparse model that has attracted a lot of interests in statistics, machine learning, and data mining. This paper establishes the possibility of stably recovering original signals from the noisy data using the adaptive group Lasso with a combination of sufficient block-sparsity and favorable block structure of the overcomplete dictionary. The corresponding theoretical results about the solution uniqueness, support recovery and representation error bound are derived based on the properties of block-coherence and subcoherence. Compared with the theoretical results on the parametrized quadratic program of conventional sparse representation, our stability results are more general. A comparison with block-based orthogonal greedy algorithm is also presented. Numerical experiments demonstrate the validity and correctness of theoretical derivation and also show that in case of noisy situation, the adaptive group Lasso has a better reconstruction performance than the quadratic program approach if the observed sparse signals have a natural block structure.