Pure Greedy Algorithm Research Articles

The learning performance of the weak rescaled pure greedy algorithms

We investigate the regression problem in supervised learning by means of the weak rescaled pure greedy algorithm (WRPGA). We construct learning estimator by applying the WRPGA and deduce the tight upper bounds of the K-functional error estimate for the corresponding greedy learning algorithms in Hilbert spaces. Satisfactory learning rates are obtained under two prior assumptions on the regression function. The application of the WRPGA in supervised learning considerably reduces the computational cost while maintaining its powerful generalization capability when compared with other greedy learning algorithms.

On the Rate of Convergence of Greedy Algorithms

In this paper, a new criterion for the evaluation of the theoretical efficiency of a greedy algorithm is suggested. Using this criterion, we prove some results on the rate of convergence of greedy algorithms, which provide expansions. We consider both the case of Hilbert spaces and the more general case of Banach spaces. The new component of this paper is that we bound the error of approximation by the product of two norms—the norm of f and the A1-norm of f. Typically, only the A1-norm of f is used. In particular, we establish that some greedy algorithms (Pure Greedy Algorithm (PGA) and its modifications) are as good as the Orthogonal Greedy Algorithm (OGA) in this new sense of the rate of convergence, while it is known that the PGA is much worse than the OGA in the standard sense. Our new results provide better bounds for the accuracy than known results in the case of small ∥f∥.

Approximation Properties of the Vector Weak Rescaled Pure Greedy Algorithm

We first study the error performances of the Vector Weak Rescaled Pure Greedy Algorithm for simultaneous approximation with respect to a dictionary D in a Hilbert space. We show that the convergence rate of the Vector Weak Rescaled Pure Greedy Algorithm on A1(D) and the closure of the convex hull of the dictionary D is optimal. The Vector Weak Rescaled Pure Greedy Algorithm has some advantages. It has a weaker convergence condition and a better convergence rate than the Vector Weak Pure Greedy Algorithm and is simpler than the Vector Weak Orthogonal Greedy Algorithm. Then, we design a Vector Weak Rescaled Pure Greedy Algorithm in a uniformly smooth Banach space setting. We obtain the convergence properties and error bound of the Vector Weak Rescaled Pure Greedy Algorithm in this case. The results show that the convergence rate of the VWRPGA on A1(D) is sharp. Similarly, the Vector Weak Rescaled Pure Greedy Algorithm is simpler than the Vector Weak Chebyshev Greedy Algorithm and the Vector Weak Relaxed Greedy Algorithm.

Comparison of Pure Greedy Algorithm with Pure Greedy Algorithm in a Pair of Dictionaries

Comparison of Pure Greedy Algorithm with Pure Greedy Algorithm in a Pair of Dictionaries

Learning capability of the rescaled pure greedy algorithm with non-iid sampling

<abstract><p>We consider the rescaled pure greedy learning algorithm (RPGLA) with the dependent samples drawn according to a non-identical sequence of probability distributions. The generalization performance is provided by applying the independent-blocks technique and adding the drift error. We derive the satisfactory learning rate for the algorithm under the assumption that the process satisfies stationary $ \beta $-mixing, and also find that the optimal rate $ O(n^{-1}) $ can be obtained for i.i.d. processes.</p></abstract>

Sharp sufficient condition for the convergence of greedy expansions with errors in coefficient computation

Abstract Generalized approximate weak greedy algorithms (gAWGAs) were introduced by Galatenko and Livshits as a generalization of approximate weak greedy algorithms, which, in turn, generalize weak greedy algorithm and thus pure greedy algorithm. We consider a narrower case of gAWGA in which only a sequence of absolute errors { ξ n } n = 1 ∞ {\left\{{\xi }_{n}\right\}}_{n=1}^{\infty } is nonzero. In this case sufficient condition for a convergence of a gAWGA expansion to an expanded element obtained by Galatenko and Livshits can be written as ∑ n = 1 ∞ ξ n 2 < ∞ {\sum }_{n=1}^{\infty }{\xi }_{n}^{2}\lt \infty . In the present article, we relax this condition and show that the convergence is guaranteed for ξ n = o 1 n {\xi }_{n}=o\left(\frac{1}{\sqrt{n}}\right) . This result is sharp because the convergence may fail to hold for ξ n ≍ 1 n {\xi }_{n}\hspace{0.33em}\asymp \hspace{0.33em}\frac{1}{\sqrt{n}} .

Efficiency of the weak Rescaled Pure Greedy Algorithm

We study the efficiency of the Weak Rescaled Pure Greedy Algorithm (WRPGA) with respect to a dictionary in a Hilbert space or more generally a Banach space. We obtain the sufficient and necessary conditions for the convergence of WRPGA for any element and any dictionary. This condition is weaker than the sufficient conditions for convergence of the Weak Pure Greedy Algorithm (WPGA). In addition, we establish the noisy version of the error estimate for the WRPGA, which implies the results on sparse classes obtained in [G. Petrova, Rescaled pure greedy algorithm for Hlibert and Banach spaces, Appl. Comput. Harmon. Anal. 41(3) (2016) 852–866].

Distributed Sensing With Orthogonal Multiple Access: To Code or not to Code?

We consider the estimation distortion of a distributed sensing system with finite number of sensor nodes, in which the nodes observe a common phenomenon and transmit their observations to a fusion center over orthogonal channels. In particular, we investigate whether the coded scheme (separate source-channel coding) outperforms the uncoded scheme (joint source-channel coding) or not. To this end, we explicitly derive the estimation distortion of a coded heterogeneous sensing system with diverse node and channel configurations. Based on this result, we show that in a homogeneous sensing system with identical node and channel configurations, the coded scheme outperforms the uncoded scheme if the number of nodes is $K=1$ or $K=2$. For homogenous sensing systems with $K\geq3$ nodes and general heterogeneous sensing systems, we also present explicit conditions for the coded scheme to perform better than the uncoded scheme. Furthermore, we propose to minimize the estimation distortion of heterogeneous sensing systems with hybrid coding, i.e., some nodes use the coded scheme and other nodes use the uncoded scheme. To determine the optimal hybrid coding policy, we develop three greedy algorithms, in which the pure greedy algorithm minimizes distortion greedily, the group greedy algorithm improves performance by using a group of potential sub-polices, and the sorted greedy algorithm reduces computational complexity by using a pre-solved iteration order. Our numerical and Monte Carlo results show that the proposed algorithms closely approach the optimal policy in terms average estimation distortion.

Rescaled Pure Greedy Algorithm for convex optimization

We suggest a new greedy strategy for convex optimization in Banach spaces and prove its convergence rates under a suitable behavior of the modulus of uniform smoothness of the objective function. We show that this algorithm is a generalization of the recently discovered Rescaled Pure Greedy Algorithm for approximation in Hilbert spaces.

Learning capability of the truncated greedy algorithm

Pure greedy algorithm (PGA), orthogonal greedy algorithm (OGA) and relaxed greedy algorithm (RGA) are three widely used greedy type algorithms in both nonlinear approximation and supervised learning. in this paper, we apply another variant of greedy-type algorithm, called the truncated greedy algorithm (TGA) in the realm of supervised learning and study its learning performance. We rigorously prove that TGA is better than PGA in the sense that TGA possesses the faster learning rate than PGA. Furthermore, in some special cases, we also prove that TGA outperforms OGA and RGA. All these theoretical assertions are verified by both toy simulations and real data experiments.

Two‐dimensional adaptive Fourier decomposition

One‐dimensional adaptive Fourier decomposition, abbreviated as 1‐D AFD, or AFD, is an adaptive representation of a physically realizable signal into a linear combination of parameterized Szegö and higher‐order Szegö kernels of the context. In the present paper, we study multi‐dimensional AFDs based on multivariate complex Hardy spaces theory. We proceed with two approaches of which one uses Product‐TM Systems; and the other uses Product‐Szegö Dictionaries. With the Product‐TM Systems approach, we prove that at each selection of a pair of parameters, the maximal energy may be attained, and, accordingly, we prove the convergence. With the Product‐Szegö dictionary approach, we show that pure greedy algorithm is applicable. We next introduce a new type of greedy algorithm, called Pre‐orthogonal Greedy Algorithm (P‐OGA). We prove its convergence and convergence rate estimation, allowing a weak‐type version of P‐OGA as well. The convergence rate estimation of the proposed P‐OGA evidences its advantage over orthogonal greedy algorithm (OGA). In the last part, we analyze P‐OGA in depth and introduce the concept P‐OGA‐Induced Complete Dictionary, abbreviated as Complete Dictionary. We show that with the Complete Dictionary P‐OGA is applicable to the Hardy H2 space on 2‐torus. Copyright © 2016 John Wiley & Sons, Ltd.

Rescaled pure greedy algorithm for Hilbert and Banach spaces

We show that a very simple modification of the Pure Greedy Algorithm for approximating functions by sparse sums from a dictionary in a Hilbert or more generally a Banach space has optimal convergence rates.

Comparison of the convergence rate of pure greedy and orthogonal greedy algorithms

The following two types of greedy algorithms are considered: the pure greedy algorithm (PGA) and the orthogonal greedy algorithm (OGA). From the standpoint of estimating the rate of convergence on the entire class A1(D), the orthogonal greedy algorithm is optimal and significantly exceeds the pure greedy algorithm. The main result in the present paper is the assertion that the situation can also be opposite for separate elements of the class A1(D) (and even of the class A0(D)): the rate of convergence of the orthogonal greedy algorithm can be significantly lower than the rate of convergence of the pure greedy algorithm.

On performance of greedy algorithms

We show that the Orthogonal Greedy Algorithm (OGA) for dictionaries in a Hilbert space with small coherence M performs almost as well as the best m -term approximation for all signals with sparsity close to the best theoretically possible threshold m = 1 2 ( M − 1 + 1 ) by proving a Lebesgue-type inequality for arbitrary signals. Additionally, we present a dictionary with coherence M and a 1 2 ( M − 1 + 1 ) -sparse signal for which OGA fails to pick up any atoms from the support, showing that the above threshold is sharp. We also show that the Pure Greedy Algorithm (PGA) matches the rate of convergence of the best m -term approximation beyond the saturation limit of m − 1 2 .

Lower bounds for the rate of convergence of greedy algorithms

We obtain a lower bound for the rate of convergence of a pure greedy algorithm in the spaces and , and this bound turns out to be very close to the best known upper bound. We also obtain a precise lower bound for the rate of convergence of the orthogonal greedy algorithm in the space .

A greedy approach to multiscale methods

In this paper we study the problem of compute the solution of a linear system in a separable representation form. It arises in the discretized equations appearing in various physical domains, such as kinetic theory, statistical mechanics, quantum mechanics, and in nanoscience and nanotechnology among others. In particular, we use the fact that tensors of order 3 or higher have best rank-1 approximation. This fact allow to us to propose an iterative method based in the so-called by the signal processing community as the Matching Pursuit Algorithm, also known as Projection Pursuit by the statistics community or as a Pure Greedy Algorithm in the approximation theory community. We also give some numerical examples and describe its relationship with the Finite Element Method for High-Dimensional Partial Differential Equations based on the tensorial product of one-dimensional bases. We illustrate this situation taking as a model problem the multidimensional Poisson equation with homogeneous Dirichlet boundary condition.

On n-term approximation with positive coefficients

In this paper, we consider algorithms for constructing n-terms approximations with nonnegative coefficients. The convergence theorem is proved for a “positive” analog of the Pure Greedy Algorithm. We establish a condition on the sequence of weakness coefficients which is sufficient for the convergence of the Positive Weak Greedy Algorithm. This condition is also necessary for the class of monotone sequences.

Optimality of the greedy algorithm for some function classes

The convergence rate of the pure greedy algorithm (PGA) is considered. Upper bounds for the convergence rate of the PGA are obtained in the case of the target function in the classes , , which are extensions of the class . This bound is shown to be sharp in order for . Bibliography: 14 titles.

On Lebesgue-type inequalities for greedy approximation

We study the efficiency of greedy algorithms with regard to redundant dictionaries in Hilbert spaces. We obtain upper estimates for the errors of the Pure Greedy Algorithm and the Orthogonal Greedy Algorithm in terms of the best m-term approximations. We call such estimates the Lebesgue-type inequalities. We prove the Lebesgue-type inequalities for dictionaries with special structure. We assume that the dictionary has a property of mutual incoherence (the coherence parameter of the dictionary is small). We develop a new technique that, in particular, allowed us to get rid of an extra factor m 1 / 2 in the Lebesgue-type inequality for the Orthogonal Greedy Algorithm.

Simultaneous approximation by greedy algorithms

We study nonlinear m-term approximation with regard to a redundant dictionary \(\mathcal {D}\) in a Hilbert space H. It is known that the Pure Greedy Algorithm (or, more generally, the Weak Greedy Algorithm) provides for each f∈H and any dictionary \(\mathcal {D}\) an expansion into a series $$f=\sum_{j=1}^{\infty}c_{j}(f)\varphi_{j}(f),\quad\varphi_{j}(f)\in \mathcal {D},\ j=1,2,\ldots,$$ with the Parseval property: ‖f‖2=∑j|cj(f)|2. Following the paper of A. Lutoborski and the second author we study analogs of the above expansions for a given finite number of functions f1,. . .,fN with a requirement that the dictionary elements φj of these expansions are the same for all fi, i=1,. . .,N. We study convergence and rate of convergence of such expansions which we call simultaneous expansions.