Greedy Approximation Research Articles

Greedy approximation

In this survey we discuss properties of specific methods of approximation that belong to a family of greedy approximation methods (greedy algorithms). It is now well understood that we need to study nonlinear sparse representations in order to significantly increase our ability to process (compress, denoise,etc.) large data sets. Sparse representations of a function are not only a powerful analytic tool but they are utilized in many application areas such as image/signal processing and numerical computation. The key to finding sparse representations is the concept ofm-term approximation of the target function by the elements of a given system of functions (dictionary). The fundamental question is how to construct good methods (algorithms) of approximation. Recent results have established that greedy-type algorithms are suitable methods of nonlinear approximation in bothm-term approximation with regard to bases, andm-term approximation with regard to redundant systems. It turns out that there is one fundamental principle that allows us to build good algorithms, both for arbitrary redundant systems and for very simple well-structured bases, such as the Haar basis. This principle is the use of a greedy step in searching for a new element to be added to a givenm-term approximant.

Kernel methods and flexible inference for complex stochastic dynamics

Approximation theory suggests that series expansions and projections represent standard tools for random process applications from both numerical and statistical standpoints. Such instruments emphasize the role of both sparsity and smoothness for compression purposes, the decorrelation power achieved in the expansion coefficients space compared to the signal space, and the reproducing kernel property when some special conditions are met. We consider these three aspects central to the discussion in this paper, and attempt to analyze the characteristics of some known approximation instruments employed in a complex application domain such as financial market time series. Volatility models are often built ad hoc, parametrically and through very sophisticated methodologies. But they can hardly deal with stochastic processes with regard to non-Gaussianity, covariance non-stationarity or complex dependence without paying a big price in terms of either model mis-specification or computational efficiency. It is thus a good idea to look at other more flexible inference tools; hence the strategy of combining greedy approximation and space dimensionality reduction techniques, which are less dependent on distributional assumptions and more targeted to achieve computationally efficient performances. Advantages and limitations of their use will be evaluated by looking at algorithmic and model building strategies, and by reporting statistical diagnostics.

Alignment of Overlapping Locally Scaled Patches for Multidimensional Scaling and Dimensionality Reduction

Data observations that lie on a manifold can be approximated by a collection of overlapping local patches, the alignment of which in a low dimensional Euclidean space provides an embedding of the data. This paper describes an embedding method using classical multidimensional scaling as a local model based on the fact that a manifold locally resembles an Euclidean space. A set of overlapping neighborhoods are chosen by a greedy approximation algorithm of minimum set cover. Local patches derived from the set of overlapping neighborhoods by classical multidimensional scaling are aligned in order to minimize a residual measure, which has a quadratic form of the resulting global coordinates and can be minimized analytically by solving an eigenvalue problem. This method requires only distances within each neighborhood and provides locally isometric embedding results. The size of the eigenvalue problem scales with the number of overlapping neighborhoods rather than the number of data points. Experiments on both synthetic and real world data sets demonstrate the effectiveness of this method. Extensions and variations of the method are discussed.

Approximation and learning by greedy algorithms

We consider the problem of approximating a given element $f$ from a Hilbert space $\mathcal{H}$ by means of greedy algorithms and the application of such procedures to the regression problem in statistical learning theory. We improve on the existing theory of convergence rates for both the orthogonal greedy algorithm and the relaxed greedy algorithm, as well as for the forward stepwise projection algorithm. For all these algorithms, we prove convergence results for a variety of function classes and not simply those that are related to the convex hull of the dictionary. We then show how these bounds for convergence rates lead to a new theory for the performance of greedy algorithms in learning. In particular, we build upon the results in [IEEE Trans. Inform. Theory 42 (1996) 2118--2132] to construct learning algorithms based on greedy approximations which are universally consistent and provide provable convergence rates for large classes of functions. The use of greedy algorithms in the context of learning is very appealing since it greatly reduces the computational burden when compared with standard model selection using general dictionaries.

Data Gathering with Tunable Compression in Sensor Networks

We study the problem of constructing a data gathering tree over a wireless sensor network in order to minimize the total energy for compressing and transporting information from a set of source nodes to the sink. This problem is crucial for advanced computationally intensive applications, where traditional "maximum" in-network compression may result in significant computation energy. We investigate a tunable data compression technique that enables effective trade-offs between the computation and communication costs. We derive the optimal compression strategy for a given data gathering tree and then investigate the performance of different tree structures for networks deployed on a grid topology, as well as general graphs. Our analytical results pertaining to the grid topology and simulation results pertaining to the general graphs indicate that the performance of a simple greedy approximation to the Minimal Steiner Tree (MST) provides a constant-factor approximation for the grid topology and good average performance on the general graphs. Although, theoretically, a more complicated randomized algorithm offers a polylogarithmic performance bound, the simple greedy approximation of MST is attractive for practical implementation.

Coverage breach problems in bandwidth-constrained sensor networks

Recent research in sensor networks highlights the low-power mode operation of sensor networks. In wireless sensor networks, network lifetime can be extended by organizing sensors into mutually exclusive subsets and alternatively activating each subset. Coverage breach occurs when a subset fails to cover all the targets. In bandwidth-constrained sensor networks, coverage breach is more likely to happen because when active sensors periodically send data to the base station, contention for channel access must be considered. Channel bandwidth imposes a limit on the cardinality of each subset. To make efficient use of both energy and bandwidth with minimum coverage breach requires optimal arrangement of sensor nodes. This article addresses three coverage breach problems related to the low-power operation of wireless sensor networks where channel bandwidth is limited. The three coverage breach problems are formulated using integer linear programming models. A greedy approximation algorithm and a heuristic based on the LP-relaxation method are proposed. Effects of changing different network resources on sensor network coverage are studied through simulations. One consistent result is that when the number of sensors increases, network lifetime can be improved without loss of network coverage only if there is no bandwidth constraint; with bandwidth constraints, network lifetime may be improved further at the cost of coverage breach.

Integrated Downlink Resource Management for Multiservice WiMAX Networks

In this paper, we propose a novel downlink resource management framework for multiservice WiMAX (worldwide interoperability for microwave access) networks. Our framework consists of two major components: adaptive power allocation (APA) and call admission control (CAC). We formulate each of them as an optimization problem, where the demands of both WiMAX service providers and subscribers are taken into account. To solve the optimization problems, we develop a fairness-constrained greedy revenue algorithm for downlink APA optimization and a utility-constrained greedy approximation algorithm for downlink CAC optimization. Our simulation results show that, when combining the APA and CAC optimization methods together, the proposed resource management framework can meet the expectations of both service providers and subscribers

On Lebesgue-type inequalities for greedy approximation

We study the efficiency of greedy algorithms with regard to redundant dictionaries in Hilbert spaces. We obtain upper estimates for the errors of the Pure Greedy Algorithm and the Orthogonal Greedy Algorithm in terms of the best m-term approximations. We call such estimates the Lebesgue-type inequalities. We prove the Lebesgue-type inequalities for dictionaries with special structure. We assume that the dictionary has a property of mutual incoherence (the coherence parameter of the dictionary is small). We develop a new technique that, in particular, allowed us to get rid of an extra factor m 1 / 2 in the Lebesgue-type inequality for the Orthogonal Greedy Algorithm.

Relaxation in Greedy Approximation

We study greedy algorithms in a Banach space from the point of view of convergence and rate of convergence. There are two well-studied approximation methods: the Weak Chebyshev Greedy Algorithm (WCGA) and the Weak Relaxed Greedy Algorithm (WRGA). The WRGA is simpler than the WCGA in the sense of computational complexity. However, the WRGA has limited applicability. It converges only for elements of the closure of the convex hull of a dictionary. In this paper we study algorithms that combine good features of both algorithms, the WRGA and the WCGA. In the construction of such algorithms we use different forms of relaxation. First results on such algorithms have been obtained in a Hilbert space by A. Barron, A. Cohen, W. Dahmen, and R. DeVore. Their paper was a motivation for the research reported here.

On convergence of greedy approximations for the trigonometric system

In this note we discuss the convergence of greedy approximants for trigonometric Fourier expansion in Lp(T), 1 p < 2.

Greedy expansions in Banach spaces

We study convergence and rate of convergence of expansions of elements in a Banach space X into series with regard to a given dictionary \(\mathcal{D}\) . For convenience we assume that \(\mathcal{D}\) is symmetric: \(g\in\mathcal{D}\) implies \(-g\in\mathcal{D}\) . The primary goal of this paper is to study representations of an element f∈X by a series $$f\sim\sum_{j=1}^{\infty}c_{j}(f)g_{j}(f),\quad g_{j}(f)\in\mathcal{D},\ c_{j}(f)>0,\ j=1,2,\dots.$$ In building such a representation we should construct two sequences: {gj (f)} j=1∞ and {cj (f)} j=1∞ . In this paper the construction of {gj (f)} j=1∞ will be based on ideas used in greedy-type nonlinear approximation. This explains the use of the term greedy expansion. We use a norming functional \(F_{f_{m-1}}\) of a residual fm−1 obtained after m−1 steps of an expansion procedure to select the mth element \(g_{m}(f)\in \mathcal{D}\) from the dictionary. This approach has been used in previous papers on greedy approximation. The greedy expansions in Hilbert spaces are well studied. The corresponding convergence theorems and estimates for the rate of convergence are known. Much less is known about greedy expansions in Banach spaces. The first substantial result on greedy expansions in Banach spaces has been obtained recently by Ganichev and Kalton. They proved a convergence result for the Lp , 1<p<∞, spaces. In this paper we find a simple way of selecting coefficients cm (f) that provides convergence of the corresponding greedy expansions in any uniformly smooth Banach space. Moreover, we obtain estimates for the rate of convergence of such greedy expansions for \(f\in A_{1}(\mathcal{D})\) – the closure (in X) of the convex hull of \(\mathcal{D}\) .

Robust Monitoring of Link Delays and Faults in IP Networks

In this paper, we develop failure-resilient techniques for monitoring link delays and faults in a Service Provider or Enterprise IP network. Our two-phased approach attempts to minimize both the monitoring infrastructure costs as well as the additional traffic due to probe messages. In the first phase, we compute the locations of a minimal set of monitoring stations such that all network links are covered, even in the presence of several link failures. Subsequently, in the second phase, we compute a minimal set of probe messages that are transmitted by the stations to measure link delays and isolate network faults. We show that both the station selection problem as well as the probe assignment problem are NP-hard. We then propose greedy approximation algorithms that achieve a logarithmic approximation factor for the station selection problem and a constant factor for the probe assignment problem. These approximation ratios are provably very close to the best possible bounds for any algorithm

On the Average Case Performance of Some Greedy Approximation Algorithms For the Uncapacitated Facility Location Problem

In combinatorial optimization, a popular approach toNP-hard problems is the design of approximation algorithms. These algorithms typically run in polynomial time and are guaranteed to produce a solution which is within a known multiplicative factor of optimal. Unfortunately, the known factor is often known to be large in pathological instances. Conventional wisdom holds that, in practice, approximation algorithms will produce solutions closer to optimal than their proven guarantees. In this paper, we use the rigorous-analysis-of-heuristics framework to investigate this conventional wisdom.We analyze the performance of 3 related approximation algorithms for the uncapacitated facility location problem (from [Jain, Mahdian, Markakis, Saberi, Vazirani, 2003] and [Mahdian, Ye, Zhang, 2002]) when each is applied to an instances created by placing n points uniformly at random in the unit square. We find that, with high probability, these 3 algorithms do not find asymptotically optimal solutions, and, also with high probability, a simple plane partitioning heuristic does find an asymptotically optimal solution.

Greedy approximation with respect to certain subsystems of the Walsh orthonormal system

In an article that appeared in 1967, J.J. Price has shown that there is a vast family of subsystems of the Walsh orthonormal system each of which is complete on sets of large measure. In the present work it is shown that the greedy algorithm, when applied to functions in L 1 [ 0 , 1 ] L^{1}[0,1] , is surprisingly effective for these nearly–complete families. Indeed, if Φ \Phi is such a subsystem of the Walsh system, then to each positive ε \varepsilon , however small, there corresponds a Lebesgue measurable set E E such that for every f f , Lebesgue integrable on [ 0 , 1 ] [0,1] , the greedy approximants to f f , associated with Φ \Phi , converge, in the L 1 L^{1} norm, to an integrable function g g that coincides with f f on E E .

A Transportation Problem with Minimum Quantity Commitment

We study a transportation problem with the minimum quantity commitment (MQC), which is faced by a famous international company. The company has a large number of cargos for carriers to ship to the United States. However, the U.S. Marine Federal Commission stipulates that when shipping cargos to the United States, shippers must engage their carriers with an MQC. With such a constraint of MQC, the transportation problem becomes intractable. To solve it practically, we provide a mixed-integer programming model defined by a number of strong facets. Based on this model, a branch-and-cut search scheme is applied to solve small-size instances and a linear programming rounding heuristic for large ones. We also devise a greedy approximation method, whose solution quality depends on the scale of the minimum quantity if the transportation cost forms a distance metric. Extensive experiments have been conducted to measure the performance of the formulations and the algorithms and have shown that the linear rounding heuristic behaves best.

A greedy approximation algorithm for the uniform metric labeling problem analyzed by a primal-dual technique

We consider the uniform metric labeling problem. This NP-hard problem considers how to assign objects to labels respecting assignment and separation costs. The known approximation algorithms are based on solutions of large linear programs and are impractical for moderate- and large-size instances. We present an 8log n -approximation algorithm that can be applied to large-size instances. The algorithm is greedy and is analyzed by a primal-dual technique. We implemented the presented algorithm and two known approximation algorithms and compared them at randomized instances. The gain of time was considerable with small error ratios. We also show that the analysis is tight, up to a constant factor.

Greedy algorithm for functions with low mixed smoothness

In this note, we investigate the efficiency of the greedy algorithm for the classes of multivariate periodic functions with low mixed smoothness in L q with regard to the wavelet-type basis. We find that the order of greedy approximation in the case of low smoothness is different for some range of parameters.

Peer‐based automatic configuration of pervasive applications

Pervasive computing envisions seamless support for user tasks through cooperating devices that are present in an environment. Fluctuating availability of devices, induced by mobility and failures, requires mechanisms and algorithms that allow applications to adapt to their ever‐changing execution environments without user intervention. To ease the development of adaptive applications, Becker et al. (3) have proposed the peer‐based component system PCOM. This system provides fundamental mechanisms to support the automated composition of applications at runtime. In this article, we discuss the requirements on algorithms that enable automatic configuration of pervasive applications. Furthermore, we show how finding a configuration can be interpreted as Distributed Constraint Satisfaction Problem. Based on this, we present an algorithm that is capable of finding an application configuration in the presence of strictly limited resources. To show the feasibility of this algorithm, we present an evaluation based on simulations and real‐world measurements and we compare the results with a simple greedy approximation.

A greedy approximation algorithm for the group Steiner problem

In the group Steiner problem we are given an edge-weighted graph G = ( V , E , w ) and m subsets of vertices { g i } i = 1 m . Each subset g i is called a group and the vertices in ⋃ i g i are called terminals. It is required to find a minimum weight tree that contains at least one terminal from every group. We present a poly-logarithmic ratio approximation for this problem when the input graph is a tree. Our algorithm is a recursive greedy algorithm adapted from the greedy algorithm for the directed Steiner tree problem [Approximating the weight of shallow Steiner trees, Discrete Appl. Math. 93 (1999) 265–285, Approximation algorithms for directed Steiner problems, J. Algorithms 33 (1999) 73–91]. This is in contrast to earlier algorithms that are based on rounding a linear programming based relaxation for the problem [A polylogarithmic approximation algorithm for the Group Steiner tree problem, J. Algorithms 37 (2000) 66–84, preliminary version in Proceedings of SODA, 1998 pp. 253–259, On directed Steiner trees, Proceedings of SODA, 2002, pp. 59–63]. We answer in positive a question posed in [A polylogarithmic approximation algorithm for the Group Steiner tree problem, J. Algorithms 37 (2000) 66–84, preliminary version in Proceedings of SODA, 1998 pp. 253–259] on whether there exist good approximation algorithms for the group Steiner problem that are not based on rounding linear programs. For every fixed constant ε > 0 , our algorithm gives an O ( ( log ∑ i | g i | ) 1 + ε · log m ) approximation in polynomial time. Approximation algorithms for trees can be extended to arbitrary undirected graphs by probabilistically approximating the graph by a tree. This results in an additional multiplicative factor of O ( log | V | ) in the approximation ratio, where | V | is the number of vertices in the graph. The approximation ratio of our algorithm on trees is slightly worse than the ratio of O ( log ( max i | g i | ) · log m ) provided by the LP based approaches.

Greedy approximation algorithms for directed multicuts

AbstractThe Directed Multicut (DM) problem is: given a simple directed graph G = (V, E) with positive capacities ue on the edges, and a set K ⊆ V × V of ordered pairs of nodes of G, find a minimum capacity K‐multicut; C ⊆ E is a K‐multicut if in G − C there is no (s, t)‐path for any (s, t) ⫅ K. In the uncapacitated case (UDM) the goal is to find a minimum size K‐multicut. The best approximation ratio known for DM is $O(\min\{\sqrt{n},opt\})$ by Gupta, where n = |V|, and opt is the optimal solution value. All known nontrivial approximation algorithms for the problem solve large linear programs. We give the first combinatorial approximation algorithms for the problem. Our main result is an Õ(n2/3/opt1/3)‐approximation algorithm for UDM, which improves the $O(\min\{opt,\sqrt{n}\})$‐approximation for opt = Ω(n1/2+ϵ). Combined with the article of Gupta, we get that UDM can be approximated within better than $O(\sqrt n)$, unless $opt={\tilde \Theta}(\sqrt n)$. We also give a simple and fast O(n2/3)‐approximation algorithm for DM. © 2005 Wiley Periodicals, Inc. NETWORKS, Vol. 45(4), 214–217 2005