Data Summarization Techniques Research Articles

A statistical study of pre-monsoon weather over south Bengal using descriptive and inferential techniques

Statistical data summarization techniques, curve fitting methods and statistical tests, both parametric and non-parametric, have been applied to form a comprehensive idea about pre-monsoon weather over South Bengal situated in the northeastern part of the Indian sub-continent. The work is based on surface data recorded at 15 major observatories during the period 1969-2000, spread across the western plateau and highlands, Rarh and Gangetic region of South Bengal. The homogeneity of pre-monsoon rainfall and its variability over different stations has been studied using Bartlett`s and Kruskal Wallis tests. For stations with complete weather information for at least 30 consecutive years, time series analysis has been carried out on rainfall data for a climatological overview of long-term behaviour, seasonal and cyclical fluctuations of pre-monsoon rainfall in those areas. This paper, apart from being a regional study, highlights the use of certain parametric and non-parametric tests, not so widely used in the context of climatology.

A framework for clustering massive graph streams

AbstractIn this paper, we examine the problem of clustering massive graph streams. Graph clustering poses significant challenges because of the complex structures which may be present in the underlying data. The massive size of the underlying graph makes explicit structural enumeration very difficult. Consequently, most techniques for clustering multidimensional data are difficult to generalize to the case of massive graphs. Recently, methods have been proposed for clustering graph data, though these methods are designed for static data, and are not applicable to the case of graph streams. Furthermore, these techniques are especially not effective for the case of massive graphs, since a huge number of distinct edges may need to be tracked simultaneously. This results in storage and computational challenges during the clustering process. In order to deal with the natural problems arising from the use of massive disk‐resident graphs, we propose a technique for creating hash‐compressed microclusters from graph streams. The compressed microclusters are designed by using a hash‐based compression of the edges onto a smaller domain space. We provide theoretical results which show that the hash‐based compression continues to maintain bounded accuracy in terms of distance computations. Since clustering is a data summarization technique, it can also be naturally extended to the problem of evolution analysis. We provide experimental results which illustrate the accuracy and efficiency of the underlying method. Copyright © 2010 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 3: 399‐416, 2010

Aggregation Bias in Sponsored Search Data: The Curse and the Cure

Recently there has been significant interest in studying consumer behavior in sponsored search advertising (SSA). Researchers have typically used daily data from search engines containing measures such as average bid, average ad position, total impressions, clicks, and cost for each keyword in the advertiser’s campaign. A variety of random utility models have been estimated using such data and the results have helped researchers explore the factors that drive consumer click and conversion propensities. However, virtually every analysis of this kind has ignored the intraday variation in ad position. We show that estimating random utility models on aggregated (daily) data without accounting for this variation will lead to systematically biased estimates. Specifically, the impact of ad position on click-through rate (CTR) is attenuated and the predicted CTR is higher than the actual CTR. We analytically demonstrate the existence of the bias and show the effect of the bias on the equilibrium of the SSA auction. Using a large data set from a major search engine, we measure the magnitude of bias and quantify the losses suffered by the search engine and an advertiser using aggregate data. The search engine revenue loss can be as high as 11% due to aggregation bias. We also present a few data summarization techniques that can be used by search engines to reduce or eliminate the bias.

Optimality and scalability in lattice histogram construction

The Lattice Histogram is a recently proposed data summarization technique that achieves approximation quality preferable to that of an optimal plain histogram. Like other hierarchical synopsis methods, a lattice histogram (LH) aims to approximate data using a hierarchical structure. Still, this structure is not defined a priori; it consists an unknown , not a given, of the problem. Past work has defined the properties that an LH needs to obey and developed general-purpose approximation algorithms for the construction thereof. Still, two major issues remain unaddressed: First, the construction of an optimal LH for a given error metric is a problem unsolved to date. Second, the proposed algorithms suffer from too high space and time complexities that render their application in real-world settings problematic. In this paper, we address both these questions, focusing on the case that the target error metric is a maximum error metric. Our algorithms treat both the error-bounded LH construction problem, in which the space occupied by an LH is minimized under an error constraint, as well as the classic space-bounded problem. First, we develop a dynamic-programming scheme that detects an optimal LH under a given maximum-error bound. Second, we propose an efficient, practical, greedy algorithm that solves the same problem with much lower time and space requirements. Then, we show how both our algorithms can be applied to the classic space-bounded problem, aiming at minimizing error under a bound on space. Our experimental study with real-world data sets shows the effectiveness of our methods compared to competing summarization techniques. Moreover, our findings show that our greedy heuristic performs almost as well as the optimal solution in terms of accuracy.

Querying a summary of database

For some years, data summarization techniques have been developed to handle the growth of databases. However these techniques are usually not provided with tools for end-users to efficiently use the produced summaries. This paper presents a first attempt to develop a querying tool for the SAINTETIQ summarization model. The proposed search algorithm takes advantage of the hierarchical structure of the SAINTETIQ summaries to efficiently answer questions such as "how are, on some attributes, the tuples which have specific characteristics?" Moreover, this algorithm can be seen both as a boolean querying mechanism over a hierarchy of summaries, and as a flexible querying mechanism over the underlying relational tuples.

Building and querying e-catalog networks using P2P and data summarisation techniques

One of the critical issues in Web-based e-commerce has been how to efficiently and effectively integrate and query heterogeneous, diverse e-catalogs. We propose an integration framework for building and querying catalogs. Our approach is based on a hybrid of peer-to-peer data sharing paradigm and Web-services architecture. Peers in our system serve as domain-specific data integration mediators. Links between peers are established based on the similarity of the domain they represent. The relationships are used for routing queries among peers. As the number of catalogs involved grow larger, the need for filtering irrelevant data sources will become increasingly high. We apply a summarisation technique to summarise the content of catalogs. The summaries are used to pre-selecting data sources that are relevant to a user query.