Crawling is difficult in clouds

Abstract

Sensor rich mobile communication devices combined with cloud infrastructure are resulting in novel applications that merge physical and cyber spaces and start providing real time information for many important applications. One of the most common approaches to find information on the Web has been crawling to identify and index data sources. Crawling has been a key component of traditional search engines.The convergence of mobile devices, cloud based systems, and the Web brings in several new challenges. One of the most important new challenges is to detect events, trends, and situations in real time. Crawling to perform real time tasks is not possible because of the latency issues in most real time applications. Filtering oriented approaches are emerging as a solution in such applications. We adopt a perspective that mobile devices and sensors produce data in the form of micro-events, which should be analyzed to detect events, trends, and situations. We are developing abstractions and techniques to aggregate and analyze spatio-temporal micro-events and implement real time search for actionable information. We developed an approach for aggregating social interest around any particular theme from any particular location into 'social pixels'. Aggregating social pixels spatiotemporally allows creation of images and videos, which represent meaningful information about events and situations. One could use these images for visualization. More importantly, however, various media processing techniques (like segmentation, convolution) can be used to derive semantic situation information from these image and video representations, which in turn provide situation information in real world. By developing a declarative set of operators we are building an environment for users to formulate queries to visualize, characterize, and analyze such data. In this presentation, we will discuss our approach and present results of this approach for emerging applications