Large-scale Web Search Engines Research Articles

Experimental Framework to Simulate Rescue Operations after a Natural Disaster

Computational simulation is a powerful tool for performance evaluation of computational systems. It is useful to make capacity planning of data center clusters, to obtain profiling reports of software applications and to detect bottlenecks. It has been used in different research areas like large scale Web search engines, natural disaster evacuations, computational biology, human behavior and tendency, among many others. However, properly tuning the parameters of the simulators, defining the scenarios to be simulated and collecting the data traces is not an easy task. It is an incremental process which requires constantly comparing the estimated metrics and the flow of simulated actions against real data. In this work, we present an experimental framework designed for the development of large scale simulations of two applications used upon the occurrence of a natural disaster strikes. The first one is a social application aimed to register volunteers and manage emergency campaigns and tasks. The second one is a benchmark application a data repository named MongoDB. The applications are deployed in a distributed platform which combines different technologies like a Proxy, a Containers Orchestrator, Containers and a NoSQL Database. We simulate both applications and the architecture platform. We validate our simulators using real traces collected during simulacrums of emergency situations.

Performance evaluation of single vs. batch of queries on GPUs

SummaryThe WAND processing strategy is a dynamic pruning algorithm designed for large scale Web search engines where fast response to queries is a critical service. The WAND is used to reduce the amount of computation by scoring only documents that may become part of the top‐k document results. In this paper, we present two parallel strategies for the WAND algorithm and compare their performance on GPUs. In our first strategy (named size‐based), the posting lists are evenly partitioned among thread blocks. Our second strategy (named range‐based) partitions the posting lists according to document identifier intervals; thus, partitions may have different sizes. We also propose three threshold sharing policies, named Local, Safe‐R, and Safe‐WR, which emulate the WAND algorithm global pruning technique. We evaluated our proposals with different amounts of work, from short to extra‐large queries, using single query processing and batch of queries. Results show that the size‐based strategy reports the highest speedups but at the cost of low quality of results. The range‐based algorithm retrievals the exact top‐k documents and maintains a good speedup. Moreover, both strategies are capable of scaling as the amount of work is increased. In addition, there is no significant difference in the performance of the three threshold sharing policies.

Hybrid capacity planning methodology for web search engines

Capacity planning studies are suitable for supporting decision making in management and operation of Web search engines deployed on large clusters of processors. Among many possibilities, they enable ensuring that a sufficient amount of computational resources are timely provisioned to efficiently deal with the ever changing streams of user queries. In this paper, we present a simulation based methodology devised to perform capacity planning in large scale Web search engines. It combines classical operational analysis formulae with discrete event simulation to significantly reduce the number of deployments that are evaluated to find an optimal assignment for a target workload. We experimentally evaluate our proposal for demanding cases such as service nodes with temporary failures. The results show that the proposed methodology is able to produce good quality results in practical running times.

Fast Ranking with Additive Ensembles of Oblivious and Non-Oblivious Regression Trees

Learning-to-Rank models based on additive ensembles of regression trees have been proven to be very effective for scoring query results returned by large-scale Web search engines. Unfortunately, the computational cost of scoring thousands of candidate documents by traversing large ensembles of trees is high. Thus, several works have investigated solutions aimed at improving the efficiency of document scoring by exploiting advanced features of modern CPUs and memory hierarchies. In this article, we present Q uick S corer , a new algorithm that adopts a novel cache-efficient representation of a given tree ensemble, performs an interleaved traversal by means of fast bitwise operations, and supports ensembles of oblivious trees. An extensive and detailed test assessment is conducted on two standard Learning-to-Rank datasets and on a novel very large dataset we made publicly available for conducting significant efficiency tests. The experiments show unprecedented speedups over the best state-of-the-art baselines ranging from 1.9 × to 6.6 × . The analysis of low-level profiling traces shows that Q uick S corer efficiency is due to its cache-aware approach in terms of both data layout and access patterns and to a control flow that entails very low branch mis-prediction rates.

Scalability Challenges in Web Search Engines

In this book, we aim to provide a fairly comprehensive overview of the scalability and efficiency challenges in large-scale web search engines. More specifically, we cover the issues involved in the design of three separate systems that are commonly available in every web-scale search engine: web crawling, indexing, and query processing systems. We present the performance challenges encountered in these systems and review a wide range of design alternatives employed as solution to these challenges, specifically focusing on algorithmic and architectural optimizations. We discuss the available optimizations at different computational granularities, ranging from a single computer node to a collection of data centers. We provide some hints to both the practitioners and theoreticians involved in the field about the way large-scale web search engines operate and the adopted design choices. Moreover, we survey the efficiency literature, providing pointers to a large number of relatively important research papers. Finally, we discuss some open research problems in the context of search engine efficiency.

Reprint of: The anatomy of a large-scale hypertextual web search engine

In this paper, we present Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext. Google is designed to crawl and index the Web efficiently and produce much more satisfying search results than existing systems. The prototype with a full text and hyperlink database of at least 24million pages is available at http://google.stanford.edu/To engineer a search engine is a challenging task. Search engines index tens to hundreds of millions of web pages involving a comparable number of distinct terms. They answer tens of millions of queries every day. Despite the importance of large-scale search engines on the web, very little academic research has been done on them. Furthermore, due to rapid advance in technology and web proliferation, creating a web search engine today is very different from 3years ago. This paper provides an in-depth description of our large-scale web search engine – the first such detailed public description we know of to date.Apart from the problems of scaling traditional search techniques to data of this magnitude, there are new technical challenges involved with using the additional information present in hypertext to produce better search results. This paper addresses this question of how to build a practical large-scale system which can exploit the additional information present in hypertext. Also we look at the problem of how to effectively deal with uncontrolled hypertext collections, where anyone can publish anything they want.

Semantifying queries over large-scale Web search engines

Abstract In many situations, searching the web is synonymous to information seeking. Currently, web search engines are the most popular vehicle via which people get access to the web. Their popularity is partially due to the intrinsic way that people interact with them, i.e., by typing some keywords to the corresponding input box. Despite their popularity, search engines often fail to satisfy certain information needs, especially when the latter are haze and poorly articulated. In this paper, we focus on the occasions when large-scale web search engines find it difficult to cope with specific information-seeking behaviors and we accordingly introduce a query construction service that is targeted towards the solution of this problem. The proposed service leverages information coming from various DBpedia datasets and provides an intuitive GUI via which searchers determine the semantic orientation of their queries before these are addressed to the underlying search engine. The evaluation of the query construction service justifies the motive of this paper and indicates that it can considerably improve the searchers’ querying ability when search engines fail to provide adequate help.

Distributed Search Engine System Productivity Modeling and Evaluation

针对分布式搜索引擎系统效能建模与评估问题,通过对当前分布式搜索引擎系统的建模与分类,扩展了能耗与网络开销的成本模型;对5种构建搜索引擎系统的设计方案,从系统成本、系统规模和查询响应时间等角度进行了详尽的理论分析与评价.由此发现,由广域网分布式采集系统和多机群索引系统组成的半广域网搜索引擎系统相对于其他系统具有相对较高的效能,同时能够较好地兼顾用户的服务质量.;This study extends the current productivity models for a typical Web search engine system, which consists of a Web crawling system and an indexing system. Five different design schemata are characterized according to this model and are compared through power consumption, networking cost, system scale, and query efficiency. The half-WAN scheme, which consists of a WAN-based crawling system and a multi-cluster indexing system, is proved to be the best choice for a large-scale highly-efficient Web search engine.

Document replication strategies for geographically distributed web search engines

Large-scale web search engines are composed of multiple data centers that are geographically distant to each other. Typically, a user query is processed in a data center that is geographically close to the origin of the query, over a replica of the entire web index. Compared to a centralized, single-center search engine, this architecture offers lower query response times as the network latencies between the users and data centers are reduced. However, it does not scale well with increasing index sizes and query traffic volumes because queries are evaluated on the entire web index, which has to be replicated and maintained in all data centers. As a remedy to this scalability problem, we propose a document replication framework in which documents are selectively replicated on data centers based on regional user interests. Within this framework, we propose three different document replication strategies, each optimizing a different objective: reducing the potential search quality loss, the average query response time, or the total query workload of the search system. For all three strategies, we consider two alternative types of capacity constraints on index sizes of data centers. Moreover, we investigate the performance impact of query forwarding and result caching. We evaluate our strategies via detailed simulations, using a large query log and a document collection obtained from the Yahoo! web search engine.

A five-level static cache architecture for web search engines

Caching is a crucial performance component of large-scale web search engines, as it greatly helps reducing average query response times and query processing workloads on backend search clusters. In this paper, we describe a multi-level static cache architecture that stores five different item types: query results, precomputed scores, posting lists, precomputed intersections of posting lists, and documents. Moreover, we propose a greedy heuristic to prioritize items for caching, based on gains computed by using items’ past access frequencies, estimated computational costs, and storage overheads. This heuristic takes into account the inter-dependency between individual items when making its caching decisions, i.e., after a particular item is cached, gains of all items that are affected by this decision are updated. Our simulations under realistic assumptions reveal that the proposed heuristic performs better than dividing the entire cache space among particular item types at fixed proportions.

Locality-Based pruning methods for web search

This article discusses a novel approach developed for static index pruning that takes into account the locality of occurrences of words in the text. We use this new approach to propose and experiment on simple and effective pruning methods that allow a fast construction of the pruned index. The methods proposed here are especially useful for pruning in environments where the document database changes continuously, such as large-scale web search engines. Extensive experiments are presented showing that the proposed methods can achieve high compression rates while maintaining the quality of results for the most common query types present in modern search engines, namely, conjunctive and phrase queries. In the experiments, our locality-based pruning approach allowed reducing search engine indices to 30% of their original size, with almost no reduction in precision at the top answers. Furthermore, we conclude that even an extremely simple locality-based pruning method can be competitive when compared to complex methods that do not rely on locality information.

Google

Click to increase image sizeClick to decrease image size Notes 1. Maurice Saatchi, ‘The strange death of modern advertising’, Financial Times, 22 June 2006. 2. Robert O. Keohane & Joseph S. Nye, ‘Power and Interdependence in the Information Age’, Foreign Affairs, Vol. 77, No. 5 (1998), p. 84. 3. Thomas. L Friedman, ‘Is Google God?’, The New York Times, 29 June 2003. 4. Bill Thomson, ‘Is Google too powerful?’, BBC, 21 February 2003. 5. Robert W. McChesney, ‘The New Global Media’, in David Held & Anthony McGrew (eds.), The Global Transformations Reader: An Introduction to the Globalization Debate, 2nd edn (Polity, 2002), p. 260. 6. Manuel Castells, The Rise of the Network Society (Blackwell, 1996). 7. See Susan Strange, The Retreat of the State: The Diffusion of Power in the World Economy (Cambridge University Press, 1996). 8. Sergey Brin & Larry Page, ‘The Anatomy of a Large-Scale Hypertextual Web Search Engine’, Computer Networks, Vol. 30, Nos 1-7 (1998), http://www-db.stanford.edu/∼backrub/google.html (accessed 24 August 2006). 9. S. A. Mathieson, ‘Google – Swiss Army Knife for Hackers?’, Infosecurity Today, November–December (2005), pp. 39–41. 10. Saul Hansell, ‘Yahoo is unleashing a new way to turn ad clicks into ka-ching’, The New York Times, 8 May 2006. 11. Google, ‘Corporate Information–Technology Overview’, http://www.google.co.uk/intl/en/corporate/tech.html (accessed 24 June 2006). 12. Quoted in Steve Lohr, ‘Microsoft and Google grapple for supremacy as stakes escalate’, The New York Times, 10 May 2006. 13. Figures are from Danny Sullivan, ‘Nielsen NetRatings Search Engine Ratings’, 22 August 2006, at http://searchenginewatch.com/showPage.html?page=2156451 (accessed 22 August 2006); Bill Tancer, ‘Google's weekly search numbers’, Hitwise Intelligence, 22 August 2006, http://weblogs.hitwise.com/bill-tancer/2006/08/googles_weekly_search_numbers.html (accessed 23 August 2006). 14. Google, ‘Company Overview’, http://www.google.co.uk/intl/en/corporate/index.html (accessed 20 June 2006). 15. According to a recent survey, 56.3 per cent of internet users in the USA make use of search engines at least once a day. See iProspect, Search Engine User Attitudes Survey, April–May 2004, http://www.iprospect.com/premiumPDFs/iProspectSurveyComplete.pdf#search=‘iProspect%20search%20engine%20attitude%20survey ’ (accessed 23 August 2006). 16. Google, ‘Corporate Information: Features Overview’ (2004), http://www.google.com/intl/en/corporate/features.html (accessed 20 June 2006). 17. The results used in the comparison were produced by the author's own search for the query term ‘China’, using three search engines including Google (www.google.com), Yahoo! (www.yahoo.com) and Microsoft MSN (www.msn.com) conducted on 23 August 2006. 18. Deborah Fallows, Search Engine Users, Pew Internet & American Life Project, 23 January 2005, http://www.pewinternet.org/ (accessed 23 August 2006). 19. David Reid, ‘French answer to Google Library’, BBC, 24 June 2005. 20. Rhys Blakely, ‘Chirac wants a rival to beat Google’, The Times, 29 April 2006. 21. As a brainchild of Jacques Chirac and former German Chancellor Gerhard Schröder, Quaero was conceived in April 2005. 22. ‘Support for EU “digital library”’, BBC, 4 May 2005. 23. For more details about the Gallica project (http://gallica.bnf.fr/), see Reid, ‘French Answer to Google Library’. 24. Peter Sayer, ‘Google France Loses Appeal in AdWords Trademark Dispute’, 17 March 2005, at http://www.infoworld.com/article/05/03/17/HNgooglefranceadwords_1.html (accessed 12 September 2006). 25. BBC, ‘Support for EU “Digital Library”’. 26. ‘Quaero's challenge: I search, therefore I am’, Financial Times, 23 January 2006. 27. Reid, ‘French answer to Google Library’. 28. Bobbie Johnson, ‘The question: does France really need its own search engine?’, The Guardian, 27 April 2006. 29. The COPA was struck down by the Supreme Court for its potential to violate adults' online freedom of speech. 30. Financial Times, 15 March 2006. 31. Hansell, ‘Yahoo is unleashing a new way’. 32. Associated Press, 24 January 2006. 33. Google, ‘Technology Overview’, at http://www.google.com/corporate/tech.html (accessed 20 June 2006). 34. The search was conducted by the author on the 2 March 2006 using the standard version (www.google.com) and the Chinese version (www.google.com.cn) of Google search. 35. Original wording displayed by Google search was in Chinese characters. Pinyin and translation by the author. 36. John B. Thomson, ‘The Globalization of Communication’, in Held & McGrew, The Global Transformations Reader, pp. 246–59. 37. Jonathan D. Aronson, ‘Causes and Consequences of the Communications and Internet Revolution’, in John Baylis & Steve Smith (eds.), The Globalisation of World Politics: An Introduction to International Relations, 3rd edn (Oxford University Press, 2005), p. 641.

Informetrics and webometrics for measuring impact, visibility, and connectivity in science, politics, and business

The paper reports the application of classical bibliometric methods to evaluate the impact of scientific, political, and business developments. The novel aspect is to regard the Web as a citation network where the traditional information entities (scientific articles and the citations given to and taken from them) are replaced by webpages with external and internal links. The methodological studies are intended to test the usability of large international citation databases and the World Wide Web as feasible and reliable tools in quantitative analyses for gathering useful information for business intelligence. Formerly, the impact of authors and their scientific production was measured by the average citation frequencies of journals publishing their research: the Journal Impact Factor (JIF), calculated by the Institute for Scientific Information (ISI) in the United States and published annually in the Journal Citation Reports (JCR)—the most frequently used quantitative indicator to measure the quality/value/impact of research works published in the core international journals. It has been suggested that, by calculating the number of webpages pointing to a given site, analogously, a Web Impact Factor can be calculated as a way of comparing the attractiveness of sites or domains on the World Wide Web. A key to webometric studies has been the use of large-scale Web search engines such as Alta Vista that allow measurements to be made of the total number of pages in a Web space and links to those Web spaces. These search engines provide similar possibilities for the investigation of links between documents to those provided by the citation databases created by ISI. To illustrate the scope and nature of informetric research methods applied in competitive intelligence analysis, a short presentation is given of samples of research studies carried out at the Centre for Informetic Studies in Copenhagen (CIS). © 2001 John Wiley & Sons, Inc.

The anatomy of a large-scale hypertextual Web search engine

In this paper, we present Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext. Google is designed to crawl and index the Web efficiently and produce much more satisfying search results than existing systems. The prototype with a full text and hyperlink database of at least 24 million pages is available at http://google.stanford.edu/ To engineer a search engine is a challenging task. Search engines index tens to hundreds of millions of Web pages involving a comparable number of distinct terms. They answer tens of millions of queries every day. Despite the importance of large-scale search engines on the Web, very little academic research has been done on them. Furthermore, due to rapid advance in technology and Web proliferation, creating a Web search engine today is very different from three years ago. This paper provides an in-depth description of our large-scale Web search engine — the first such detailed public description we know of to date. Apart from the problems of scaling traditional search techniques to data of this magnitude, there are new technical challenges involved with using the additional information present in hypertext to produce better search results. This paper addresses this question of how to build a practical large-scale system which can exploit the additional information present in hypertext. Also we look at the problem of how to effectively deal with uncontrolled hypertext collections where anyone can publish anything they want.