Collection Of Web Pages Research Articles

Effects of Term Distributions on Binary Classification

In order to support decision making, text classification is an important tool. Recently, in addition to term frequency and inverse document frequency, term distributions have been shown to be useful to improve classification accuracy in multi-class classification. This paper investigates the performance of these term distributions on binary classification using a centroid-based approach. In such one-against-the-rest, there are only two classes, the positive (focused) class and the negative class. To improve the performance, a so-called hierarchical EM method is applied to cluster the negative class, which is usually much larger and more diverse than the positive one, into several homogeneous groups. The experimental results on two collections of web pages, namely Drug Information (DI) and WebKB, show the merits of term distributions and clustering on binary classification. The performance of the proposed method is also investigated using the Thai Herbal collection where the texts are written in Thai language.

Reducing human interactions in Web directory searches

Consider a website containing a collection of webpages with data such as in Yahoo or the Open Directory project. Each page is associated with a weight representing the frequency with which that page is accessed by users. In the tree hierarchy representation, accessing each page requires the user to travel along the path leading to it from the root. By enhancing the index tree with additional edges (hotlinks) one may reduce the access cost of the system. In other words, the hotlinks reduce the expected number of steps needed to reach a leaf page from the tree root, assuming that the user knows which hotlinks to take. The hotlink enhancement problem involves finding a set of hotlinks minimizing this cost. This article proposes the first exact algorithm for the hotlink enhancement problem. This algorithm runs in polynomial time for trees with logarithmic depth. Experiments conducted with real data show that significant improvement in the expected number of accesses per search can be achieved in websites using this algorithm. These experiments also suggest that the simple and much faster heuristic proposed previously by Czyzowicz et al. [2003] creates hotlinks that are nearly optimal in the time savings they provide to the user. The version of the hotlink enhancement problem in which the weight distribution on the leaves is unknown is discussed as well. We present a polynomial-time algorithm that is optimal for any tree for any depth.

Visualization of web spaces

The World Wide Web is a dominant global communication medium and knowledge repository. It is used by a great number of people with a variety of computer skills hence its usability is critical. As with many large information collections, the challenge with web usability is understanding the structure of a collection of information objects (web pages) to find relevant ones for satisfying a specific information need. Web sites are organized in a hyperlinked structure that somewhat addresses this challenge. However, this "connectedness" also causes the now well-known "lost in cyberspace" phenomenon where one may get confused within the complex organization of a web site. Meanwhile, information exploration on the web is not limited to browsing a web site. The problem of finding relevant information applies to a collection of pages that come from various web sites as in the case of the results of a "less than perfectly constructed" search query. Information visualization has been proposed as a way to cope with these problems by taking advantage of people's innate perceptual skills to support their cognitive skills. Many paradigms have been proposed for the visual presentation of web spaces (i.e. structured or unstructured collection of web pages). This study surveys these paradigms to provide a map of where the research in this field is, and what directions future research and practice can take. For this, we introduce a classification scheme to help in the systematic understanding of web visualization and for providing a framework for the development of future visualizations.

Automatic Detection of Shared Fragments in Large Collections of Web Pages and its Applications

To reduce network-related delays in serving dynamic web pages, various approaches have been proposed. However, one of the common fundamental problems encountered in some representatives of them is how to automatically find shared fragments in large numbers of web pages. Besides, this problem is also encountered in studies of web content characteristics at fragment granularity. This paper gives a formal definition of the problem, presents an efficient and scalable algorithm for it, and introduces the applications of the algorithm. In the problem definition, we introduce the notion of compound fragment, and our definition of maximal shared fragment captures the real characteristics of fragments that are appropriate for delivery and caching individually. Our algorithm has two unique features: (1) it is able to find real maximal shared fragments (2) it is able to effectively handle large collections of web pages by utilizing database techniques. The algorithm has been implemented and applied to 16 large sets of web pages. The experiments show that the algorithm can effectively handle large numbers of web pages, and can provide significant bandwidth saving and latency reduction when used in fragment-based web caching.

The impact of web site structure on link analysis

PurposeThis web mining paper aims at analyzing whether the number of links pointing to a web page is biased by the structure of web sites.Design/methodology/approachBy web‐design mining methods, two collections of web pages are extracted and the in‐links counts are determined by querying web search engines.FindingsThe paper finds that the structure bias and pages on a higher hierarchical level are likely to receive more links than other pages.Research limitations/implicationsThe links are set by web page authors whose behaviour is not yet fully understood and which requires further research.Practical implicationsThe paper shows that the structure bias of in‐links should be considered by link analysis measures used in search engines. Potential consequences are presented.Originality/valueThe number of links toward a web page are considered to be an indicator for the quality of that page. This measure is biased and cannot be solely trusted as a knowledge source for the quality of a page.

A method for language‐specific Web crawling and its evaluation

AbstractMany countries have created Web archiving projects aiming at long‐term preservation of Web information, which is now considered precious in cultural and social aspects. However, because of its borderless character, the Web poses obstacles to comprehensively gathering information originating in a specific nation or culture.This paper proposes an efficient method for selectively collecting Web pages written in a specific language. First, a linguistic graph analysis of real Web data obtained from a large crawl is conducted in order to derive a crawling guideline, which makes use of language attributes per Web server. The guideline then is formed into a few variations of link selection strategies. Simulation‐based evaluation reveals that one of the strategies, which carefully accepts newly discovered Web servers, shows superior results in terms of harvest rate/coverage and runtime efficiency. © 2007 Wiley Periodicals, Inc. Syst Comp Jpn, 38(2): 10–20, 2007; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.20693

Web Page Retrieval System by Automatic Detection of Topic Words

In order to search out some documents on a specific topic from a huge amount of documents on the Internet, we sometimes use a Web site that is called a search engine and retrieve documents by using a keyword. However, since a lot of pages are obtained by a keyword, it is difficult to find out the page that we really want from the obtained pages. In this paper, we propose a system supporting to look for web pages on the Internet that a user desires to find. The system actively collects web pages of a topic that relates to a keyword specified by a user and presents them to the user. The experimental result shows the effectiveness of the proposed system.

Computing customized page ranks

In this article, we present a new approach to page ranking. The page rank of a collection of Web pages can be represented in a parameterized model, and the user requirements can be represented by a set of constraints. For a particular parameterization, namely, a linear combination of the page ranks produced by different forcing functions, and user requirements represented by a set of linear constraints, the problem can be solved using a quadratic programming method. The solution to this problem produces a set of parameters which can be used for ranking all pages in the Web. We show that the method is suitable for building customized versions of PageRank which can be readily adapted to the needs of a vertical search engine or that of a single user.

Ideological Discourses in the Making of Internet Career Sites

This paper examines the ideological discourses evident in related categories of commercial World Wide Web sites during the last year of the dot-com bubble (1995–2001). It analyzes two collections of employment-related Web pages: third-party Internet career sites (such as Monster.com) and the employee recruitment sections of corporate Web sites (such as General Electric). After reviewing the historical development of Internet use for job search purposes and the effect on labor market dynamics, I estimate the extent to which discourses of community permeate these sites by using methodologies that focus on the content of Web sites and the messages conveyed by advertisements. The conclusions highlight the importance of considering ideological constructions in grasping the meaning of “discursive domains” among commercial sites on the Web.

Stanford WebBase components and applications

We describe the design and performance of WebBase, a tool for Web research. The system includes a highly customizable crawler, a repository for collected Web pages, an indexer for both text and link-related page features, and a high-speed content distribution facility. The distribution module enables researchers world-wide to retrieve pages from WebBase, and stream them across the Internet at high speed. The advantage for the researchers is that they need not all crawl the Web before beginning their research. WebBase has been used by scores of research and teaching organizations world-wide, mostly for investigations into Web topology and linguistic content analysis. After describing the system's architecture, we explain our engineering decisions for each of the WebBase components, and present respective performance measurements.

Wikis for Teaching and Learning

An academic course presents an opportunity for all participants to collaborate for improving their knowledge. This collective improvement of knowledge is typically documented via material provided by the instructor and notes and assignments prepared by students. Over the course of a semester, these materials provide not only the collective knowledge in that course, but also provide a chronological history of how the knowledge base evolved. The focus of this study is wiki collaboration in teaching & learning contexts. A wiki is a medium in which a group of individuals can work together asynchronously on an idea and easily capture the essence in a reusable format. Technically, a wiki is a collection of hyperlinked Web pages that are assembled with wiki software. With wikis, the line between reader and contributor is intentionally blurred. Further, wiki use reflects the view of an instructor as one who facilitates information sharing among learners rather than simply transmitting knowledge from themselves to their students. Our initial motivation to explore the usefulness of wikis for teaching and learning was driven by the fact that wikis provide a medium in which several individuals could asynchronously work together on an idea and easily capture the essence in a reusable format. We found that: (1) Wikis can be can be used for a variety of tasks ranging from business analyses to analyzing policy positions to serving as a signup sheet for students to self organize; (2) Instructor support and facilitation is key; and (3) Ease-of-use issues are present but do not preclude success.

Introducing JCE ChemInfo: Organic

“JCE ChemInfo: Organic” is a collection of Web pages containing information useful to teachers, researchers, and students in organic chemistry, biochemistry, and medicinal chemistry. The pages have...

The SPIRIT collection

A large scale collection of web pages has been essential for research in information retrieval and related areas. This paper provides an overview of a large web collection used in the SPIRIT project for the design and testing of spatially-aware retrieval systems. Several statistics are derived and presented to show the characteristics of the collection.

Discovering and Analyzing World Wide Web Collections

With the explosive growth of the World Wide Web, it is becoming increasingly difficult for users to discover Web pages that are relevant to a topic. To address this problem we are developing a system that allows the collection and analysis of Web pages related to a particular topic. In this paper we present the system’s overall architecture and introduce the focused crawler used by the system. We also discuss the various techniques we use to allow the user to analyze and gain useful insights about a collection. Finally, we present some statistics on the collections.

Hierarchical Categorization Based on k-Nearest Neighbor Approach for Web Site Classification

Automatic categorization is a viable method to deal with the scaling problem on the World Wide Web. In this paper, we consider how to apply general text categorization techniques to Web site classification tasks. Two relevant issues concern, first, the case where the classifying object is not one document, as with a home page, but a Web site, which is a collection of Web pages. Second, real world Web directories have a complex hierarchical structure, in which leaf and non-leaf categories are directly assigned to Web sites, unlike the hierarchical structure treated in most previous research. On the first issue, this paper proposes the use of Web pages linked by a home page in addition to the home page itself. To accomplish this, we propose a Web site classification method based on connectivity analysis, as well as content analysis of Web sites. On the second issue, the hierarchical structure of classifiers is transformed into a flattened structure, but the classifier for each category uses features of its next-of-kin categories to take advantage of the hierarchical relationship. In experiments on a Korean commercial Web directory, the proposed classification method achieved an amazing improvement of microaveraging breakeven point by 36.6%, compared with an ordinary classifier.

Uncovering Information Hidden in Web Archives

The Internet has turned into an important aspect of our information infrastructure and society, with the Web forming a part of our cultural heritage. Several initiatives thus set out to preserve it for the future. The resulting Web archives are by no means only a collection of historic Web pages. They hold a wealth of information that waits to be exploited, information that may be substantial to a variety of disciplines. With the time-line and metadata available in such a Web archive, additional analyzes that go beyond mere information exploration become possible. In the context of the Austrian On-Line Archive (AOLA), we established a Data Warehouse as a key to this information. The Data Warehouse makes it possible to analyze a variety of characteristics of the Web in a flexible and interactive manner using on-line analytical processing (OLAP) techniques. Specifically, technological aspects such as operating systems and Web servers used, the variety of file types, forms or scripting languages encountered, as well as the link structure within domains, may be used to infer characteristics of technology maturation and impact or community structures.

CI Spider: a tool for competitive intelligence on the Web

Competitive Intelligence (CI) aims to monitor a firm's external environment for information relevant to its decision-making process. As an excellent information source, the Internet provides significant opportunities for CI professionals as well as the problem of information overload. Internet search engines have been widely used to facilitate information search on the Internet. However, many problems hinder their effective use in CI research. In this paper, we introduce the Competitive Intelligence Spider, or CI Spider, designed to address some of the problems associated with using Internet search engines in the context of competitive intelligence. CI Spider performs real-time collection of Web pages from sites specified by the user and applies indexing and categorization analysis on the documents collected, thus providing the user with an up-to-date, comprehensive view of the Web sites of user interest. In this paper, we report on the design of the CI Spider system and on a user study of CI Spider, which compares CI Spider with two other alternative focused information gathering methods: Lycos search constrained by Internet domain, and manual within-site browsing and searching. Our study indicates that CI Spider has better precision and recall rate than Lycos. CI Spider also outperforms both Lycos and within-site browsing and searching with respect to ease of use. We conclude that there exists strong evidence in support of the potentially significant value of applying the CI Spider approach in CI applications.

Towards personalized Internet health information: the STEPPS architecture.

STEPPS (STructured Evaluated Personalized Patient Support) proposes a strategy for integration of electronic patient records with Internet health-related content and its consequent use in personalized information retrieval for patient education. The application domain is the post-discharge support of burn patients in the Netherlands. We developed an electronic patient record interface for structured data collection in burn care. The system's thesaurus was projected to UMLS terms and the corresponding codes were incorporated in our software. A list of topics central to burn patient education was identified and a collection of related Web pages was compiled using meta-search software (Copernic2001Pro). The HTML pages were filed into catalogues by the Collexis indexing-matching software, using the UMLS Metathesaurus as indexing vocabulary. The bilingual (English and Dutch) structured data interface is currently used to create a database of retrospective patient data. Each patient's profile, i.e. set of characteristics employed to personalize information retrieval, can be automatically extracted. We have assembled a collection of more than 2500 Internet pages containing relevant information for burn patients. When patient data is available, the Collexis matching engine will accept the patient's profile as input and retrieve the most relevant HTML documents available in the catalogues. We have addressed some basic issues around the technical feasibility of linking electronic patient record data to online content. Although the functionality of STEPPS is not yet optimal, it contributes to the efforts towards improved relevance of information retrieval. Electronic patient record applications in conjunction with Internet resources can give a significant boost to the availability of tailored health education material. In this context, quality assurance of online health information is an indispensable element.

Building a distributed full-text index for the web

We identify crucial design issues in building a distributed inverted index for a large collection of Web pages. We introduce a novel pipelining technique for structuring the core index-building system that substantially reduces the index construction time. We also propose a storage scheme for creating and managing inverted files using an embedded database system. We suggest and compare different strategies for collecting global statistics from distributed inverted indexes. Finally, we present performance results from experiments on a testbed distributed Web indexing system that we have implemented.

The W Life Cycle Model and Associated Methodology for Corporate Web Site Development

The last few years witnessed the increasing internal and external use of the Internet by organizations. Web sites grew in sophistication from conventional sites composed of a simple collection of Web pages for public relations or marketing to complex Web information systems dealing with business-tocustomer transactions or business-to-business networks. However, most organizations still do not have a formal process of Web site development, and corporate Web sites are often developed in an unorganized and uncoordinated fashion. The results of this chaotic situation include slow delivery, conflicting standards, discrepancies with respect to corporate objectives, and redundant development efforts. To help alleviate this situation, this paper presents the W software life cycle model for corporate web site development along with an associated methodology to guide Web development groups in their endeavors. The proposed process model and methodology are based on insights gleaned by studying development projects for three types of corporate Web sites: intranets, Web-presence sites, and transactional sites. However, because Web information systems are more complex, the article also provides Communications of AIS, Volume 5 Article 7 3 The W Lifecycle Model and Associated Methodology for Corporate Web Site Development by L. B. Sherrell and L. Chen recommendations for how the methodology can be adapted to handle these types of applications better.