Web Robot Research Articles

Knowledge retrieval and the World Wide Web

The Web is currently a distributed mass of simple hypertext documents. The authors present a new tool, WebKB that interprets semantic statements stored in Web-accessible documents. WebKB lets its users combine lexical, structural and knowledge-based techniques to exploit or generate Web documents. WebKB is an ontology server and directed Web robot.

Will technology kill the healthcare library?

Will technology kill the healthcare library?

Internet robots: a new robotics niche

Web robots are a concept that has captured the interest of researchers and developers. Their potential is being investigated at the University of Western Australia, where a six-axis robot has been made teleoperable over the World Wide Web. The site has been active since September 1994 and the robot has also been exhibited at the Australian National Science and Technology Centre in Canberra. Someone considering offering services with this technology will be interested in which groups of people find it attractive, the level of commitment that operators show, and how the robot users behave. These questions are addressed in the article with some demographic data on operators, an analysis of how long they remain interested, and an examination of how long they will wait to access the service. The user interface affects operator behavior, so some interface variations and their effect on operator actions are considered. The opportunities presented, likely application areas, and the potential for generating an income are examined.

Keep your bots to yourself [software law

There are now a variety of Web bots or spiders available to scour the Web to automatically gather information. Bots, sometimes called agents when they are instructed to perform particular searches, are useful in performing the time-consuming and repetitive searches required to gather comparison data or to find an elusive item. In some cases, agents can not only find information but can make a purchase or take another action based on the data they collect. Although bots are great for consumers, some Internet retailers hate them. Just as a brick and mortar store wants potential customers to walk in the door, Web retailers want potential customers to personally view their sites. Once there, consumers might purchase the initial products they were seeking and then see advertisements or buy accessories, upgrades, or unrelated additional items as well. Because bots can reduce or eliminate these possibilities, some Internet retailers are trying to stop them. It's one thing to be upset about bots, spiders, agents, and data miners, but it is quite another to put an end to the practice. It should come as no surprise that federal and state laws in the US do not expressly mention bots. Those seeking to challenge their use must resort to a variety of more generic laws, none of which fit very well.

Guiding Students in Finding information on the Web

Searching for information on the Web can be a daunting, frustrating, mind-bog gling, and sometimes-futile activity. To increase the odds of finding the right infor mation, one needs to understand the operation of four search tools: Web directo ries, search engines, indexes, and spiders or robots. Understanding Boolean logic also helps. Business communication instructors can aid students in their research by introducing them to the terminology and functions of an efficient Web search process.

World wide web robots: an overview

This paper traces the development of World Wide Web Robots and provides an overview of their main functions and workings. The focus is on search robots and illustrations will be drawn from two major search engines: AltaVista and Excite. In the concluding section, prblems associated with the use of Web Robots and their implications for electronic publishing will be examined.

ELFI: information brokering for the domain of research funding

The domain of finding and disseminating information on funding sources for research is introduced. We present a brokering system called ELFI (Electronic Funding Information) that supports researchers and funding consultants in this task. We discuss the architecture of the system, including the use of web robots for information gathering and the use of Active Views for personalized information delivery. Examples illustrate the ELFI system in action.

NETTIPS: Smarter Searching

Using the URL at the [end][1] of this item, readers can immediately offer feedback and suggestions on this topic. The importance of good search tools [[HN1][2]] has been apparent since the early days of the Internet. There are now an estimated 100 million Web pages, and with about 200,000 added each day, it is not surprising that the main complaint among users is the difficulty of finding the documents they want. An important class of tools is the search “engine,” which takes keywords as input and returns a set of Web documents. Effective use of these tools requires some knowledge of how they work. Search engines are not directories. The latter are sites where several thousand Web sites have been categorized by human reviewers. The largest is Yahoo! ([www.yahoo.com][3]), with links to about 370,000 Web sites. Although directories often point users to quality sites, they cannot possibly catalog all of the Web. For a more thorough search, users must turn to search engines. Because these engines are not limited by the availability of human filters, they can catalog literally millions of Web documents. Each engine consists of three distinct components: the spider, the index, and the query module. The spider (also referred to as the robot)[[HN2][4]] is a program that “moves” on the Net from page to page in search of new documents. Each robot uses its own specific algorithm [[HN3][5]] for finding and navigating Web pages. The harvested pages are then entered into a database on one of the search engine's computers. The database is organized by an index whose architecture is also specific for each search engine. Users search the index through a predefined query module, an interface specific to each engine. Two concepts should be borne in mind. First, search engines do not scour the Net in real time, but rather query an index of Web pages compiled on the search engine's site. This explains why some search results point to outdated or nonexistent links. Indeed, it may take days or weeks for any particular search engine to traverse the entire Net. Second, the same query on two different search engines will not yield identical results, because the combination of proprietary robot, index, and user interface will be unique. The best strategy is to try several engines when searching for a specific Web document. With the proliferation of commercial search engines, a new tool—the metasearch engine—has occupied a slot above them all. Metasearch engines will send a query to several other engines in parallel and return a composite report (with duplicate entries removed). This can save the effort of performing several searches in series. But metaengines tend to be slower and tend to return a maximum of about only 100 results. Because each engine has its own quirks, it is worth reading the search tips usually listed on the home page. Armed with a few search engines and perseverance, users should be able to find relevant documents in what is fast becoming the world's largest library. Some top search engines are listed in the accompanying Site Finder section, and links to additional search sites can be found at [www.MedsiteNavigator.com/techsight/nettips_3.html][6] ### General Hypernotes 1. This WWW Search Engines Register site is for the expert. It compiles a running tab of the most significant [search tools][7] in current operation. 2. Here you can find details of how to build and operate your own [Web robot][8]. 3. The [Harvest project][9] at the University of Colorado is a popular search and retrieve algorithm for Web search engines. It can be obtained free of charge. [1]: /lookup/doi/10.1126/science.277.5328.976 [2]: #p-9 [3]: http://www.yahoo.com [4]: #p-10 [5]: #p-11 [6]: http://www.MedsiteNavigator.com/techsight/nettips_3.html [7]: http://coombs.anu.edu.au/CoombswebPages/SearchEngines.html [8]: http://info.webcrawler.com/mak/projects/robots/robots.html [9]: http://harvest.transarc.com/

Eight Internet search engines compared

Compares search performance and special features of eight robotic Internet search engines, which use a Web robot to retrieve a significant number of documents from the World Wide Web.