Why you can’t afford to ignore the invisible Web

Abstract

While most users of the World Wide Web rely heavily on the standard range of commercial search engines and metasearching agents, they should be careful to note their limitations and inability to furnish access to material residing in the so-called “invisible Web”, which the search engines either cannot or will not index. Although it is felt by many users that the invisible Web is not worth covering, being full of duplication, “spam” and other ephemeral material, the fact remains that some of the best resources available on the Web exists in this form and to overlook it would be to detract from the full value of the Web. Users should seriously consider using the invisible Web on account of: the large size of the resource; the quality of the sites; the fact that a lot of the sites have quite specialized content within very clearly defined subject areas; the fact that it is often possible to tailor the output to meet users’ needs; and the fact that here may be the only location on the Web where the information can be found. The major obstacle to using the invisible Web is the intrinsic difficulty in accessing it. Personal recommendation can also be a useful means of being alerted to useful resources on the invisible Web but the main key to success lies in knowing where to look for such information and, having found a useful Web site, rely on it for future use.