On the visibility of information on the Web: an exploratory experimental approach

Abstract

On the Web, information that is not presented by the search engine in response to a specific query is in fact inaccessible. This problem has been defined as the invisible web or the deep web problem. Specific characteristics of the format of information may make it inaccessible to the crawlers that create the databases of search engines. We explore the dimensions of the invisible Web in the European Research Area. We propose that information visibility is an emergent property of the Web as complex system. Visibility of information is a highly unstable feature that is determined by a complex interaction between the local structure of the web environment, the search engine, the websites on which the information resides, the format of the information, and the temporal dimensions of the search.