Building HyperView wrappers for publisher Web sites

Abstract

Electronic journals are becoming a major source of scientific information. Researchers interested in certain topics only do not have time to scan all possibly relevant journals on a regular basis. A digital library can assist them by providing a uniform, searchable interface for electronic journals. For this purpose, a catalogue of metadata on the available journals such as authors and titles of articles must be established by the digital library. If there is no cooperation with journal publishers, this metadata must be extracted from the publishers’ Web sites, overcoming the intrinsic heterogeneity problems. Within the framework of the ongoing Natural Sciences Digital Library project DARWIN at the Free University of Berlin, we have developed a wrapper-mediator mechanism that copes with the heterogeneity problems of automatic metadata acquisition. It is based on our generic HyperView methodology for integration of Web sites. This methodology is distinguished from ad hoc solutions by two elegant and effective features. First, the structure of the publisher site is specified with abstract graph-schemata, instead of being hard-coded in scripts for data acquisition. Second, a powerful view concept based on declarative graph-transformation rules is used for information extraction. This view concept observes the information extracted from a Web site as a virtual graph that is materialized when being traversed. By using the HyperView methodology the effort for wrapping a Web site is considerably reduced. This is important because of the number of publisher Web sites involved in our project and the need to adapt wrappers to the structural evolution of those sites.