A framework for evaluating semantic annotations of Web services: A network theory based approach for measuring annotation quality

Abstract

In the past years various methods have been developed which require semantic annotations of Web services as an input. Such methods typically leverage discovery, match-making, composition and execution of Web services in dynamic settings. At the same time a number of automated Web service annotation approaches have been proposed for enabling application of these methods in settings where it is not feasible to provide the annotations manually. However, lack of effective automated evaluation frameworks has seriously limited proper assessment of the constructed annotations in settings where the overall annotation quality of large quantities of Web services needs to be evaluated. This paper describes an evaluation framework for measuring the quality of semantic annotations for a large number of real-world Web services from heterogeneous application domains. The evaluation framework is generally based on analyzing properties of Web service networks constructed from semantic annotations of the Web services. More specifically, we measure scale-free, small-world and correlation degree properties of the networks to evaluate the overall quality of annotations. The evaluation is demonstrated using annotations constructed semi-automatically for a set of publicly available WSDL documents containing descriptions of about 200 000 Web service operations.