Clustering Algorithms and Latent Semantic Indexing to Identify Similar Pages in Web Applications

Abstract

In this paper, we analyze some clustering algorithms that have been widely employed in the past to support the comprehension of Web applications. To this end, we have defined an approach to identify static pages that are duplicated or cloned at the content level. This approach is based on a process that first computes the dissimilarity between Web pages using latent semantic indexing, a well known information retrieval technique, and then groups similar pages using clustering algorithms. We consider five instances of this process, each based on three variants of the agglomerative hierarchical clustering algorithm, a divisive clustering algorithm, k-means partitional clustering algorithm, and a widely employed partitional competitive clustering algorithm, namely Winner Takes All. In order to assess the proposed approach, we have used the static pages of three Web applications and one static Web site.