Duplicates Detection, Counting, and Removal

Abstract

The need to detect and eliminate duplicate elements arises in many applications such as the processing of relational database operations, comparison of complex objects, transitive closure, and protocol verification. Given a multiset, the process of detecting duplicates, eliminating them, sorting the remaining distinct elements, and counting the number occurrences of each in the multiset is a computationally intensive task, especially when complex objects are involved. In this paper, the computational complexity of performing such a task is addressed. The computational complexity study is based on a modified comparison-based decision tree. It is shown that, to a multiset M(n,L) of n elements L of which are distinct, we can associate a decision tree of L!S(n,L) external nodes. S(n,L) represents Stirling number of the second kind. This result suggests that upper and lower bounds to perform such a task are different from those of sorting a set of the same cardinality. It is also shown that a comparison based sorting algorithm can be adapted to perform such a task. In addition, the analytical performance of the adapted sorting algorithm is addressed.