Aggregate queries on constrained probabilistic similarity join pairs

Abstract

Join aggregate queries, where join operations are followed by aggregate operations, are very common in data processing. In some application scenarios where data are incomplete and ambiguous, probabilistic similarity join (PSJ) is widely used, which assigns each joined pair a probability to reflect the likelihood that the pair belongs to the join result set. According to the mapping constraints, we formally define the possible world semantics for three PSJ types (i.e., many-to-many, one-to-many, and one-to-one), and propose algorithms to evaluate aggregate queries on these constrained PSJ pairs. First, for many-to-many PSJ pairs, we model them with a tuple-level uncertainty model, and propose two aggregate algorithms based on dynamic programming and divide-and-conquer strategy, respectively. Then, we model one-to-many PSJ pairs with an attribute-level uncertainty model, and extend the aggregate algorithms for many-to-many PSJ pairs to this model. Finally, we model one-to-one PSJ pairs with a probabilistic graphical model, and propose a new aggregate algorithm that is based on a combination of generating function method, dynamic programming, and divide-and-conquer strategy. Extensive experiments on real datasets have demonstrated order-of-magnitude improvements of our algorithms over baselines.