Detection of Semantic Duplicates in Temporal Domains Using Directed Acyclic Graphs

Abstract

We demonstrate the feasibility of a generalized technique for semantic deduplication in temporal data domains using graph-based representations of data records. Structured data records with multiple timestamp attributes per record may be represented as a directed graph where the nodes represent the events and the edges represent event sequences. Edge weights are based on elapsed time between connecting nodes. In comparing two records, we may merge these directed graphs and determine a representative directed acyclic graph (DAG) inclusive of a subset of nodes and edges that maintain the transitive weights of the original graphs. This DAG may then be evaluated by weighting elapsed time equivalences between records at each node and measuring the fraction of nodes represented in the DAG versus the union of nodes between the records being compared. With this information, we establish a duplication score and use a specified threshold requirement to assert duplication. This method is referred to as Temporal Deduplication using Directed Acyclic Graphs (TD:DAG). TD:DAG significantly outperformed established ASNM and ASNM+LCS methods for datasets rep-resenting two disparate domains, COVID-19 government policy data and PlayStation Network (PSN) trophy data. TD:DAG produced highly effective and comparable F1 scores of 0.960 and 0.972 for the two datasets, respectively, versus 0.864/0.938 for ASNM+LCS and 0.817/0.708 for ASNM.