Scalable Change Analysis and Representation Using Characteristic Function

Abstract

In this paper, we propose a novel framework to help human operators- who are domain experts but not necessarily familiar with statistics- analyze a complex system and find unknown changes and causes. Despite the prevalence, researchers have rarely tackled this problem. Our framework focuses on the representation and explanation of changes occurring between two datasets, specifically the normal data and data with the observed changes. We employ two-dimensional scatter plots which can provide comprehensive representation without requiring statistical knowledge. This helps a human operator to intuitively understand the change and the cause. An analysis to find two-attribute pairs whose scatter plots well explain the change does not require high computational complexity owing to the novel characteristic function-based approach. Although a hyper-parameter needs to be determined, our analysis introduces a novel appropriate prior distribution to determine the proper hyper-parameter automatically. The experimental results show that our method presents the change and the cause with the same accuracy as that of the state-of-the-art kernel hypothesis testing approaches, while reducing the computational costs by almost 99% at the maximum for all popular benchmark datasets. The experiment using real vehicle driving data demonstrates the practicality of our framework.