Evaluation of Event Impact on Key Performance Indicators

Abstract

Monitoring and understanding the performance of equipment and operations is a real and challenging problem in industrial and commercial settings. Although many operations have a set of agreed-upon Key Performance Indicators (KPIs) that have been defined by domain experts, most of these KPIs suffer from many issues including but not limited to sensitivity to seasonality and load conditions, approximate and inaccurate calculations, and measurement noise. This makes further analysis of these measurements a real challenge. This paper focuses on answering a simple yet crucial question in regards to KPI monitoring: "Have a given event made an impact on the performance of an operation?". The paper presents an approach to answer this question in a systematic and statistically-sound way while alleviating the aforementioned problems with the provided KPIs. The key idea behind the proposed approach is to predict the would-be performance as if the event has not happened. The approach then applies a three-way hypothesis testing on the difference between the would-be and actual performance to determine whether the event resulted in improvement, degradation or no-change in performance. Results from simulation study and experiments on real use cases demonstrate the effectiveness and usability of the proposed approach.