An empirical exploration of performance metrics for event detection algorithms in Non-Intrusive Load Monitoring

Abstract

The field of Non-Intrusive Load Monitoring (NILM) gained prominence due to its promise of inferring the energy consumption of individual appliances by analyzing only the aggregated consumption. Still, despite some research efforts towards producing meaningful comparisons between approaches, it is not yet possible to find a proven and formally accepted set of metrics to do this. Against this background, this paper focuses on understanding the challenges of defining a consistent set of performance metrics for this problem. More concretely, it reports on an empirical exploration of 23 performance metrics’ behavior when applied to event-detection algorithms, identifying relationships and clusters between them. The results indicate that when applied to this problem, the performance metrics will show some considerable differences in behavior compared to other, more traditional, machine-learning domains. The results also suggest that most of the differences occur due to the unbalanced nature of the event detection problem, in which the number of positive cases (True Positives and True Negatives) is much higher than the number of false situations (False Positives and False Negatives). Furthermore, the results suggest that additional research is needed to find proper domain-specific performance metrics that take full consideration of the properties of the aggregated load.