Exploratory Data Analytics and PCA-Based Dimensionality Reduction for Improvement in Smart Meter Data Clustering

Abstract

The smart meter sends the meter readings to the utilities at desired frequency allowing better visibility of consumer electricity consumption behaviour by providing more data points for in-depth analysis and for generating insights using advanced data analytics and data science techniques. The granulated data helps utilities in designing schemes for audience suitable for demand response management to shift the peak hour demand to off-peak hours. In this paper, a method is proposed for load profile segmentation which can be used by utilities for identifying the characteristics of different users and targeting those whose demand curve can be flattened during peak hours with various demand response management schemes. Firstly, exploratory data analysis is done on the cleaned dataset to find the optimal epoch size, understand the distribution of data in each epoch, and use it for dimensionality reduction. For reducing the clustering computation time, dimensionality reduction is done by around 64% using Principal Component Analysis. The first six principal components are identified as carrying maximum variance using the cumulative variance technique in each epoch. Unsupervised Machine Learning based k-means clustering technique is applied to these principal components. The optimal value of k is evaluated using the WCSS technique where k = 5 and k = 3 for residential and SME users respectively is found. The average silhouette coefficient for residential users is 0.48 and for SME users is 0.51. Hence, well-separated clusters are formed with minimum intra-cluster distance using PCA for dimensionality reduction which is used for load profile segmentation and Post Clustering Analysis.