Abstract
Residential electricity load data can include numerous types of bad data, even clustered bad data, as they that are typically captured by simple measurement instruments. For example, in the case of a time-series of Not-a-Number (NaN) errors, the values before or next to a NaN may appear as the sum of actual values during the times of the NaN series. To utilize load data that includes such erroneous data for prediction or data mining analysis, customized detection and imputation should be conducted. This study proposes a new joint detection and imputation method for handling clustered bad data in residential electricity loads. Examples of these data are known invalid data points, such as consecutive NaN or zero values followed by or being ahead of an outlier. The proposed joint detection and imputation scheme first investigates the neighbors of the invalid data points, using probabilistic forecasting techniques. These techniques are implemented by the next valid neighbors to determine whether there is an anomaly or not. Then, adaptive imputations are applied on the basis of the detection, the candidate point should be imputed simultaneously or not. To assess the potential of the newly proposed scheme to characterize the clustered bad data, we analyzed the electricity loads of 354 households. Moreover, joint detection and imputations are conducted to test with the randomly injected synthesized clustered bad data (containing NaNs of various lengths) that is followed by the summation of the actual NaN values. The proposed scheme succeeded in detecting clustered bad data with an accuracy of 95.5% and a false alarm rate of 3.6% for all households in the dataset. Outlier detection-assisted imputation schemes are evaluated for NaNs with optional outliers. Results demonstrate that these schemes improve the overall accuracy significantly compared to schemes without outlier detection.
Highlights
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.With the growing concerns on energy and environmental sustainability, a huge research effort has been made to achieve a smart and efficient energy management for decreasing the carbon footprint [1,2]
true negative (TN) and false positive (FP) were improved in the proposed method; TN was increased by 6.8%, and FP was decreased by 63.0%
We implemented two cases to compare the performance of the accumulated outlier detection aware imputation (AOD-AI)—one without AOD-AI and the other one with AOD-AI
Summary
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Real residential energy consumption data has the tendency to include bad data, such as Not-a-Number (NaN) or zero points that are found scattered in several cases, even in the shape of clusters; and anomalies whose value is the sum of the actual values during the clustered bad data points. These outliers can significantly affect the performance of data-driven methods when handling clustered bad data. The imputation range should be changed to include the previous point with clustered bad data when the detection result suspects that the previous point is the outlier
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
