MPdist-based missing data imputation for supporting big data analyses in IoT-based applications

Abstract

Recent years have witnessed an enormous growth in the number of wireless IoT devices and thereby Internet of Things (IoT) is classified as one of the novel cutting edge technologies that has redesigned the traditional industry into a smart industry, given its applicability for a wide range of applications in providing data-driven decision-making. Meanwhile, the data with missing values is still emerging as one of the longstanding challenges in an IoT architecture due to the common-mode failures, potentially leading to both bias and loss of precision. In spite of the fact that numerous techniques have been developed for imputing missing values, the major issue in terms of imputation precision or computational complexity for large missing subsequences is still a matter under debate. To address this issue, this paper proposes a newly developed algorithm called MP-BMDI that ensures high imputation performance for supporting big data analyses in IoT-based applications, where the absence of large missing subsequences is fully required to offer unbiased results. In our approach, we initially seek a finite number of subsequences that are mostly similar to the subsequence before the missing values, then adjust the height of these following subsequences to optimal locations. Once the most proper subsequence for replacing is chosen among them based on the pattern score function PSF(r) introduced in this paper, the missing gap is completely filled by the corresponding subsequence. Numerical results are here presented to validate the merits of the proposed algorithm compared to the alternative benchmark approaches by leveraging sensor data collected from real-time environmental monitoring and deliver significant insights on the effectiveness of the proposed algorithm from various perspectives.