A multistage deep imputation framework for missing values large segment imputation with statistical metrics

Abstract

The presence of missing values is a pervasive and unavoidable phenomenon in sensor data. Despite numerous efforts from researchers to address this issue through imputation techniques, particularly in deep learning models, the unique data distributions and periods inherent in real-world sensor data are often neglected. This paper presents a novel, multistage deep learning-based imputation framework with adaptability to missing value imputation. The framework incorporates a mixture measurement index that accounts for both low- and higher-order statistical aspects of data distribution and a more adaptive evaluation metric, which improves upon traditional mean squared error. Additionally, a multistage imputation strategy and dynamic data length adjustment are integrated into the imputation process to account for variations in data periods. Empirical results on diverse sensor data demonstrate the superiority of the proposed framework, particularly in addressing large segment imputation issues, as evidenced by improved imputation performance. The implementation and experimental results have been made publicly available on GitHub. 11https://github.com/BomBooooo/MLSIF/tree/main.