Estimating Hidden Parameters from a Dataset with Large Amount of Missing Data

Abstract

Listen

Translate article

Dealing with missing data is a fundamental problem in data science. In some rare cases, the large amount of missing data can be a big problem for any data analysing tasks. In this paper, we propose a method based on maximum likelihood estimation and expectation maximization to get estimation on the hidden parameter with large amount of missing data in the dataset. We perform the numerical experiments to validate the feasibility and stability of our method. We test the algorithm for different patterns of missing data, different amount of samples and different noise levels. The result indicates that the algorithm is effective in handling large amount of missing data if enough samples are provided.