Relative performance of expected and observed fisher information in covariance estimation for maximum likelihood estimates

Abstract

Covariance matrix and confidence interval calculations for maximum likelihood estimates (MLEs) are commonly used in system identification and statistical inference. To accurately construct such confidence intervals, one typically needs to know the covariance of the MLE. Standard statistical theory tells that the normalized MLE is asymptotically normally distributed with mean zero and covariance being the inverse of the Fisher Information Matrix (FIM) at the unknown parameter. Two common estimates for the covariance of MLE are the inverse of the observed FIM (the same as the Hessian of negative log-likelihood) and the inverse of the expected FIM (the same as FIM). Both of the observed and expected FIM are evaluated at the MLE from the sample data. We show that, under reasonable conditions, the expected FIM outperforms the observed FIM under a mean squared error criterion. This result suggests that, with certain conditions, the expected FIM is a better estimate for the covariance of MLE in confidence interval calculations.