A cautionary note on the variance of empirical pair correlation function

Abstract

Point processes are popular models for data consisting of a random configuration of points, such as positions of trees of a given species in a forest stand, locations of earthquakes' epicentres or positions of proteins in a cell membrane. Pair correlation function is a functional summary characteristic quantifying the interactions between pairs of points of the process at different distances. In practice, kernel estimators of the pair correlation function are mostly used, which require the user to choose the bandwidth of the smoothing kernel. When looking for the optimal bandwidth, higher order terms of the variance of the estimator are often neglected. We present the detailed formulas for the variance and argue that the higher order terms contribute the most to the variance and should not be ignored. Furthermore, we adapt a well‐known variance approximation formula to accommodate the popular translation edge correction factor. We compare the original and the adjusted approximation formula to the exact variance obtained by numerical integration and to the empirical variance obtained from simulated realizations. We conclude that the variance approximations do not perform well and that modern cross‐validation approaches to bandwidth selection should be preferred in practice.