Phonon lifetimes are a key component of quasiparticle theories of transport, yet first-principles lifetimes are rarely directly compared to inelastic neutron scattering (INS) results. Existing comparisons show discrepancies even at temperatures where perturbation theory is expected to be reliable. In this work, we demonstrate that the reciprocal space voxel ($q$-voxel), which is the finite region in reciprocal space required in INS data analysis, must be explicitly accounted for within theory in order to draw a meaningful comparison. We demonstrate accurate predictions of peak widths of the scattering function when accounting for the $q$-voxel in CaF$_2$ and ThO$_2$. Passing this test implies high fidelity of the phonon interactions and the approximations used to compute the Green's function, serving as critical benchmark of theory, and indicating that other material properties should be accurately predicted; which we demonstrate for thermal conductivity.