The sign problem is a widespread numerical hurdle preventing us from simulating the equilibrium behavior of various problems at the forefront of physics. Focusing on an important sub-class of such problems, bosonic $(2+1)$-dimensional topological quantum field theories, here we provide a simple criterion to diagnose intrinsic sign problems---that is, sign problems that are inherent to that phase of matter and cannot be removed by any local unitary transformation. Explicitly, \textit{if the exchange statistics of the anyonic excitations do not form complete sets of roots of unity, then the model has an intrinsic sign problem}. This establishes a concrete connection between the statistics of anyons, contained in the modular $S$ and $T$ matrices, and the presence of a sign problem in a microscopic Hamiltonian. Furthermore, it places constraints on the phases that can be realised by stoquastic Hamiltonians. We prove this and a more restrictive criterion for the large set of gapped bosonic models described by an abelian topological quantum field theory at low-energy, and offer evidence that it applies more generally with analogous results for non-abelian and chiral theories.