It has been proposed that quantum features of the gravitational field can be exposed experimentally by employing gravity as a mediator of entanglement. We show that in order to witness this type of entanglement experimentally, strong limits on acceleration noise, which has been neglected in previous work, must be overcome. In the case of two particles of similar mass, Casimir-Polder forces lead to a fundamental limit of tenths of femtometers per second squared per root Hertz. Limits are between three and six orders of magnitude less strict for two particles of unequal mass, depending on collisional decoherence.