Traditional direct searches for dark matter, looking for nuclear recoils in deep underground detectors, are challenged by an almost complete loss of sensitivity for light dark matter particles. Consequently, there is a significant effort in the community to devise new methods and experiments to overcome these difficulties, constantly pushing the limits of the lowest dark matter mass that can be probed this way. From a model-building perspective, the scattering of sub-GeV dark matter on nucleons essentially must proceed via new light mediator particles, given that collider searches place extremely stringent bounds on contact-type interactions. Here we present an updated compilation of relevant limits for the case of a scalar mediator, including a new estimate of the near-future sensitivity of the NA62 experiment as well as a detailed evaluation of the model-specific limits from Big Bang nucleosynthesis. We also derive updated and more general limits on DM particles upscattered by cosmic rays, applicable to arbitrary energy- and momentum dependences of the scattering cross section. Finally we stress that dark matter self-interactions, when evaluated beyond the common s-wave approximation, place stringent limits independently of the dark matter production mechanism. These are, for the relevant parameter space, generically comparable to those that apply in the commonly studied freeze-out case. We conclude that the combination of existing (or expected) constraints from accelerators and astrophysics, combined with cosmological requirements, puts robust limits on the maximally possible nuclear scattering rate. In most regions of parameter space these are at least competitive with the best projected limits from currently planned direct detection experiments.