Charged particles scattering on moving inhomogenities of the magnetised interstellar medium can gain energy through the process of second-order Fermi acceleration. This energy gain depletes in turn the magnetic wave spectrum around the resonance wave-vector k∼ 1/RL, where RL is the Larmor radius of the charged particle. This energy transfer can prohibit the cascading of magnetic turbulence to smaller scales, leading to a drop in the diffusion coefficient and allowing the efficient exchange of charged dark matter particles in the disk and the halo. As a result, terrestial limits from direct detection experiments apply to charged dark matter. Together with the no-observation of a drop in the diffusion coefficient, this excludes charged dark matter for 103 GeV≲ m/q ≲ 1011 GeV, even if the charged dark matter abundance is only a small part of the total relic abundance.