If dark matter has strong self-interactions, future astrophysical and cosmological observations, together with a clearer understanding of baryonic feedback effects, might be used to extract the velocity dependence of the dark matter scattering rate. To interpret such data, we should understand what predictions for this quantity are made by various models of the underlying particle nature of dark matter. In this paper, we systematically compute this function for fermionic dark matter with light bosonic mediators of vector, scalar, axial vector, and pseudoscalar type. We do this by matching to the nonrelativistic effective theory of self-interacting dark matter and then computing the spin-averaged viscosity cross section nonperturbatively by solving the Schrödinger equation, thus accounting for any possible Sommerfeld enhancement of the low-velocity cross section. In the pseudoscalar case, this requires a coupled-channel analysis of different angular momentum modes. We find, contrary to some earlier analyses, that nonrelativistic effects only provide a significant enhancement for the cases of light scalar and vector mediators. Scattering from light pseudoscalar and axial vector mediators is well described by tree-level quantum field theory.