Astrophysical observations spanning dwarf galaxies to galaxy clusters indicate that dark matter (DM) halos are less dense in their central regions compared to expectations from collisionless DM N-body simulations. Using detailed fits to DM halos of galaxies and clusters, we show that self-interacting DM (SIDM) may provide a consistent solution to the DM deficit problem across all scales, even though individual systems exhibit a wide diversity in halo properties. Since the characteristic velocity of DM particles varies across these systems, we are able to measure the self-interaction cross section as a function of kinetic energy and thereby deduce the SIDM particle physics model parameters. Our results prefer a mildly velocity-dependent cross section, from σ/m≈2 cm^{2}/g on galaxy scales to σ/m≈0.1 cm^{2}/g on cluster scales, consistent with the upper limits from merging clusters. Our results dramatically improve the constraints on SIDM models and may allow the masses of both DM and dark mediator particles to be measured even if the dark sector is completely hidden from the standard model, which we illustrate for the dark photon model.