Trends in the planet population with host star mass provide an avenue to constrain planet formation theories. We derive the planet radius distribution function for Kepler stars of different spectral types, sampling a range in host star masses. We find that M dwarf stars have 3.5 times more small planets (1.0-2.8 R_Earth) than main-sequence FGK stars, but two times fewer Neptune-sized and larger planets (>2.8 R_Earth). We find no systematic trend in the planet size distribution between spectral types F, G, and K to explain the increasing occurrence rates. Taking into account the mass-radius relationship and heavy-element mass of observed exoplanets, and assuming those are independent of spectral type, we derive the inventory of the heavy-element mass locked up in exoplanets at short orbits. The overall higher planet occurrence rates around M stars are not consistent with the redistribution of the same mass into more, smaller planets. At the orbital periods and planet radii where Kepler observations are complete for all spectral types, the average heavy-element mass locked up in exoplanets increases roughly inversely with stellar mass from 4 M_Earth in F stars to 5 M_Earth in G and K stars to 7 M_Earth in M stars. This trend stands in stark contrast with observed protoplanetary disk masses that decrease towards lower mass stars, and provides a challenge for current planet formation models. Neither models of in situ formation nor migration of fully-formed planets are consistent with these results. Instead, these results are indicative of large-scale inward migration of planetary building blocks --- either through type-I migration or radial drift of dust grains --- that is more efficient for lower mass stars, but does not result in significantly larger or smaller planets.