The response of semiarid grasslands to small, non‐colonial herbivores has received little attention, focusing primarily on the effects of granivore assemblages on annual plant communities. We studied the long‐term effects of both small and large herbivores on vegetation structure and species diversity of shortgrass steppe, a perennial semiarid grassland considered marginal habitat for small mammalian herbivores. We hypothesized that 1) large generalist herbivores would affect more abundant species and proportions of litter‐bare ground‐vegetation cover through non‐selective herbivory, 2) small herbivores would affect less common species through selective but limited consumption, and 3) herbivore effects on plant richness would increase with increasing aboveground net primary production (ANPP). Plant community composition was assessed over a 14‐year period in pastures grazed at moderate intensities by cattle and in exclosures for large (cattle) and large‐plus‐small herbivores (additional exclusion of rabbits and rodents). Exclusion of large herbivores affected litter and bare ground and basal cover of abundant, common and uncommon species. Additional exclusion of small herbivores did not affect uncommon components of the plant community, but had indirect effects on abundant species, decreased the cover of the dominant grass Bouteloua gracilis and total vegetation, and increased litter and species diversity. There was no relationship between ANPP and the intensity of effects of either herbivore body size on richness. Exclusion of herbivores of both body sizes had complementary and additive effects which promoted changes in vegetation composition and physiognomy that were linked to increased abundance of tall and decreased abundance of short species. Our findings show that small mammalian herbivores had disproportionately large effects on plant communities relative to their small consumption of biomass. Even in small‐seeded perennial grasslands with a long history of intensive grazing by large herbivores, non‐colonial small mammalian herbivores should be recognized as an important driver of grassland structure and diversity.