Abstract Urbanization is a key driver of habitat loss and fragmentation worldwide, yet many urban ecosystems contain vegetated habitat patches that support diverse wildlife communities. Managing urban systems to support robust wildlife communities requires us to understand the mechanisms that drive the response of species to the urban environment. Small mammals are key components of terrestrial ecosystems (e.g., seed predators, prey) and likely also carry out these roles in urban ecosystems; however, the effects of urbanization on small mammal communities are understudied. To identify how species-specific responses to urban environments shape community composition, we quantified both overall community and species-specific changes in small mammal abundance across an urbanization gradient in a Midwestern US metropolitan area. We combined small mammal trapping and land cover data to generate a hierarchical community abundance model. Species diversity increased with increasing proportional cover of human-modified land cover (i.e., impervious surfaces and turfgrass). This finding is driven by high species diversity on sites bordering streams in mowed parks and low diversity on sites with high tree canopy closure. Additionally, modeling results indicated that species responded differently to landscape attributes, leading to variation in small mammal community composition across the urbanization gradient: prairie-associated species tended to be more abundant in tall vegetation bordering mowed parks while habitat generalists tended to be more abundant on sites with greater canopy closure and shrub cover. Our results suggest that studies that focus on community-level responses (e.g., species richness) to urbanization may miss important species-specific responses. It may be particularly important to assess both species-specific and community-level responses in cities at ecotones (e.g., between forest and grassland) where species with different habitat requirements may replace one another in different types of green spaces, thereby changing community composition without affecting species diversity or richness. Our findings also indicate that vegetated urban patches, especially patches with tall vegetation cover and low canopy cover, are important habitat for prairie-associated small mammal communities, providing conservation options in heavily altered landscapes.