Soil microbiomes play important roles in supporting agricultural ecosystems. However, it is still not well-known how soil microbiomes and their functionality respond to fertilization in various cropping systems. Here we examined the effects of 36 years of phosphorus, nitrogen, and manure application on soil bacterial communities, functionality and crop productivity in three contrasting cropping systems (i.e., continuous leguminous alfalfa (AC), continuous winter wheat (WC), and grain-legume rotation of winter wheat + millet - pea - winter wheat (GLR)) in a highland region of China’s Loess Plateau. We showed that long-term fertilization significantly affected soil bacterial communities and that the effects varied with cropping system. Compared with the unfertilized control, fertilization increased soil bacterial richness and diversity in the leguminous AC system, whereas it decreased those in the GLR system. Fertilization, particularly manure application, enlarged the differences in soil bacterial communities among cropping systems. Soil bacterial communities were mostly affected by the soil organic carbon and nitrogen contents in the WC and GLR systems, but by the soil available phosphorous content in the AC system. Crop productivity was closely associated with the abundance of fertilization-responsive taxa in the three cropping systems. Our study highlights that legume and non-legume cropping systems should be disentangled when assessing the responses of soil microbial communities to long-term fertilizer application.