Using patterns of belowground biodiversity to predict ecological functions and to manage productivity has been a longstanding objective for agronomists and ecologists. However, inconsistent associations between microbial diversity and fertilization have been found in frequency, which can be partly due to the dilution effect of immense silent species in microbial communities. Therefore, we studied the responding pool of the microbial communities, which was determined by filtering out taxa that did not vary significantly between fertilization treatments and the control. With responding species, we hypothesized that the difference of microbial response to different fertilization regimes and between bacterial and fungal response to treatments can be enlarged. In this study, bacterial and fungal community compositions and networks were characterized from soils with chemical fertilization (F), the combined application of chemical fertilizers and organic residues (MRF) and without fertilization (Control) in a 7-year experiment located at the Fengqiu Agroecological Experimental Station. As a result, >90% of the phyla, 75% of the genera and 92% of the OTUs were silent microbes. When filtering out these species, 9 phyla, 88 genera and 185 OTUs of bacteria responded significantly to the MRF treatment compared with those of the Control with an average abundance variation of 0.06% in responding OTUs. In contrast, only 2 phyla, 9 genera and 30 OTUs of bacteria significantly responded to the F treatment with an average amplitude of 0.11%. The higher connectedness and the lower average geodesic distance under the MRF treatment showed a more connective and closer bacterial network compared with the F treatment. Similar but stronger responses were detected in the fungal community composition and networks. To conclude, combined fertilization exerted a moderate impact on many microbial taxa (MM pattern), and chemical fertilization showed a substantial effect on a small amount of taxa (SS pattern). It was further inferred that a variation of the “MM” pattern can result in a more stable, harmonious and efficient ecosystem via maintenance of microbial diversity, stimulation of beneficial species, and exhibition of a more efficient microbial network. Our study showed explicit patterns of microbial communities under fertilization regimes revealed by responding species and highlighted the advantages of combined fertilization with organic matter in agroecosystems.