To compare the differences in the soil microbial community structure in yellow soil paddy fields after the long-term application of chemical fertilizer, the role and mechanism of chemical fertilizer in maintaining soil microbial diversity were analyzed, and a theoretical basis for fertilization management in farmlands was provided. In this study, long-term experiments were conducted at the Scientific Observation and Experimental Station of the Arable Land Conservation and Agricultural Environment (Guizhou), Ministry of Agriculture and Rural Affairs. Soil samples were collected from five treatments: fertilizer N, P, and K (NPK); fertilizer P and K (PK); fertilizer N and K (NK); fertilizer N and P (NP); and no fertilizer (CK). High-throughput sequencing technology was used to analyze the microbial composition and diversity of the colonies, and the influencing factors are discussed. An analysis of the soil bacterial α diversity indices under the different fertilization treatments revealed that the long-term application of NPK fertilizers had less of an effect on the soil bacterial α diversity indices than did the CK. The long-term application of chemical fertilizers significantly reduced the soil bacterial Chao1, Shannon, and Simpson indices, while there was no significant difference in the bacterial Pielou e index among the treatments. The long-term application of chemical fertilizers significantly increased the soil fungal Chao1 index, but the effects on the other indices were not significant (p < 0.05). An analysis of the bacterial and fungal species under different fertilization treatments showed found that compared with CK, the long-term application of chemical fertilizer increased the relative abundance of Proteobacteria to varying degrees while reducing the relative abundance of Chloroflexi. The impact of other phyla was relatively small, and the difference in the relative abundance of fungi was not significant (p < 0.05). Principal component analysis revealed that, at the genus level, the bacterial and fungal community structures in the CK and NK treatments were relatively independent, while those in the NPK, PK, and NP treatments were similar. Random forest analysis revealed that OM, TP, and TK are the dominant factors that affect soil bacteria α diversity. The dominant factors affecting fungi α diversity are pH, OM, and AK. Redundancy analysis indicated that AK and TP were the main factors affecting bacterial community structure, while AP, AK, and pH were the main factors affecting fungal community structure. The conclusion drawn from this study is that the long-term application of nitrogen, phosphorus, and potassium fertilizer; phosphorus and potassium fertilizer; and nitrogen and phosphorus fertilizer can improve soil fertility, alter the soil environment, enhance microbial diversity, and improve the microbial community structure in yellow soil paddy fields, promoting soil ecosystem stability and health.