Grassland afforestation strongly influences the structure and function of soil microorganisms. Yet the mechanisms of how afforestation could simultaneously alter both the soil fungal and bacterial communities and its implications for ecosystem management are poorly understood, especially in nitrogen-limited ecosystems. Using high-throughput sequencing of 16S rRNA and ITS rRNA genes, the present study investigated the changes in soil properties and soil microorganisms after afforestation of natural grasslands with Chinese pine (Pinus tabuliformis) on the Loess Plateau in China. Results showed that soil bacterial diversity had no significant differences among the grassland (GL), forest–grassland transition zone (TZ), and forestland (FL), while soil fungal diversity in the GL was significantly higher than that in the FL and TZ (P < 0.05). The proportion of shared OTUs in the soil bacterial community was higher than that in the soil fungal community among the three land use types. The dominant bacterial phylum shifted from Proteobacteria to Actinobacteria, while the dominant fungal phylum shifted from Ascomycota to Basidiomycota after the GL conversion to the FL. The functional groups of ECM fungi increased significantly while biotrophic fungi decreased significantly after grassland afforestation. Both the soil bacterial and fungal communities in the TZ showed great similarity with those in the FL. In addition, among all examined soil properties, soil nitrogen (N) showed a more significant effect on the soil microbial communities. The reduction of soil N after grassland afforestation resulted in both the structure and function changes in soil microbial communities. Our results demonstrated simultaneously differential changes in the composition and diversity of both soil bacterial and fungal communities after afforestation from grasslands to planted forests.