Soil biodiversity is threatened by intensive agriculture that relies on annual grain crop production, thus leading to a decline in soil functions and ecosystem services. Perennial grain crops have a positive impact on the soil microbial community, but the responsive microbial groups and the magnitude of their response remain uncertain. To elucidate this, we analysed soil microbial biomass and community composition, bacterial growth and soil total carbon in five crops: organic perennial intermediate wheatgrass (IWG, Thinopyrum intermedium, Kernza®), organic IWG-alfalfa intercrop, organic biennial grass-legume mixture, organic annual wheat or rye and conventional annual wheat. The analysis was carried out at three time points under two growing seasons at four different soil depths. Five years after establishment, IWG had greater amounts of soil total fungi and bacteria, and of arbuscular mycorrhizal (AM) fungi, saprotrophic fungi, gram-negative (G−) and gram-positive (G+) bacteria compared to annual wheat. Crop perenniality influenced the soil microbial community structure although precipitation, soil temperature and water content were the main drivers of the patterns of and temporal variations in the microbial community. Perennial crops, with reduced tillage and low nitrogen input management increased the proportions of fungi relative to bacteria, AM fungi to saprotrophic fungi, G− bacteria to G+ bacteria, and the growth rate of total bacteria. This resulted in a more active soil microbial community with higher microbial biomass than annual wheat and contributed to the increased soil total carbon storage in the 0–5 cm soil layer in a humid continental climate. The findings emphasize the importance of combining a no tillage strategy with long-term vegetation cover to increase soil quality.