Sustainable agricultural practices such as cover crops (CCs) and residue retention are increasingly applied to counteract detrimental consequences on natural resources. Since agriculture affects soil properties partly via microbial communities, it is critical to understand how these respond to different management practices. Our study analyzed five CC treatments (oat, rye, radish, rye-radish mixture and no-CC) and two crop residue managements (retention/R+ or removal/R−) in an 8-year diverse horticultural crop rotation trial from ON, Canada. CC effects were small but stronger than those of residue management. Radish-based CCs tended to be the most beneficial for both microbial abundance and richness, yet detrimental for fungal evenness. CC species, in particular radish, also shaped fungal and, to a lesser extent, prokaryotic community composition. Crop residues modulated CC effects on bacterial abundance and fungal evenness (i.e., more sensitive in R− than R+), as well as microbial taxa. Several microbial structure features (e.g., composition, taxa within Actinobacteria, Firmicutes and Ascomycota), some affected by CCs, were correlated with early biomass production of the following tomato crop. Our study suggests that, whereas mid-term CC effects were small, they need to be better understood as they could be influencing cash crop productivity via plant-soil feedbacks.