Cover cropping is a diversifying agricultural practice that can improve soil structure and function by altering the underground litter diversity and soil microbial communities. Here, we tested how a wheat cover crop alters the decomposition of cucumber root litter. A three-year greenhouse litterbag decomposition experiment showed that a wheat cover crop accelerates the decomposition of cucumber root litter. A microcosm litterbag experiment further showed that wheat litter and the soil microbial community could improve cucumber root litter decomposition. Moreover, the wheat cover crop altered the abundances and diversities of soil bacterial and fungal communities, and enriched several putative keystone OTUs, such as Bacillus spp. OTU1837 and Mortierella spp. OTU1236, that were positively related to the mass loss of cucumber root litter. The representative bacterial and fungal strains B186 and M3 were isolated and cultured. In vitro decomposition tests demonstrated that both B186 and M3 had cucumber root litter decomposition activity and a stronger effect was found when they were co-incubated. Overall, a wheat cover crop accelerated cucumber root litter decomposition by altering the soil microbial communities, particularly by stimulating certain putative keystone taxa, which provides a theoretical basis for using cover crops to promote sustainable agricultural development.