The root-associated microbiomes play important roles in plant growth. However, it is largely unknown how wheat variety evolutionary relatedness shapes each subcommunity in the root microbiome and, in turn, how these microbes affect wheat yield and quality. Here we studied the prokaryotic communities associated with the rhizosphere and root endosphere in 95 wheat varieties at regreening and heading stages. The results indicated that the less diverse but abundant core prokaryotic taxa occurred among all varieties. Among these core taxa, we identified 49 and 108 heritable amplicon sequence variants, whose variations in relative abundances across the root endosphere and rhizosphere samples were significantly affected by wheat variety. The significant correlations between phylogenetic distance of wheat varieties and prokaryotic community dissimilarity were only observed in non-core and abundant subcommunities in the endosphere samples. Again, wheat yield was only significantly associated with root endosphere microbiota at the heading stage. Additionally, wheat yield could be predicted using the total abundance of 94 prokaryotic taxa as an indicator. Our results demonstrated that the prokaryotic communities in the root endosphere had higher correlations with wheat yield and quality than those in the rhizosphere; thus, managing root endosphere microbiota, especially core taxa, through agronomic practices and crop breeding, is important for promoting wheat yield and quality.