Feed efficiency is a highly important economic trait in sheep production and has a significant impact on the economic benefits of sheep farming. Microbial fermentation of the rumen has a vital role in the host's nutrition; the rumen microbiota might affect host feed efficiency. However, the relationship between the rumen microbiota and feed efficiency in sheep is unclear. In the present study, the microbiota of 195 Hu sheep was investigated and their residual feed intake (RFI), a commonly used measure of feed efficiency, was determined. From birth, all sheep were subjected to the same management practices. At slaughter, samples of liquid rumen contents were collected and subjected to amplicon sequencing for the 16S rDNA gene on the IonS5™XL platform. To identify the bacterial taxa differentially represented at the genus or higher taxonomy levels, we used linear discriminant analysis coupled with effect size and curve fitting. In the sheep rumen, the four most abundant phyla were Firmicutes, Bacteroidetes, Fibrobacteres, and Proteobacteria; and the dominant genera were unidentified Prevotellaceae, Fibrobacter, unidentified Lachnospiraceae, Saccharofermentans, and Succinivibrio. Pathway analysis of the 16S rDNA sequencing data from the rumen microbiota identified that carbohydrate metabolism was enriched. Using α-diversity analysis, we further identified that Observed species, ACE, Good's coverage, and Chao1 are more abundant (P < 0.01) in the low-RFI (L-RFI) group compared to the high-RFI (H-RFI) group. High-RFI sheep had a higher abundance of three bacterial taxa (Prevotellaceae, Negativicutes, and Selenomonadales), and one taxa was overrepresented in the L-RFI sheep (Succinivibrio), respectively. Furthermore, model fitting showed that Veillonellaceae, Sphaerochaeta, Negativibacillus, Saccharofermentans, and members of the Tenericutes, Kiritimatiellaeota, Deltaproteobacteria, and Campylobacterales were correlated with the sheep RFI classification and thus were indicative of a role in animal efficiency. Tax4Fun analysis revealed that metabolic pathways such as “energy metabolism,” “metabolism of cofactors and vitamins,” “poorly characterized,” and “replication recombination and repair proteins” were enriched in the rumen from H-RFI sheep, and “genetic information processing” and “lipopolysaccharide biosynthesis” were overrepresented in L-RFI sheep rumen. In addition, six Kyoto Encyclopedia of Genes and Genomes orthology pathways were identified as different between H-RFI and L-RFI groups. In conclusion, the low RFI phenotype (efficient animals) consistently (or characteristically) exhibited a more abundant and diverse microbiome in sheep.