The aim of this study was to investigate the effects of diets on the composition and function of rumen microbiome and metabolites in Sanhe heifers. Metagenomic and metabolomic analyses were performed using rumen fluid samples collected from Sanhe heifers (n = 20) with similar body weights and ages from grass-fed and grain-fed systems. The grain-fed group exhibited more intensive rumen fermentation than the grass-fed group. However, the grass-fed group exhibited carbohydrate metabolism and methane production higher than that of the grain-fed group; these increases were observed as a higher abundance of various bacterial phyla (Firmicutes, Bacteroidetes, Actinobacteria, Lentisphaerae, and Verrucomicrobia), families (Lachnospiraceae, Eubacteriaceae, and Eggerthellaceae), and the archaeal family Methanobacteriaceae. A comparison of genes encoding carbohydrate-active enzymes, using Kyoto Encyclopedia of Genes and Genome profiles, revealed noteworthy differences in the functions of rumen microbiota; these differences were largely dependent on the feeding system. These results could help manipulate and regulate feed efficiency in Sanhe cattle.