Alterations in mucosal microbiota and metabolites are critical to intestinal homeostasis and host health. This study used a combination of 16S rRNA gene sequencing and liquid chromatography-mass spectrometry (LC/MS) to investigate mucosal microbiota and their metabolic profiles in the ileum of Hu sheep fed different diets. Here, we randomly allocated 15 Hu sheep to three diets, a non-pelleted low-grain diet (control diet; CON), a non-pelleted high-grain diet (HG), and a pelleted high-grain diet (HP). After 60 days of treatment, ileal mucosal samples were collected for microbiome and metabolome analysis. The results of principal coordinate analysis and permutation multivariate analysis showed that there was a tendency for microbial differentiation between the CON and HG groups (P < 0.1), although no significant difference between the HG and HP groups was observed (P > 0.05). Compared with the CON diet, the HG diet decreased (P < 0.05) the abundance of some probiotic species (e.g., Sphingomonas and Candidatus Arthromitus) and increased (P < 0.05) the abundance of acid-producing microbiota (e.g., Succiniclasticum, Nesterenkonia, and Alloprevotella) in the ileal mucosa. Compared with the HG diet, the HP diet decreased (P < 0.05) the abundance of Alloprevotella and increased (P < 0.05) the abundance of Mycoplasma in the ileal mucosa. Furthermore, partial least squares discriminant analysis and orthogonal partial least-squared discriminant analysis indicated that different dietary treatments resulted in different metabolic patterns in the ileal mucosa of the CON, HG, and HP groups. The HG diet altered (VIP > 1 and P < 0.05) the metabolic patterns of amino acids, fatty acids, and nucleotides/nucleosides (such as increased amounts of ornithine, tyrosine, cis-9-palmitoleic acid, and adenosine) compared with the CON diet. However, 10 differential metabolites (VIP > 1 and P < 0.05; including tyrosine, ornithine, and cis-9-palmitoleic acid) identified in the HG group exhibited a diametrically opposite trend in the HP group, suggesting that the HP diet could partially eliminate the changes brought upon by the HG diet. Collectively, our findings demonstrate that different diets altered the ileal mucosal microbiota and metabolites and provide new insight into the effects of high-grain diets on the intestinal health of ruminant animals.