Background: Fat infiltration in skeletal muscle is related to declining muscle strength, whereas excess subcutaneous fat is implicated in the development of metabolic diseases. Methods: Using multi-slice axial T2-weighted (T2w) MR images, this retrospective study characterized muscle fat infiltration (MFI) and fat distribution in the lower extremity of 107 subjects (64M/43F, age 11-79 years) with diverse ethnicities (including White, Black, Latino, and Asian subjects). Results: MRI data analysis shows that MFI, evaluated by the relative intensities of the pixel histogram profile in the calf muscle, tends to increase with both age and BMI. However, statistical significance was found only for the age correlation in women (p < 0.002), and the BMI correlation in men (p = 0.04). Sex disparities were also seen in the fat distribution, which was assessed according to subcutaneous fat thickness (SFT) and the fibula bone marrow cross-sectional area (BMA). SFT tends to decrease with age in men (p < 0.01), whereas SFT tends to increase with BMI only in women (p < 0.01). In contrast, BMA tends to increase with age in women (p < 0.01) and with BMI in men (p = 0.04). Additionally, MFI is positively correlated with BMA but not with SFT, suggesting that compromised bone structure may contribute to fat infiltration in the surrounding skeletal muscle. Conclusions: The findings of this study highlight a sex factor affecting MFI and fat distribution, which may offer valuable insights into effective strategies to prevent and treat MFI in women versus men.