Inulin is a well-recognized prebiotic ingredient established to modulate the gut microbiome and its metabolic functionality. However, little is known about how the food matrix interacts with the prebiotic efficacy of inulin. The aim of the present study was to investigate the interaction between the food matrix (milk vs. yogurt) and the gut microbiome modulatory effects of inulin and its influence on calcium bioavailability as reflected in bone mineralization. For this purpose, a 6-week dietary intervention was conducted in healthy young growing male rats (n = 36) which received a diet matrix that included: (1) milk, (2) milk supplemented with 5% inulin, (3) yogurt, or (4) yogurt supplemented with 5% inulin. All diets were limited in calcium content and provided a daily intake of 46 mg calcium per rat. We found that inulin fortification of a yogurt diet exerted a larger effect on gut fermentation as reflected in pH and the generation of acetate in the distal part of the intestine and feces compared with inulin fortification of milk. Inulin was also associated with a higher acetate concentration in plasma when supplied in yogurt compared with milk. No effects of inulin supplementation were found on bone parameters. In conclusion, the present study suggested that the prebiotic efficacy of inulin is higher when supplied in a fermented dairy product than milk. However, neither adding inulin to yogurt or milk affected bone mineralization or the bone structure.