(1) Background: Food-based dietary guidelines promote population health and well-being through dietary patterns that reduce chronic disease risk while providing adequate energy and nutrients. In Australia, recommended dietary patterns based on servings per day from the five food groups—fruits, vegetables, cereals and grains, meats and alternatives, and dairy—have been developed for toddlers 1–2 years of age. However, no study has assessed the intake of the five food groups in this age group nationally. (2) Aim: To compare daily servings and the percentage of energy from the five food groups and discretionary foods in toddlers 1–2 years old to the Australian Dietary Guidelines. (3) Methods: Dietary intake was assessed using a one-day food record for 475 toddlers. (4) Results: Apart from fruit and dairy, servings of the five food groups were below the recommendations. Two-thirds of toddlers did not consume enough vegetables, and only 10% consumed the recommended number of servings for cereals and grains. On average, toddlers consumed only half the recommended servings of meat and alternatives. Nearly all toddlers (89%) consumed discretionary foods, which accounted for ~12% of total energy. Forty-five percent of toddlers received breastmilk. On average, breastfed toddlers consumed fewer servings from the five food groups than non-breastfed toddlers. Dairy contributed 20% of daily energy in all toddlers; however, this food group accounted for 13% in breastfed and 32% in non-breastfed toddlers on the day of the food record. (4) Conclusions: Compared to the recommendations, alignment with the servings of the five food group foods was not achieved by most toddlers, except for fruit and dairy. Discretionary foods may have displaced nutritious family foods. Consistent with Australian Infant Feeding Guidelines, many toddlers in this study continued to receive breastmilk but the recommended dietary patterns do not include breastmilk. Dietary modeling, including breastmilk as the primary milk source, is urgently needed, along with practical advice on incorporating breastmilk in a toddler’s diet while optimizing food consumption.