Categories allow humans to anticipate sensory properties of a food before eating it. Placement of foods into different categories has implications for marketing and food product development. Current research in food categorization has focused on card sorting methods using pictures or words, but our understanding of sensory-based food categorization and its potential variability across individuals remains limited. The purpose of this study was to determine if a person’s experiences and habits affect the way they categorize foods in the category of milk and milk alternatives. In this two-part study, focus groups were first conducted with 34 participants to determine how usage and important attributes of milk differed across diet demographic groups. Then, the influence of demographics on categorization was examined using agglomerative free sorting in which 69 panellists sorted 24 distinct milk alternative products and 31 panellists sorted 30 milk and milk alternative products. The results from a DISTATIS analysis revealed that sugar content was a primary factor in categorization across all demographic groups, indicating a potential universal hierarchical categorization process. When removing sugar content as a factor, differences in categorization were related to attitudes towards creaminess, nutrition, price, colour, and usage of milk in drinks. These insights trigger new questions about what other attributes would drive categorization when removing sugar as a factor and how sugar content would affect categorization of solid foods.