The effects of heterogeneity in group composition remain a major hurdle to our understanding of collective behavior across disciplines. In social insects, division of labor (DOL) is an emergent, colony-level trait thought to depend on colony composition. Theoretically, behavioral response threshold models have most commonly been employed to investigate the impact of heterogeneity on DOL. However, empirical studies that systematically test their predictions are lacking because they require control over colony composition and the ability to monitor individual behavior in groups, both of which are challenging. Here, we employ automated behavioral tracking in 120 colonies of the clonal raider ant with unparalleled control over genetic, morphological, and demographic composition. We find that each of these sources of variation in colony composition generates a distinct pattern of behavioral organization, ranging from the amplification to the dampening of inherent behavioral differences in heterogeneous colonies. Furthermore, larvae modulate interactions between adults, exacerbating the apparent complexity. Models based on threshold variation alone only partially recapitulate these empirical patterns. However, by incorporating the potential for variability in task efficiency among adults and task demand among larvae, we account for all the observed phenomena. Our findings highlight the significance of previously overlooked parameters pertaining to both larvae and workers, allow the formulation of theoretical predictions for increasing colony complexity, and suggest new avenues of empirical study.