Complex group behavior can emerge from simple inter-individual interactions. Commonly, these interactions are considered static and hardwired and little is known about how experience and learning affect collective group behavior. Young larvae use well described visuomotor transformations to guide interindividual interactions and collective group structure. Here, we use naturalistic and virtual-reality (VR) experiments to impose persistent changes in population density and measure their effects on future visually evoked turning behavior and the resulting changes in group structure. We find that neighbor distances decrease after exposure to higher population densities, and increase after the experience of lower densities. These adaptations develop slowly and gradually, over tens of minutes and remain stable over many hours. Mechanistically, we find that larvae estimate their current group density by tracking the frequency of neighbor-evoked looming events on the retina and couple the strength of their future interactions to that estimate. A time-varying state-space model that modulates agents' social interactions based on their previous visual-social experiences, accurately describes our behavioral observations and predicts novel aspects of behavior. These findings provide concrete evidence that inter-individual interactions are not static, but rather continuously evolve based on past experience and current environmental demands. The underlying neurobiological mechanisms of experience dependent modulation can now be explored in this small and transparent model organism.