Group size is an important trait for many ecological and evolutionary processes. However, it is not a trait possessed by individuals but by social groups, and as many genomes contribute to group size understanding its genetic underpinnings and so predicting its evolution is a conceptual challenge. Here I suggest how group size can be modelled as a joint phenotype of multiple individuals, and so how models for evolution accounting for indirect genetic effects are essential for understanding the genetic variance of group size. This approach makes it clear that (a) group size should have a larger genetic variance than initially expected as indirect genetic effects always contribute exactly as much as direct genetic effects and (b) the response to selection of group size should be faster than expected based on direct genetic variance alone as the correlation between direct and indirect effects is always at the maximum positive limit of 1. Group size should therefore show relatively rapid evolved increases and decreases, the consequences of which and evidence for I discuss.