In social species across the animal kingdom, conspecific outsiders threaten the valuable resources of groups and their members. This outgroup conflict is recognised as a powerful selection pressure, but we argue that studies explicitly quantifying the fitness consequences need to be broader in scope: more attention should be paid to delayed, cumulative, and third-party fitness consequences, not just those arising immediately to group members involved in physical contests. In the first part of this review, we begin by documenting how single contests can have survival and reproductive consequences either immediately or with a delay. Then, we step beyond contests to describe fitness consequences that can also result from interactions with cues of rival presence and the general landscape of outgroup threat, and beyond single interactions to describe cumulative effects of territorial pressure and elevated outgroup-induced stress. Using examples from a range of taxa, we discuss which individuals are affected negatively and positively, considering both interaction participants and third-party group members of the same or the next generation. In the second part of the review, we provide suggestions about how to move forward. We highlight the importance of considering how different types of outgroup conflict can generate different selection pressures and of investigating variation in fitness consequences within and between species. We finish by discussing the value of theoretical modelling and long-term studies of natural populations, experimental manipulations, and meta-analyses to develop further our understanding of this crucial aspect of sociality.