In many species, males influence phenotypic traits in their offspring through non-genetic paternal effects. Such effects can represent a form of paternal investment, and males may benefit by adjusting the effects depending on environmental parameters, such as operational sex ratio, so as to maximize offspring fitness. In the neriid fly Telostylinus angusticollis, fathers reared on a nutrient-rich larval diet produce larger offspring, independent of the rearing environment of the offspring. Here we asked whether this paternal effect was influenced by the social environment to which fathers were exposed. We found significant interactions of the effects of paternal larval diet quality and social environment (same-sex vs. mixed-sex groups) on offspring fitness-related traits. Fathers reared on a nutrient-rich diet produced larger male offspring when housed in mixed-sex groups. However, fathers reared on a nutrient-rich diet produced more viable offspring (or more viable sperm) when housed in same-sex groups prior to mating. These results suggest that fitness-enhancing paternal effects can trade off, consistent with parental investment theory on the offspring size-number trade-off, which suggests that these traits represent alternative investment options and parents are selected to optimize the balance based on a range of environmental variables. This is the first study to show that males can facultatively modulate paternal effects based on the social environment.