Sex differences in morphology, physiology, development, and behavior are widespread, yet the sexes inherit nearly identical genomes, causing most traits to exhibit strong and positive cross‐sex genetic correlations. In contrast to most other traits, estimates of cross‐sex genetic correlations for fitness and fitness components (rW fm ) are generally low and occasionally negative, implying that a substantial fraction of standing genetic variation for fitness might be sexually antagonistic (i.e., alleles benefitting one sex harm the other). Nevertheless, while low values of rW fm are often regarded as consequences of sexually antagonistic selection, it remains unclear exactly how selection and variation in quantitative traits interact to determine the sign and magnitude of rW fm , making it difficult to relate empirical estimates of cross‐sex genetic correlations to the evolutionary processes that might shape them. We present simple univariate and multivariate quantitative genetic models that explicitly link patterns of sex‐specific selection and trait genetic variation to the cross‐sex genetic correlation for fitness. We show that rW fm provides an unreliable signal of sexually antagonistic selection for two reasons. First, rW fm is constrained to be less than the cross‐sex genetic correlation for traits affecting fitness, regardless of the nature of selection on the traits. Second, sexually antagonistic selection is an insufficient condition for generating negative cross‐sex genetic correlations for fitness. Instead, negative fitness correlations between the sexes (rW fm <0) can only emerge when selection is sexually antagonistic and the strength of directional selection on each sex is strong relative to the amount of shared additive genetic variation in female and male traits. These results imply that empirical tests of sexual antagonism that are based on estimates of rW fm will be conservative and underestimate its true scope. In light of these theoretical results, we revisit current data on rW fm and sex‐specific selection and find that they are consistent with the theory.