Sexual selection on males is predicted to increase population fitness, and delay population extinction, when mating success negatively covaries with genetic load across individuals. However, such benefits of sexual selection could be counteracted by simultaneous increases in genome-wide drift resulting from reduced effective population size caused by increased variance in fitness. Resulting fixation of deleterious mutations could be greatest in small populations, and when environmental variation in mating traits partially decouples sexual selection from underlying genetic variation. The net consequences of sexual selection for genetic load and population persistence are therefore likely to be context dependent, but such variation has not been examined. We use a genetically explicit individual-based model to show that weak sexual selection can increase population persistence time compared to random mating. However, for stronger sexual selection such positive effects can be overturned by the detrimental effects of increased genome-wide drift. Furthermore, the relative strengths of mutation-purging and drift critically depend on the environmental variance in the male mating trait. Specifically, increasing environmental variance caused stronger sexual selection to elevate deleterious mutation fixation rate and mean selection coefficient, driving rapid accumulation of drift load and decreasing population persistence times. These results highlight an intricate balance between conflicting positive and negative consequences of sexual selection on genetic load, even in the absence of sexually antagonistic selection. They imply that environmental variances in key mating traits, and intrinsic genetic drift, should be properly factored into future theoretical and empirical studies of the evolution of population fitness under sexual selection.