Earlier studies have shown that Stanleya pinnata benefits from selenium hyperaccumulation through ecological benefits and enhanced growth. However, no investigation has assayed the effects of Se hyperaccumulation on plant fitness in the field. This research aimed to analyze how variation in Se accumulation affects S. pinnata fitness, judged from physiological and biochemical performance parameters and herbivory while growing naturally on two seleniferous sites. Natural variation in Se concentration in vegetative and reproductive tissues was determined, and correlations were explored between Se levels with fitness parameters, herbivory damage, and plant defense compounds. Leaf Se concentration varied between 13- and 55-fold in the two populations, averaging 868 and 2482 mg kg−1 dry weight (DW). Furthermore, 83% and 31% of plants from the two populations showed Se hyperaccumulator levels in leaves (>1000 mg kg−1 DW). In seeds, the Se levels varied 3–4-fold and averaged 3372 and 2267 mg kg−1 DW, well above the hyperaccumulator threshold. Plant size and reproductive parameters were not correlated with Se concentration. There was significant herbivory pressure even on the highest-Se plants, likely from Se-resistant herbivores. We conclude that the variation in Se hyperaccumulation did not appear to enhance or compromise S. pinnata fitness in seleniferous habitats within the observed Se range.