Observation of neural firing patterns can constrain theories for the types of activity patterns that the brain uses to guide behavior. However, directly perturbing these patterns, ideally with great specificity, is required to causally test any particular theory. We combined two-photon imaging and cellular resolution optogenetic photo-stimulation to causally test how neural activity in the mouse visual cortex is read out to detect visual stimuli. Contrary to expectations, targeted activation of highly sensitive neural ensembles did not preferentially modify behavior compared to random ensembles, contradicting a longstanding hypothesis for how neural activity drives stimulus detection. Instead, the main predictor of a targeted neural ensemble's impact on perception was its effect on network activity. This argues that downstream regions summate visual cortex activity without preferentially weighting more informative neurons to make sensory detection decisions. Comparing mouse behavioral performance to decoding models of neural activity implies that mice employ this simple, albeit suboptimal strategy to solve the task. This work challenges conventional notions for how sensory representations mediate perception and demonstrates that specific neural perturbations are critical for determining which features of neural activity drive behavior.