Anecdotal evidence suggests that people have a desire not only to punish transgressors but also some of their bystanders. Remaining unknown are how the demand for punishing bystanders relates to existing theories of altruistic punishment against transgressors, and what psychological rationale might account for any differences observed. We show (Study 1) that people exhibit consistently lower demand for punishing bystanders than transgressors. We argue that the psychological mechanism underlying this pattern could also help explain why some bystanders are seen as more blameworthy than others. We develop and test the theory that differences in expectations influence intent-related cognition, and this accounts for differences in punishment across targets. In Study 1, we gather process evidence that participants judging bystanders, who are less likely to be seen as having violated expectations, have longer reaction times, perceive lower clarity in judging intent, and more easily generate exonerating thoughts. In Study 2, we show that participants are also less likely to spontaneously generate thoughts about the intent of the bystander, but increasing the accessibility of thoughts about negative (positive) intent leads to increased (decreased) punishment for the bystander. Manipulating the accessibility of intent-related thoughts can reduce differences in punishment between bystanders and transgressors. Additionally, we show that this psychological mechanism explains variation in different people’s responses to the same bystander. This research provides insight into why people condemn some bystanders while accepting the silence of others and expands the scope of the current discussion on altruistic punishment.