The importance of the right hemisphere in emotion perception in general has been well documented but its precise role is disputed. We compared the performance of 30 right hemisphere damaged (RHD) patients, 30 left hemisphere damaged (LHD) patients, and 50 healthy controls on both facial and vocal affect perception tasks of specific emotions. Brain damaged subjects had a single episode cerebrovascular accident localised to one hemisphere. The results showed that right hemisphere patients were markedly impaired relative to left hemisphere and healthy controls on test performance: labelling and recognition of facial expressions and recognition of emotions conveyed by prosody. This pertained at the level of individual basic emotions, positive versus negative, and emotional expressions in general. The impairment remained highly significant despite covarying for the group’s poorer accuracy on a neutral facial perception test and identification of neutral vocal expressions. The LHD group were only impaired relative to controls on facial emotion tasks when their performance was summed over all the emotion categories and before age and other cognitive factors were taken into account. However, on the prosody test the LHD patients showed significant impairment, performing mid-way between the right hemisphere patients and healthy comparison group. Recognition of positive emotional expressions was better than negative in all subjects, and was not relatively poorer in the LHD patients. Recognition of individual emotions in one modality correlated weakly with recognition in another, in all three groups. These data confirm the primacy of the right hemisphere in processing all emotional expressions across modalities—both positive and negative—but suggest that left hemisphere emotion processing is modality specific. It is possible that the left hemisphere has a particular role in the perception of emotion conveyed through meaningful speech.