Contextual influences on responses to facial expressions of emotion were studied using a context-target paradigm that allowed distinguishing the effects of affective congruency (context and target of same/different valence: positive or negative) and emotional congruency (context and target representing the same/different emotion: anger, fear, happiness). Sentences describing anger, fear or happiness-inducing events and faces expressing each of these emotions were used as contexts and targets, respectively. While between-valence comparisons (context and target of similar/different valence) revealed affective congruency effects, within-valence comparisons (context and target of similar valence and same/different emotion) revealed emotional congruency effects. In Experiment 1 no evidence of emotional congruency and limited evidence of affective congruency were found with an evaluative task. In Experiment 2 effects of both affective and emotional congruency were observed with an emotion recognition task. In this case, angry and fearful faces were recognized faster in emotionally congruent contexts. In Experiment 3 the participants were asked explicitly to judge the emotional congruency of the target faces. Emotional congruency effects were again found, with faster judgments of angry and fearful faces in the corresponding emotional contexts. Moreover, judgments of angry expressions were faster and more accurate in happy than in anger contexts. Thus, participants found easier to decide that angry faces did not match a happy context than to judge that they did match an anger context. These results suggest that there are differences in the way that facial expressions of positive and negative emotions are discriminated and integrated with their contexts. Specifically, compared to positive expressions, contextual integration of negative expressions seems to require a double check of the valence and the specific emotion category of the expression and the context.