In natural vision both stimulus features and cognitive/affective factors influence an observer's attention. However, the relationship between stimulus-driven (bottom-up) and cognitive/affective (top-down) factors remains controversial: How well does the classic visual salience model account for gaze locations? Can emotional salience counteract strong visual stimulus signals and shift attention allocation irrespective of bottom-up features? Here we compared Itti and Koch's [2000] and Spectral Residual (SR) visual salience model and explored the impact of visual salience and emotional salience on eye movement behavior, to understand the competition between visual salience and emotional salience and how they affect gaze allocation in complex scenes viewing. Our results show the insufficiency of visual salience models in predicting fixation. Emotional salience can override visual salience and can determine attention allocation in complex scenes. These findings are consistent with the hypothesis that cognitive/affective factors play a dominant role in active gaze control.