Currently, in the study of second language acquisition, affective factors have attached great attention. “Among the affective factors influencing language learning, anxiety ranks high.” (Arnold, 1999, p.59). Its importance has led to significant researches and discussions from different perspectives. In general, studies on language learning anxiety have developed to a more mature stage, while the empirical studies of specific language-skill anxiety, especially, second language listening anxiety, are inadequate. The present research made an investigation on students’ anxiety in English listening comprehension by using quantitative and qualitative methods. The subjects were 82 freshmen of English majors in a normal university. A modified foreign language listening questionnaire was administered to examine the subjects’ listening anxiety level, and a series of interviews were done for exploring the sources of anxiety deeply. Through thorough analyses, the paper arrived at the following conclusion: (a) the freshmen of English majors did experience a little higher listening anxiety; (b) there existed a significantly negative correlation between anxiety and listening comprehension; (c) the main sources were lack of confidence and listening strategies, characteristics of listening comprehension and materials as well as fear of negative evaluation. Based on the above conclusion, some useful suggestions were put forward to enhance the students’ listening proficiency and improve the current situations of listening classes.