In the context of tertiary education, the ability to evaluate sources is of paramount importance for students’ academic reading and writing development, particularly for first-year university students who are still in the transitional stage. Some researchers have argued that Chinese students may lack critical thinking skills, including evaluation skills. However, little attention has been given to this higher-order thinking skill when it comes to first-year Chinese university students reading academic literature. Therefore, this study aimed to investigate the criteria used by 117 Chinese first-year undergraduates when evaluating academic literature and how these criteria influence their evaluation accuracy. The results revealed that students generally demonstrated the ability to distinguish reliable from unreliable academic documents by utilizing a range of evaluation criteria. Among the identified criteria, Topicality and Publication Information were prominently employed by students. Furthermore, these criteria accounted for approximately 21% of the variance in evaluation accuracy, with Publication Information, Reference, and Data significantly predicting evaluation performance, particularly Publication Information. The study also discussed the pedagogical implications of teaching academic literacy to first-year undergraduates.