Abstract This study addresses the application of rough set theory in English hierarchical teaching and explores how to optimize the teaching mode and improve the teaching quality and learning effect through this theory. This paper adopts a literature analysis and questionnaire survey method to study rough set theory’s knowledge expression and systematic decision-making mechanism. It applies it to the design of the curriculum system and the innovation of the teaching mode in the English platform of special-purpose universities. The optimized English hierarchical teaching mode using rough set theory significantly improved students’ learning interest and learning effect, in which 87.53% of the students believed it could improve their mastery of knowledge. The average score of 90.3 was 23.4 points higher than that of the control group, which was statistically significant (P<0.05). This study also found that the level of teacher specialization, students’ study habits, and school policies are key factors that influence the implementation of the teaching model. The introduction of rough set theory in this study provides new perspectives and methods for English teaching, promoting innovation in teaching methods and improving students’ learning ability.