In the current education field, the assessment of teaching management quality mostly relies on subjective judgment and static data, and lacks a real-time and dynamic feedback mechanism. In this study, we propose a deep learning-based human behavior analysis method, which aims to assess teaching management quality in real time by analyzing the behaviors of teachers and students in the classroom. First, in order to detect individual students in the video stream, an augmented detection framework based on YOLO v5s is introduced to process and analyze human actions and interaction patterns in the video data. Immediately after that, we design a channel residual decoupled convolutional neural network to recognize the different states of students. Teaching management quality is assessed by detecting students' classroom attention scores. By conducting experiments in different disciplines and teaching management environments to collect and train the model, the results show that the method can effectively improve the objectivity and accuracy of teaching management quality assessment.