E-learning provides a flexible way of learning, which can provide corresponding learning content and tasks according to students’ interests, abilities and learning progress, so that students can choose learning content and learning progress more independently. In traditional teaching methods, teachers need to spend a lot of time and energy to evaluate students’ learning. The e-learning system can automatically record students’ learning behavior and learning results, and give feedback and evaluation in time during the learning process. Automatic speech recognition systems have been widely applied in various fields, and the demand for more user-friendly and convenient user operating systems is becoming increasingly urgent. This article proposes an English automatic speech recognition system based on hidden Markov model algorithm and conditional random field algorithm to improve recognition accuracy and stability, and applies it to the computer field. Label inference and optimization for speech recognition were carried out using the conditional random field algorithm. By establishing the correlation between features and labels, label inference was performed on the speech signal, which was compared with existing speech recognition systems to evaluate whether the recognition accuracy and stability of the new system had been improved. The results show that the English automatic speech recognition system based on hidden Markov model algorithm and conditional random field algorithm exhibits high performance in terms of recognition accuracy and stability. Compared with traditional speech recognition systems, this system has made significant improvements in the application of computer science.